Create Pipeline

Summary

The CREATE PIPELINE command creates a pipeline domain object. Pipelines allow developers to build and store complex ETL and data processing logic.

Syntax

create pipeline [if not exists] <pipeline_name>

[(ContinueOnError=<true>|<false>,

OnError=<step or pipeline to invoke>,

Finally=<step or pipeline to invoke>)];

Parameters

Parameter

Description

[if not exists] Create the pipeline only if the pipeline does not currently exist (the pipeline has not already been created).

<pipeline_name> The name of the pipeline.

[as (ContinueOnError=<true|false>,
OnError= <step or pipeline to invoke>,
Finally= <step or pipeline to invoke>)]

(Optional) These parameters determine how a pipeline behaves if one of its step or child pipeline members throws an error during execution.

ContinueOnError = true or false. Connect throws exceptions when a pipeline member fails during processing due to unanticipated system conditions. Alternatively, developers may use the Assert and ThrowException functions to force a pipeline member to throw an exception to trigger a pipeline error. The boolean flags determine if the pipeline should continue or halt processing if one of its members fails. This should probably be set to false in most circumstances.

OnError = the name of the step or pipeline to execute if a pipeline member throws an error. You might want to send an email to your IT staff if a fatal error was thrown, for example.

Finally = the name of the step or pipeline to execute after the parent pipeline has completed. Finally will fire when a pipeline completes successfully. It will also fire if a pipeline member throws an error. In other words, Finally will run no matter what happens during pipeline execution.

Details

A pipeline is a wrapper object used to organize a series of steps and sub-pipelines into a single ETL process flow. A pipeline can contain any number of members. A pipeline member can be a step object or another pipeline object. Each pipeline member is assigned an execution position or sequence. When invoked, a pipeline will sequentially execute each member associated with the pipeline. Pipeline objects are stored on the Connect server.

Use CREATE PIPELINE to define a pipeline. Use ALTER PIPELINE to add step and sub-pipeline members to the pipeline.

Examples

create pipeline p_import_sfdc_opportunities;

create pipeline if not exists p_process_hr_employees
(ContinueOnError=false,
OnError=s_send_error_email,
Finally=s_archive_source_file);

create_pipeline_02

Related Commands

ALTER PIPELINE

DROP PIPELINE

INVOKE PIPELINE