A Pipeline is the term given to the complete end-to-end process of collecting data from a Source and passing it through optional Processes which can filter, transform, or aggregate the data to send to one or more Targets.
To create a Pipeline, click the Create Pipeline button in the top right-hand corner of the Hub’s home page. You will then be presented with a new Pipeline displayed on an area called the Pipeline Canvas, with an undefined Source and an undefined Target. To be able to save and start a Pipeline you must:
- Give it a valid name. Click the pencil icon
at the top-right of the screen to set a name for the Pipeline.Note: The name cannot be empty and cannot contain these characters: * ( # % ) @ , = : % ; / \ | - Configure the Source and Target and optionally, add and configure any Processes you wish to define. For more information, see the Defining Sources, Defining Targets, and Defining Processes sections below.
- All the items on the Pipeline Canvas must be valid, which is denoted by a green ‘check mark’ against them. Items on the Canvas that are invalid or incomplete will have an orange exclamation mark against them. Mandatory fields that are empty or incorrect in Source, Process or Target definitions will be outlined in red and will need your input or attention.
- A Pipeline must be valid to allow it to be saved. It must contain at least one Target and all routes through a Pipeline must start with a Source and end with a Target.
If you decide not to continue with creating a new Pipeline, you can return to the Hub’s home page by clicking the Exit button.