You can set up conditional paths within your data flow so that run time decisions about the data determine whether or not certain nodes are run. For example, you may only want to run a set of nodes if the input data contains a specific field, or if the input data contains a minimum number of records.
To create a conditional path in your data flow:
- Configure the Meta Check node to perform run time metadata checks on the data.
- Connect the Meta Check node to the conditional nodes by using Clocks, i.e. the nodes that you only want to run in certain circumstances.
You have a CSV file containing financial product information. Using a Meta Check node, you want to check if the data contains a "Product" field, and if so, you want to combine the data with product sales information contained in an Excel file.
Step 1 - Importing data
- Drag a CSV/Delimited Input node onto the canvas and set the Data property to import the CSV file.
The following is an example of the data imported from CSV:
Drag an Excel File node onto the canvas and configure the File property to import the product sales data.Product
unicode
Interest rate
unicode
Cash ISA 1 Fixed rate mortgage 2 Current account NULL Credit card NULL Insurance NULL The following is an example of the data imported from Excel:
Product
unicode
October_sales
double
November_sales
double
December_sales
double
Cash ISA 10 2 5 Fixed rate mortgage 1 5 0 Current account 11 5 4 Credit card 5 4 4 Insurance 6 5 1
Step 2 - Creating a conditional path within the data flow
- Connect the output of the CSV/Delimited Input node to a Meta Check node and also to an Output Excel node.
- Select the Meta Check node and the Output Excel node, right-click and select Run Dependencies > Wait for Completed Run by > Meta Check.
- Configure the Meta Check node as follows:
- Set the TerminusAction property to Clock. This means that the Output Excel node will run if the conditions of the Meta Check node are not met. In this case, the Output Excel node will run if the "Product" field is not present in the input.
- On the SuccessAction property, select Output. This means that if the "Product" field is present on the input data, the node connected to the Meta Check output pin will run, followed by all downstream nodes. Any nodes connected with a clock dependency will not run.
- In the ExpectedFields property, type
Product.
- Add an output pin to the Meta Check node. For details on how to create a new output pin, see Defining input and output pins.
Step 3 - Combining the data streams
Now that you have created a conditional path within the data flow, you can continue to build the remainder of the data flow which will be executed if the conditions of the Meta Check node are met.
-
Drag a Lookup node onto the canvas. Connect the Excel File node to the lookup input of the Lookup node and connect the Meta Check node to the data input.
-
Configure the Lookup node to merge the two data streams on the "Product" field.
- In the Match Keys grid, select
Productfor both the Left Field and Right Field. - In the Field List grid, untick
Productfrom the lookup Fields list.
- In the Match Keys grid, select
-
Run the Lookup node. The merged output is as follows:
Product
unicode
Interest rate
unicode
October_sales
double
November_sales
double
December_sales
double
Cash ISA 1 10 2 5 Fixed rate mortgage 2 1 5 0 Current account NULL 11 5 4 Credit card NULL 5 4 4 Insurance NULL 6 5 1 - Connect the output of the Lookup node to an Output Excel node. Rename the Output Excel node to "Merged output". In this example, the CSV data includes a "Product" field, so the conditions of the Meta Check node are met. As a result, the CSV data is combined with the Excel data and the result is output by the "Merged output" node.
However, if the CSV input had not contained a "Product" field, the Lookup and "Merged output" nodes would not have run. Instead, only the first Output Excel node would run.