| Change type | Description |
|---|---|
| Introduced in version 3.18 | Capability to design Data Flows as services with defined inputs and outputs. |
Data Flows can be designed as services with defined inputs, known as run properties, and outputs, referred to as Data Flow Outputs. These services can be invoked using various methods:
-
Analyze REST API: Provides a set of APIs to run data flows by passing run properties as inputs and retrieving Data Flow Outputs. It also includes an API to retrieve datasets.
-
Analyze CLI: Offers a command-line interface to run data flows by passing run properties as inputs and retrieving Data Flow Outputs. For more information, see Data360 Analyze CLI Help.
-
Execute Data Flow node: Executes data flows by passing run properties as inputs and retrieving Data Flow Outputs. This method is particularly useful for defining a chain of dependent data flows within a master data flow. For more information, see Execute Data Flow node.
-
Schedules: Allows periodic execution of data flows. Run properties can be defined on a per-schedule basis. For more information, see Scheduling.
Run properties
Run properties are input parameters that enable you to customize a data flow for each execution. These properties are defined within the Data Flow. For more information, see Define run properties. The values for run properties can be specified individually for each run.
Data flow outputs
Data Flow Outputs are output properties defined within the Data Flow. They allow you to specify outputs at the data flow level. For more information, see Data flow outputs.
Publishing data flow output values
The Publish To Data Flow Outputs node is responsible for setting the Data Flow Output values. You can include one or more Publish To Data Flow Outputs nodes at any point in the Data Flow to define output values based on the output of an upstream node. For more information, see Publish To Data Flow Outputs node.
Dataset Output Location
By default, Data Flow Output Dataset files are saved to the temporary execution data
location for the run. However, this location can be modified at the Data Flow level
by setting the predefined run property DataSetOutputTargetLocation.
As a run property, it can also be configured on a per-run basis, similar to other
run properties.
Additionally, the dataset output location can be specified on a per-node basis within
the Publish To Data Flow Outputs node. This setting will
override any run property configuration for
DataSetOutputTargetLocation.