Process model variables - Data360_DQ+ - Latest

Data360 DQ+ Help

Product type
Software
Portfolio
Verify
Product family
Data360
Product
Data360 DQ+
Version
Latest
Language
English
Product name
Data360 DQ+
Title
Data360 DQ+ Help
Copyright
2024
First publish date
2016
ft:lastEdition
2024-07-09
ft:lastPublication
2024-07-09T15:09:58.774265

Variables are part of every process model node.

You can define variables to be passed between nodes and to be used in logical expressions which determine how the process model executes.

Passing variables between nodes

When passing a variable from one node to the next, your selection will be limited to "one node prior."

To pass a variable through multiple nodes, you can use Property Reference variables.

 

Tip: You can also pass variables between process models, see Passing variables between Process Models.

System defined variables

When a node is used to select an executable stage, system defined variables come attached to that stage. These variables are created by Data360 DQ+ when the stage executes or loads.

You cannot change system defined variables, but they can be passed from node to node and used in logical expressions.

Defining a new variable

You can define new Output Variables on a node within your process model for custom use.

  1. Select the node on which you want to define the Output Variable.
  2. From the Properties panel on the right of the screen, click the Add button on the Output Variables property.
  3. Enter a Name for the new variable.
  4. Choose from the following property types (note that the property types that are available will vary according to the node that is selected):
    • Literal - Creates a variable with a fixed value. Literals may then be used by other parts of a process model, such as an Expression variable, or in other nodes.Available data types include String, Datetime, Number, and Boolean.
    • Expression - Applies logic to other variables. May use variables from an incoming node, from the node on which the expression is being built, or from the process model itself.

      When writing expressions, you can use the following syntax to denote the source of the variable:

      • input - Incoming node
      • result - Current node
      • process - Entire process model

      Since multiple variable sources can be used in a single Expression, variables must be prepended by their source using curly braces and a period.

      For example:

      {input}.resultCode - refers to the system defined result code variable from an incoming node.

      {result}.literalX - refers to a user defined literal variable called literalX within the current node.

      {process}.refEndTime - refers to the system defined end time variable that exists for the entire process model.

    • Property Reference - References variables from other nodes.

      TaskA (X) TaskB (TaskA.X) = (Y)

      For example, if there were a variable (X) in TaskA that you'd like to use in TaskB, you could create a Property Reference variable (Y). Doing so would effectively pass variable X to TaskB, making it available for use in logical expressions.

      Note: Property References are restricted to only "one node upstream". To pass a variable through multiple nodes, you can define multiple Property Reference variables.

      TaskA (X) TaskB (TaskA.X) = (Y) TaskC (TaskB.Y) = (Z)

      For example, if there were a variable (X) in TaskA that you'd like to use in TaskC, you would first create a Property Reference variable (Y) in TaskB. Then, you'd need to create another Property Reference variable (Z) in TaskC, referencing Y.

    • External - Available in the External Start Event node. References system level variables that enable specific functionality.

      External type variables:

      Name

      Purpose

      commentLink

      Used to access a Dashboard Control's system generated comment link. Useful when constructing Custom Notification Processes, with a Notify Task.

      thresholdResults

      Used to access a JSON object representing a Dashboard Control's threshold violations. Useful when constructing Custom Notification Processes, with a Notify Task.

  5. Select a Data Type and enter a Value for the new variable.
  6. Click OK.

Creating variables for use in an analysis

If a process model is to be executing an Analysis, you can access variables that exist in the process model within the Analysis by using the RUNTIME function.

For example, if you wanted access to the workId of the Analysis, you could create a new column within the Analysis using the expression RUNTIME('workId'). Provided the Analysis was being run by a process model, the new column would then populate with the workId of the Analysis at runtime. See Runtime Properties.

Execution result codes

The following numeric codes represent the execution status of a Data Stage. They can be used in process model expressions, to create the Model's logic:

 

Result code

Status

0

Succeeded - Indicates that the Data Stage has executed without error.

1

Failed - Indicates that something has caused the execution to fail. More details should be available in the Result Detail field.

9

Lease Expired - Data360 DQ+'s execution engine features an Expired Lease Job, which checks for inactivity due to random error from machine or process crashes.

If an execution has crashed, it will be marked with a Status of "Lease Expired" and a Result Code of 9.

8

Terminated - Applied to an execution that either crashed or was manually terminated by a user.

If the Expired Lease Job finds a crashed Data Stage execution and that Data Stage is part of a process model, the Process Model will be Terminated.

Execution result types

Alternatively, result type strings can be used in Process Model expressions. The following result types are available for use:

Result Type

Description

'SUCCESS'

Indicates that the Data Stage has executed without error.

'COMPLETED_WITH_ERRORS'

Indicates that the Data Stage has executed successfully but that minor errors occurred. This code can usually be safely ignored.

'FAIL'

Something has caused the execution to fail. More details should be available in the Result Detail field.

'TERMINATE'

Applied to an execution that either crashed or was manually terminated by a user.