Script - Data360_DQ+ - Latest

Properties

Outputs

Use the Outputs tab to specify the fields that will appear in the node output.

To add a field to the output, Click Add Field.

To use the fields from the input node, select Input Fields, and use the Field Selector to add input fields to the output.

Within your script, Input fields can be referred to using the following syntax:

input.fieldName

To add a new field, select New Field, enter a valid field name , select the field output type, and click Update.

To use a New Field in your script, use the following syntax:

output.fieldName

Script Input

The Script Input tab displays the name and type of the fields from the input node.

To edit the name or type of an input field, double-click in the row for the field. Click Update to save your changes.

Script Output

The Script Output tab displays the name and type of the fields that will be included in the node output.

To edit the name or type of an output field, double-click in the row for the field. Click Update to save your changes.

Script Rules

You can use Script Rules from Rule Libraries in your Script node. You can use functions in your script from any rules that you add.

Complete the following steps to add a Script Rule.

1. Select the Script Rules tab.
2. Click Add.
3. Select the Rule Library, Rule Group, and Rule by using the Add Script Rule dialog.

JavaScript processing time limit

To prevent infinite loops from occurring, Data360 DQ+ has a time limit for the processing of JavaScript in the Script node. The maximum time is 60 seconds for each top-level record. Note that complex records that include nested arrays need to be processed in this time irrespective of the number of records in the arrays.

Using JavaScript

Click Edit Script to open the Java Script Code Editor dialog to create or update the JavaScript that will be used to generate the node output.

Passing Input Fields through the Script node

When working with the Script node, you need to include all fields that you wish to output from it in your Script.

If there are fields from an incoming node that you need to "pass through" to the output without transforming them, you need to select these fields as Input Fields and then use the following statement in your script:

output.fieldName = input.fieldName;

This statement should be repeated for each input field you need to pass through. Also note that for fields passed using the statement above, you do not need to create Output fields by using the Add Field drop down.

Passing all fields

Another way to pass fields from an input node into a JavaScript node is with the copyInputsToOutputs() function available within the Script editor.

To pass all fields from an input node into a JavaScript node, place the following line at the top of your script:

copyInputsToOutputs();

Using copyInputsToOutputs() is effectively the same as writing output.fieldName = input.fieldName for every field from your input node.

Passing specific fields with a RegEx Parameter

You can use copyInputsToOutputs() with a regex parameter to pass specific fields. Providing a parameter tells the function to test whether field names match the regex. Only field names that match the regex are passed.

For example, if you input fields with the names YearStarted, YearEnded, and MonthRenewed, placing the following in your script would only create outputs for the first two fields:

var r = new RegExp(‘Year\\S+');
copyInputsToOutputs(r);

For more information about RegExp syntax in JavaScript, see the JavaScript developer documentation.

Passing fields with a Join

You can pass fields through a Script node by using a Join node.

In this setup, you would only need to include new output fields and a field to Join on in your script. You could then Join the Script node's outputs with fields from your original Data Store.

 

This type of setup could come in useful if you have a large number of fields to pass through a Script node, and you don't necessarily have the patience to include each one in your script.

Equality type when comparing variables in Scripts (=== vs. ==)

When comparing Fields and Variables in the Script node, it is important to consider data and equality types you are using.

For Strict equality, use the === operator.For Lenient equality, use the == operator.

Binding Functions

In addition to the data transformation functions available throughout Data360 DQ+, the JavaScript node also features some Binding Functions to help you move or change Fields and Records.

copyInputsToOutputs()

Passes fields from an input node to the JavaScript node.

dropRecord()

Removes records from a data set and is best combined with conditional logic.

For example, given the following input data set.

You could remove the third record from this data set using the following in your script:

if(input.topic == ‘hangGliding') {
      dropRecord();
}

The output would contain two rows:

emitRecord()

Can be used to change the field values of records.

Input parameters are the field names and values to write to them.For example, consider the following data set once again.

You could change values in the third record of this data set using the following in your script:

if(input.rank<=50) {    emitRecord(output.topic = ‘classification', output.rank = 95);}

Within the Script node, the data set would then become:

Using the Script Input and Script Output tab to handle Javascript's Numeric Limitations

When using the Javascript node it is important to be aware of the language's limitations, particularly when dealing with numeric fields.

The maximum numeric integer that can be reliably used in Javascript is 9007199254740991. Any number greater than this may potentially be truncated when using the Javascript node.

To avoid truncation of field values greater than 9007199254740991, you will need to tell the Javascript node to treat a field's values as Strings. To do so, use the Javascript node's Script Input and Script Output tabs to mark the field as a String type. If you would still like to treat the field as an Integer or Big Integer outside of the Javascript node, you can mark the field as such in the Javascript node's Outputs tab.