When working with data stores, the Transform tab allows you to perform custom transformations on your source data using scripts written using JavaScript. With data stores, one useful application of the Transform tab's scripting is the parsing of a single field into multiple, new fields.
Script inputs
To work with data store fields within your script, you must first add the fields as Input Fields, using the Add Fields button. Once input fields have been selected, you can then refer to them in your script as input.fieldName.
Input fields
Click Add Field then select Input Fields.
Specifying a field as an input field makes the data store field available for manipulation in the script.
New field
Click Add Field then select New Field.
Creating a New Field creates a field that can be used in your script, to hold parsed data as it is passed from input field to output field.
As a very simple example, consider parsing an input field named line, which you want to output into an output field named line. If you created a New Field named temp, you could do something like this in your script (assuming all fields were strings):
input.temp = input.line.slice(0,2);
output.line = input.temp + " that is a slice";
Doing so would create records in output.line where each record contained the first two characters of each line in line with " that is a slice" concatenated.
Script outputs
Script Outputs are the fields that have been created in the Fields tab of your data store. They are the containers that will hold the outputs of your script.
To use an output field in your script, you can refer to it as output.fieldName.
Preserve source order
Selecting Preserve Source Order will cause the incoming records to remain in the same order as they appear in the data source. If left unchecked, records will be parsed by the script in random order.
Script Controls Emit
Select Script Controls Emit if you want to manually control which records from your data source are emitted by the script, typically via use of the emitRecord() function. Leaving this box unchecked will cause all records from your source to be emitted by the script.
Script limitations
Numeric limitations
When using Javascript it is important to be aware of the language's limitations, particularly when dealing with numeric fields.
The maximum numeric integer that can be reliably used in Javascript is 9007199254740991. Any number greater than this may potentially be truncated when using Javascript.
To avoid truncation of field values greater than 9007199254740991, you will need to tell the Transform script to treat a field's values as strings. To do so, use the Script Inputs and Script Outputs tabs to mark the field as a string type. If you would still like to treat the field as an integer or big integer in the data store, you can mark the field as such in the data store's Fields tab.
JavaScript processing time limit
To prevent infinite loops from occurring, Data360 DQ+ has a time limit for the processing of JavaScript in data stores. The maximum time is 60 seconds for each top-level record. Note that complex records that include nested arrays need to be processed in this time irrespective of the number of records in the arrays.
Script rules tab
Here, you can incorporate existing script rules from rule libraries into your transform script. Doing so will allow you to call functions from the incorporated rules by name, within your script. To incorporate a script rule, you just need to identify its parent Rule Library and Rule Group, and then select the script Rule that you want to use.