API and script bindings

For both the ConfigureFields and ProcessRecords Python scripts the following variables are available:

Variable name	Type	Descripton
node	Node	The node object used for logging and failure control, properties access and explicit record writing.
inputs	When used in ConfigureFields , "inputs" is a collection of Metadata. When used in ProcessRecords , "inputs" is a collection of Record objects.	Please note that in the Transform node these lists will only contain one item as there is only one input. Access via: `inputs['InputName']` or `inputs[inputIndex]`
outputs	When used in ConfigureFields, "outputs" is a collection of MetadataBuilder objects. When used in ProcessRecords, "outputs" is a collection of output Record objects.	Access via: `outputs['InputName']` or `outputs[inputIndex]`
patterns	Pattern	Used in ConfigureFields to build patterns to search for input fields. Can also be used in record key properties (i.e. in the Advanced tab of the Match Keys, GroupBy and SortBy properties).
fields	When used in ConfigureFields, "fields" contains the metadata for all of the fields in the inputs. In the Transform and Aggregate nodes, this is just one input, but in other nodes, this would contain references to all the fields against all inputs (unless explicitly restricted to a particular input). When used in ProcessRecords, "fields" references the fields on input records, as opposed to field metadata.	Access the field metadata using `fields['fieldName']` or `fields.fieldName`. In this example, the metadata for the input field "fieldName" is referenced. `fields.todict()` can be used to construct a dictionary representation of the fields. If there are multiple fields with the same name across the inputs to the node, any attempt to reference that field directly, or to use the `todict()` function will raise an error due to an ambiguous field reference. Otherwise, the key is the name of the field, and the value is the field metadata (in ConfigureFields) or the value of the field on the input record (in ProcessRecords).
fn	Null-safe functions module.	A set of Null-safe functions for data comparisons and string manipulation.
group	Grouping module.	A set of functions for easy aggregation of data. Note: The aggregation functions are only applicable to nodes which define some form of data grouping. For example, they can be used in the GroupBy property of the Aggregate and Transform nodes. The aggregation functions are not available on nodes that do not have a GroupBy property, including the Merge node.

In addition, each input and output will be directly bound as a variable available within the Python script (e.g. in1, out1), as long as the input or output name meets the following criteria:

Is a valid python identifier (e.g. does not contain spaces).
Does not conflict with any of the other bound variables (node, inputs, outputs, patterns).
Does not conflict with any Python built-in type or keyword (e.g. print, raise, str, int).
Does not conflict with an imported module provided with Data360 Analyze.

If the input or output does not meet the above criteria, it must be accessed from the corresponding inputs or outputs collection, respectively.

The bound input and output variables will have the same type as the corresponding entry in the inputs/outputs collection – Metadata/Record for input in ConfigureFields/ProcessRecords respectively, and MetadataBuilder/Record for output in ConfigureFields/ProcessRecords respectively.

Each of these variables will be bound into ConfigureFields and then bound into ProcessRecords prior to each invocation of the ProcessRecords script. The scope of the local variable space is shared between the script in ConfigureFields and ProcessRecords. This means that any variable, function definition or import made within ConfigureFields will be available for use within the ProcessRecords script so long as the name does not conflict with one of the variables that are directly bound into ProcessRecords. This means, for example, that you can define a local variable "myVariable" in ConfigureFields then use that variable in the ProcessRecords script. However, if there is an input or output on the node named "myVariable", then the variable will be overwritten by the input/output bound into ProcessRecords.

For more information on the different object types, see:

Node objects
Metadata objects
FieldMetadata objects
MetadataBuilder objects
Record objects
Logger objects
Properties objects
Patterns objects
FieldPattern objects

Node objects

Members

properties

Used to access node properties.
Type: Properties

logger

Used to write messages to the node log.
Type: Logger

firstExec

Returns True for the first time the ProcessRecords script is invoked (on the first input record), thereafter returns False.
Type: bool

lastExec

Returns True for the last time the ProcessRecords script is invoked (on the last input record), otherwise returns False.
Type: bool

firstInGroup

Returns True for each time the ProcessRecords script is invoked where the record is the start of a new group – based on the GroupBy property otherwise returns False. If no GroupBy is specified, this will be the same as firstExec.
Type: bool

lastInGroup

Returns True for each time the ProcessRecords script is invoked where the record is the end of a group – based on the GroupBy property otherwise returns False. If no GroupBy is specified, this will be the same as lastExec.
Type: bool

execCount

Returns a count of the current record number in the input data set.
Type: long

Methods

write(outputName|outputIndex, outputRecord)

Writes the given record to the specified node output.
The output must be specified as the name (str) of the output or the index (int) of the output.
Arguments:
outputName Type: str

outputIndex Type: int

outputRecord Type: Record
Return type: None

fail()

Marks the node as failed.
For readability, recommended use is: raise node.fail()
Indicates to the node that an error has occurred, and all necessary logging has been performed. No additional entries will be written to the log. Therefore, recommended usage is to first use node.logger.error(<message>) prior to calling node.fail()
Raises an exception. Also declared to throw the Exception.
Return type: Exception

Metadata objects

Metadata objects are a container of FieldMetadata.

The metadata object largely emulates a Python dict type with the key being the name of the field and the value, the field metadata.

The __len__, __getitem__, __iter__, __reversed__ and __contains__ functions are implemented on the Metadata type as defined in the Python documentation available at: https://docs.python.org/2/reference/datamodel.html#emulating-container-types* .

This means that along with being able to use the reversed and contains methods directly, the following operations are supported from the Python dict interface as defined in https://docs.python.org/2/library/stdtypes.html#mapping-types-dict*:

len(d)
d[key]
key in d
key not in d
iter(d)

Members

Each field on the metadata is accessible directly as a member as long as it does not contain spaces.
The field metadata can be used in the += and -= operators on a MetadataBuilder to construct the output metadata. In addition, FieldMetadata can be directly assigned to an output field on a metadata builder, for example by using the following syntax: out1.newField = in1.inputField
Type: FieldMetadata

The Metadata itself is also just a collection of fields such that individual fields can be accessed using either of the following forms:

metadata['fieldName']
metadata[fieldIndex]

Methods

todict()

Constructs and returns a new dictionary representing the Metadata.
The keys in the dictionary are the field names.
The values in the dictionary are the corresponding FieldMetadata objects.
Return type: dictionary

intersection(other)

Generally, the intersection method is only useful on nodes with multiple inputs (e.g. the Merge node).
The "other" argument must be either a Metadata or a MetadataBuilder object.
When a MetadataBuilder object is provided as the "other" argument, the intersection operation operates on all of the output fields which exist on the MetadataBuilder at that time.
A field within this object is considered to be part of the intersection if there is a field in the "other" object with the same (case-insensitive) field name.
The list of FieldMetadata returned can be used to update an output MetadataBuilder or provided as an argument to the rename method on FieldPattern objects for renaming purposes.
Returns a list of FieldMetadata objects containing all of the fields in this metadata which are also in the provided object.
Arguments:
other Type: Metadata or MetadataBuilder
Return type: list
Example:

#Add all of the fields from the input 'leftInput' to the output 'matches'

matches += leftInput

#Locate all fields in input 'rightInput' which are also in 'leftInput'

conflicts = rightInput.intersection(leftInput)

#Add these fields to the "matches" output, renaming them to have the prefix #"Right."

matches += patterns.all(conflicts).rename('Right.$0')

difference(other)

Generally, the difference method is only useful on nodes with multiple inputs (e.g. the Merge node).
The "other" argument must be either a Metadata or a MetadataBuilder object.
When a MetadataBuilder object is provided as the "other" argument, the difference operation operates on all of the output fields which exist on the MetadataBuilder at that time.
A field within this object is considered to be part of the intersection if there is a field in the "other" object with the same (case-insensitive) field name.
The list of FieldMetadata returned can be used to update an output MetadataBuilder or provided as an argument to the rename method on FieldPattern objects for renaming purposes.
Returns a list of FieldMetadata objects containing all of the fields in this metadata which are not in the provided object.
Arguments:
other Type: Metadata or MetadataBuilder
Return type: list
Example:

#Add all of the fields from the input 'leftInput' to the output 'matches'

matches += leftInput

#Add all of the fields from the input 'rightInput' which are not on the #'leftInput' to the output 'matches'

matches += rightInput.difference(leftInput)

FieldMetadata objects


Methods	name() Returns the name of the field. Return type: str type() Returns the type of the field (for example, str). Return type: Python type

Methods

name()

Returns the name of the field.
Return type: str

type()

Returns the type of the field (for example, str).
Return type: Python type

MetadataBuilder objects

The metadata builder is used for building an output metadata.

The metadata builder object largely emulates a Python dict type with the key being the name of the field and the value, the field metadata.

The __len__, __getitem__, __iter__, __reversed__, __setitem__, __delitem__ and __contains__ functions are implemented on the Metadata type as defined in: https://docs.python.org/2/reference/datamodel.html#emulating-container-types*

len(d)
d[key]
d[key] = value
del d[key]
key in d
key not in d
iter(d)

The node takes care of constructing the output metadata from the builder, therefore there are generally no members and only a few methods required on the metadata builder. However, there are various useful operations, as follows:

Operations


Operations	`metadataBuilder = metadataBuilder + obj` `metadataBuilder += obj` Configures the MetadataBuilder to add all of the fields from the provided `obj` to the output metadata. Allowable `obj` types are: Metadata – uses all fields from the provided input metadata `out1 += in1` String – treats the string as a pattern and includes all input fields matching that pattern `out1 += 'Account'` FieldMetadata – includes the provided field `out1 += in1.Account` FieldPattern (wildcard/regex/all pattern) adds all input fields matching the pattern, with optional renaming of the output fields via a rename pattern `out1 += patterns.regex('.?(due\|present).*?Date')` `out1 += patterns.all(in2).rename('Right.\g<0>')` List – adds all elements from within the list type provided, where each element must be a type that can be appended to the metadata builder `out1 += [in1.name, in1.address]` `out1 += in1.difference(in2)` Dictionary– adds each of the elements of the dictionary, where the key must be the output field name to add, and the value must be either a FieldMetadata or a type object `out1 += {'newField' : str, 'newField2' : in1.inputField}` Note: When using the `+=` operator, an error will be raised if any of the fields being added to the output would have the same (case-insensitive) output field name as one which already exists.When adding a new field, if a field with the same (case-insensitive) name already exists, to remove the existing field and add the new field, explicitly name the new field by using the following syntax: `out1.<fieldname> = obj` `metadataBuilder = metadataBuilder - obj` `metadataBuilder -= obj` Configures the MetadataBuilder to subtract all of the fields from the provided `obj` to the output metadata. The allowable types and operation follows the same rules for the '+' and '+=' cases outlined above. `metadata.<name> = type` Configures the MetadataBuilder to add a new field with the specified name and type `out1.newField = str` `metadata.<name> = FieldMetadata` Sets up the mapping from the provided input field metadata to the new output field `out1.newField = in1.oldField`
Methods	todict() Constructs and returns a new dictionary representing the MetadataBuilder. The keys in the dictionary are the field names. The values in the dictionary are the corresponding FieldMetadata objects. Return type: dictionary

metadataBuilder = metadataBuilder + obj

metadataBuilder += obj

Configures the MetadataBuilder to add all of the fields from the provided obj to the output metadata.
Allowable obj types are:
- Metadata – uses all fields from the provided input metadata
  - out1 += in1
- String – treats the string as a pattern and includes all input fields matching that pattern
  - out1 += 'Account*'
- FieldMetadata – includes the provided field
  - out1 += in1.Account
- FieldPattern (wildcard/regex/all pattern) adds all input fields matching the pattern, with optional renaming of the output fields via a rename pattern
  - out1 += patterns.regex('.*?(due|present).*?Date')
  - out1 += patterns.all(in2).rename('Right.\g<0>')
- List – adds all elements from within the list type provided, where each element must be a type that can be appended to the metadata builder
  - out1 += [in1.name, in1.address]
  - out1 += in1.difference(in2)
- Dictionary– adds each of the elements of the dictionary, where the key must be the output field name to add, and the value must be either a FieldMetadata or a type object
  - out1 += {'newField' : str, 'newField2' : in1.inputField}

Note: When using the += operator, an error will be raised if any of the fields being added to the output would have the same (case-insensitive) output field name as one which already exists.When adding a new field, if a field with the same (case-insensitive) name already exists, to remove the existing field and add the new field, explicitly name the new field by using the following syntax: out1.<fieldname> = obj

metadataBuilder = metadataBuilder - obj

metadataBuilder -= obj

Configures the MetadataBuilder to subtract all of the fields from the provided obj to the output metadata.
The allowable types and operation follows the same rules for the '+' and '+=' cases outlined above.

metadata.<name> = type

Configures the MetadataBuilder to add a new field with the specified name and type
- out1.newField = str

metadata.<name> = FieldMetadata

Sets up the mapping from the provided input field metadata to the new output field
- out1.newField = in1.oldField

Methods

todict()

Constructs and returns a new dictionary representing the MetadataBuilder.
The keys in the dictionary are the field names.
The values in the dictionary are the corresponding FieldMetadata objects.
Return type: dictionary

Record objects

Record objects are a container of field values.

The record object largely emulates a Python dict type with the key being the name of the field and the value, being the value for that field on the record.

The __len__, __getitem__, __iter__, __reversed__, __setitem__ and __contains__ functions are implemented on the Metadata type as defined in: https://docs.python.org/2/reference/datamodel.html#emulating-container-types*

len(d)
d[key]
d[key] = value
key in d
key not in d
iter(d)

Members


Members	<fieldName> The value of each field on the record is accessible directly as a member as long as it does not contain spaces. The Record itself is also just a collection of field values such that individual field values can be accessed using either of the following forms: `record['fieldName']` `record[fieldIndex]`
Operations	`record = record + obj` `record += obj` Operation only valid on output records. Input records are read-only. Configures the record to add all the field values from the provided `obj`. Allowable `obj` types are: Record – sets all of the fields on the output record from the provided input record. The mapping from the input corresponding to the specified record must have been defined in the metadata builder for the output corresponding to this record `out1 += in1` Dictionary – adds each of the elements of the dictionary, where the key must be the name of the output field and the value is the value to set. All of the specified fields must have been setup in the metadata builder for the output corresponding to this record `out1 += {'myStrField' : 'foo', 'myIntField' : 3}` `record.fieldName = value` `record['fieldName'] = value` `record[fieldIndex] = value` Sets the specified field (via index or name) to the provided value.
Methods	metadata() Returns the metadata for the record. Return type: Metadata todict() Constructs and returns a new dictionary representing the Record. The keys in the dictionary are the field names. The values in the dictionary are the corresponding field values on the Record. Return type: dictionary

The value of each field on the record is accessible directly as a member as long as it does not contain spaces.

The Record itself is also just a collection of field values such that individual field values can be accessed using either of the following forms:

record['fieldName']
record[fieldIndex]

Operations

record = record + obj

record += obj

Operation only valid on output records. Input records are read-only.
Configures the record to add all the field values from the provided obj.
Allowable obj types are:
- Record – sets all of the fields on the output record from the provided input record. The mapping from the input corresponding to the specified record must have been defined in the metadata builder for the output corresponding to this record
  - out1 += in1
- Dictionary – adds each of the elements of the dictionary, where the key must be the name of the output field and the value is the value to set. All of the specified fields must have been setup in the metadata builder for the output corresponding to this record
  - out1 += {'myStrField' : 'foo', 'myIntField' : 3}

record.fieldName = value

record['fieldName'] = value

record[fieldIndex] = value

Sets the specified field (via index or name) to the provided value.

Methods

metadata()

Returns the metadata for the record.
Return type: Metadata

todict()

Constructs and returns a new dictionary representing the Record.
The keys in the dictionary are the field names.
The values in the dictionary are the corresponding field values on the Record.
Return type: dictionary

Logger objects

Methods

debug(message)

Writes a debug message to the node log.
Arguments:
message Type: str or unicode
Return type: None

info(message)

Writes an info message to the node log.
Arguments:
message Type: str or unicode
Return type: None

warn(message)

Writes a warning message to the node log.
Arguments:
message Type: str or unicode
Return type: None

error(message)

Writes an error message to the node log.
Arguments:
message Type: str or unicode
Return type: None

Properties objects

Methods

All property methods work by taking both the Run Time Property name of the property and the property name.

The node retrieves the property via the Run Time Property name, however any errors on property retrieval are reported against the (often more user friendly) property name.

isSet(propName, runtimePropName)

Returns (bool) whether or not the specified property is set.
Arguments:
propName Type: str or unicode

runtimePropName Type: str or unicode
Return type: bool

getString(propName, runtimePropName[, default])

Returns the string value of the specified property.
If the property does not exist and no default is provided an error is raised, otherwise the default is returned.
Arguments:
propName Type: str or unicode

runtimePropName Type: str or unicode

default Type: str or unicode
Return type: str

getInt(propName, runtimePropName[, default])

Returns the integer value of the specified property.
If the property does not exist and no default is provided an error is raised, otherwise the default is returned.
If the property exists but is not a valid integer, an error is raised.
Arguments:
propName Type: str or unicode

runtimePropName Type: str or unicode

default Type: int
Return type: int

getBool(propName, runtimePropName[, default)

Returns the boolean value of the specified property.
If the property does not exist and no default is provided an error is raised, otherwise the default is returned.
This will treat any value matching "true" case-insensitively as True. Any other value will be treated as False.
Arguments:
propName Type: str or unicode

runtimePropName Type: str or unicode

default Type: bool
Return type: bool

Patterns objects

Methods

The Patterns object has simple methods for constructing and returning FieldPattern objects which can then be provided to the MetadataBuilder+= and -= operators to include or exclude input fields that match the pattern.

The Patterns object can also be used in record key properties (in the Advanced tab of the Match Keys, GroupBy and SortBy properties). This allows you to specify multiple input fields on which to join, group or sort the input data based on fields that match the pattern. For example, in the Aggregate node you can specify a wildcard pattern in the Advanced tab of the GroupBy property to group the input data by all fields that match the pattern. Note: If a pattern is specified it must match at least one input field name.

wildcard(pattern[, metadata])

Constructs and returns a FieldPattern which matches fields in the specified input metadata – or across all input metadata if no metadata is provided - using the provided wildcard pattern.
The "pattern" argument must be a string pattern and must be a valid wildcard pattern as can be used by the Python "fnmatch" module.
The "metadata" argument must be a Metadata, MetadataBuilder object or a list of FieldMetadata objects.
Arguments:
pattern Type: str

metadata Type: Metadata, MetadataBuilder or FieldMetadata
Return type: FieldPattern
Example:

#Add all fields from input 'in1' which are prefixed with 'Total' to output #'out1'

out1 += patterns.wildcard('Total*', in1)

all([metadata])

Constructs and returns a FieldPattern to match all fields in the specified input metadata – or match all input fields across all input metadata if no metadata argument is provided.
The "metadata" argument must be a Metadata, MetadataBuilder object or a list of FieldMetadata objects.
Arguments:
metadata Type: Metadata, MetadataBuilder or FieldMetadata
Return type: FieldPattern
Example:

#Add all fields from all inputs

out1 += patterns.all()

regex(pattern[, metadata])

Constructs and returns a FieldPattern which matches fields in the specified input metadata – or across all input metadata if no metadata is provided - using the provided regular expression.
The "pattern" argument must be a valid string regular expression pattern as could be used in the Python "re" module.
The "metadata" argument must be a Metadata, MetadataBuilder object or a list of FieldMetadata objects.
Arguments:
pattern Type: str

metadata Type: Metadata, MetadataBuilder or FieldMetadata
Return type: FieldPattern
Example:

#Add all fields from input 'in1' which have 'Total' anywhere in their name #followed by a whitespace character

out1 += patterns.regex('.*?Total\\s.*', in1)

FieldPattern objects

Methods

A FieldPattern object is returned as the result of the various method calls available on the Patterns object. FieldPattern objects are generally used as arguments to the MetadataBuilder+= and -= operators to include or exclude input fields that match the pattern, and can also be used in record key properties (in the Advanced tab of the Match Keys, GroupBy and SortBy properties). The FieldPattern object also allows for pattern renaming.

rename(renamePattern)

Augments and returns this FieldPattern object by adding renaming functionality using the specified renamePattern.
The "renamePattern" argument must be a standard Python string renaming pattern – as could be used by the "sub" method in the "re" module. For example, the '\g<0>' rename pattern will return the entire matched string.
The resulting FieldPattern object can be used in MetadataBuilder+= operations to add all of the fields matching the pattern used to construct this FieldPattern object, renamed on the output metadata using the provided renamePattern, and can also be used in record key properties.
Arguments:
renamePattern Type: str
Return type: FieldPattern
Example:

#Add all fields on input 'in1' starting with the string 'Total' renamed to #replace

#'Total' with 'SalesAmount' on the output 'out1'

out1 += patterns.regex('^(Total)(.*)', in1).rename('SalesAmount\2')

Tip: Standard Python regular expressions are available via the "re" module.

(* links correct at time of publishing).

Null handling

All Null fields on data records are bound in as special Null objects in Python — not the Python None value.

To check if a value is Null, use the following syntax:


        if in1.MyField is Null:

Non-ASCII characters

When working with the Python-based nodes, it is important to be aware of the following Python language notation if your data contains non-ASCIIcharacters:

Tip: Python literals which include non-ASCII characters should be prefixed with a 'u' character to convert the string to a Unicode string.For example, if you had the field name 'colör' as input to a Transform node, the following code would cause the node to fail:out1.color = str(fields['colör'])Instead, you would need to use the 'u' character prefix to convert the colör field to a Unicode string:out1.color = str(fields[u'colör'])

API and script bindings - Latest

Data360 Analyze Server Help

Node objects

Metadata objects

FieldMetadata objects

MetadataBuilder objects

Record objects

Logger objects

Properties objects

Patterns objects

FieldPattern objects

Null handling

Non-ASCII characters