Recon Check - Data360_DQ+ - Latest

Properties

Input Mapping

When inputs are connected to the Recon Check node, their display names and field names appear within the Input Mapping tab. You can change source field names in this tab, but note that they must be unique per Recon Check node, that is, no two inputs can have the same name.

Map Source Fields

With the Map Source Fields button, you will need to select the fields that you want to group by and map these fields between inputs.

For example, if you had three inputs and you wanted to perform a recon check that grouped by an id field, you would need to select the id field from each input and map it to the id fields in all other inputs.

 

 

For example, Input1, Input2, and Input3 contain the following fields and values:

input1 data set

input2 data set

input3 data set

Once we have grouped by and mapped id fields, the Recon Check node will restructure the three incoming data sets as follows, creating array fields for each input, grouped by the chosen grouping field:

Validate tab - Adding recon check matches

Once fields are mapped and grouping fields are chosen, recon check matches can be created to evaluate the values of records (contained in an array) associated to each group.

• One to One - For every one value on the left there is one value on the right. Values need not match. For example, [{001:001}] and [{001:002}] are both one to one.
• One to Many - For every one value on the left there are multiple values on the right. Values need not match. For example, [{001:001, 001}] and [{001:002, 003}] are both one to many.
• Many to One - Multiple values on the left, one value on the right. Values need not match. For example, [{001, 001: 001}] and [{002, 003:001}] are both many to one.
• Many to Many - Multiple values on the left, multiple values on the right. Values need not match. For example, [{001, 001: 001, 001}] and [{002, 003:004, 005}] are both many to many.
• Accumulate - Used to accumulate a set of values on the left and to accumulate a set of values on the right, and then check to see if accumulations match.

Accumulate example

Consider the following left and right inputs, where inputs have been mapped by id:

Left data set

Right data set

The following Accumulate Recon Checks could be created to produce results as shown.

Left Field Name: leftLeft Condition: {left}.value = 200 Left Expression: {left}.valueLeft Computation: meanRight Field Name: rightRight Condition: NoneRight Expression: {right}.valueRight Computation: meanResult: True. On the left, the Recon node will find one value that matches the Left Condition. For that one value, it will use the Left Expression in a mean calculation, resulting in 200/1.On the right, the Recon node will find three values that match the Right Condition. For those three values, it will use the Right Expression in a mean calculation, resulting in 600/3.Since 200/1 = 600/3, the check evaluates to True.

Match - Used to match left and right values. Match condition allows you to specify a condition to match on. Only values that satisfy that condition will be considered when matching. Match expression allows you to specify an expression to manipulate values before match consideration is performed.

Match example

Consider the following left and right inputs, where inputs have been mapped by id:

Left data set

Right data set

The following Match Recon Checks could be created to produce results as shown.

Match Condition: {Left}.value = 200Match Expression: {Left}.value = {Right}.valueResult: True. Only the Left record with value = 200 will be considered, against all values on the right.

Match Condition: NoneMatch Expression: {Left}.value = {Right}.valueResult: False. Since no Match condition is specified, all values on the left are compared to all values on the right - and not all of these values match.

Match Condition: {Left}.value = 100Match Expression: {Left}.value * 2 = {Right}.valueResult: True. Only the Left record with value = 100 will be considered, against all values on the right. Before the left value is considered, however, the Match Expression multiplies it by 2, making it 200.

Recon Check Match Name

When creating new recon check matches, you need to give them a name. This name will be the name of the field that contains Boolean results, indicating if a record has passed the recon check. Records that return True are those that conform to the chosen Match Type. Records that return False do not conform to the chosen Match Type.

For each recon check match rule, "_left", "_right" and "_diff" output fields are created. The left and right fields contain the accumulated value for the left and right side of the comparison. The diff field contains the difference between the two values (left - right). The output field names follow this format: <recon_check_match_name>_left, <recon_check_match_name>_right and <recon_check_match_name>_diff.