Recon Check - Data360_DQ+ - Latest

Data360 DQ+ Help

Product type
Software
Portfolio
Verify
Product family
Data360
Product
Data360 DQ+
Version
Latest
Language
English
Product name
Data360 DQ+
Title
Data360 DQ+ Help
Copyright
2024
First publish date
2016
ft:lastEdition
2024-07-09
ft:lastPublication
2024-07-09T15:09:58.774265

The Recon Check node allows you to reconcile records from up to four sources, using a number of different validation options.

Properties

Input Mapping tab

When inputs are connected to the Recon Check node, their display names and field names appear within the Input Mapping tab. Source Field names may be changed within this tab, but note that they must be unique per Recon Check node, that is, no two inputs can have the same name.

Map Source Fields

With the Map Source Fields button, you will need to select the fields you want to group by and map these fields between inputs.

For example, if you had three inputs and you wanted to perform a recon check that grouped by an id field, you would need to select the id field from each input and map it to the id fields in all other inputs.

 

Recon Check node

 

For example, Input1, Input2, and Input3 contain the following fields and values:

input1 data set

input1ID

measure

001

100

001

101

001

102

input2 data set

input2ID

measure

001

100

001

101

001

120

input3 data set

input3ID

measure

001

100

001

101

001

102

Once we have grouped by and mapped id fields, the Recon Check node will restructure the three incoming data sets as follows, creating array fields for each input, grouped by the chosen grouping field:

input1ID

input1

input2

input3

001

[{measure=100},

{measure=101},

{measure=102}]

[{measure=100},

{measure=101},

{measure=120}]

[{measure=100},

{measure=101},

{measure=102}]

Validate tab - Adding recon check matches

Once fields are mapped and grouping fields are chosen, recon check matches can be created to evaluate the values of records (contained in an array) associated to each group.

  • One to One - For every one value on the left there is one value on the right. Values need not match. For example, [{001:001}] and [{001:002}] are both one to one.
  • One to Many - For every one value on the left there are multiple values on the right. Values need not match. For example, [{001:001, 001}] and [{001:002, 003}] are both one to many.
  • Many to One - Multiple values on the left, one value on the right. Values need not match. For example, [{001, 001: 001}] and [{002, 003:001}] are both many to one.
  • Many to Many - Multiple values on the left, multiple values on the right. Values need not match. For example, [{001, 001: 001, 001}] and [{002, 003:004, 005}] are both many to many.
  • Accumulate - Used to accumulate a set of values on the left and to accumulate a set of values on the right, and then check to see if accumulations match.

Accumulate example

Consider the following left and right inputs, where inputs have been mapped by id:

Left data set

id

value

001

100

001

200

001

300

Right data set

id

value

001

200

001

200

001

200

The following Accumulate Recon Checks could be created to produce results as shown.

Left Field Name: leftLeft Condition: {left}.value = 200 Left Expression: {left}.valueLeft Computation: meanRight Field Name: rightRight Condition: NoneRight Expression: {right}.valueRight Computation: meanResult: True. On the left, the Recon node will find one value that matches the Left Condition. For that one value, it will use the Left Expression in a mean calculation, resulting in 200/1.On the right, the Recon node will find three values that match the Right Condition. For those three values, it will use the Right Expression in a mean calculation, resulting in 600/3.Since 200/1 = 600/3, the check evaluates to True.

Match - Used to match left and right values.Match condition allows you to specify a condition to match on. Only values that satisfy that condition will be considered when matching.Match expression allows you to specify an expression to manipulate values before match consideration is performed.

Match example

Consider the following left and right inputs, where inputs have been mapped by id:

Left data set

id

value

001

100

001

200

001

300

Right data set

id

value

001

200

001

200

001

200

The following Match Recon Checks could be created to produce results as shown.

Match Condition: {Left}.value = 200Match Expression: {Left}.value = {Right}.valueResult: True. Only the Left record with value = 200 will be considered, against all values on the right.

Match Condition: NoneMatch Expression: {Left}.value = {Right}.valueResult: False. Since no Match condition is specified, all values on the left are compared to all values on the right - and not all of these values match.

Match Condition: {Left}.value = 100Match Expression: {Left}.value * 2 = {Right}.valueResult: True. Only the Left record with value = 100 will be considered, against all values on the right. Before the left value is considered, however, the Match Expression multiplies it by 2, making it 200.

Recon Check Match Name

When creating new recon check matches, you will need to assign them a name. This name will be the name of the field that contains Boolean results representing whether a record has passed the recon check. Records that return True are those that conform to the chosen Match Type; records that return False are records that do not.