Processing a Large Number of Records - trillium_discovery - trillium_quality - Latest

Trillium Control Center

Product type
Software
Portfolio
Verify
Product family
Trillium
Product
Trillium > Trillium Discovery
Version
Latest
Language
English
Product name
Trillium Quality and Discovery
Title
Trillium Control Center
Copyright
2024
First publish date
2008
Last updated
2024-10-18
Published on
2024-10-18T15:02:04.502478

When you are processing a large input entity where the number of records exceeds 99,999,999, the values in some of the output attributes are truncated and warning messages are generated for each attribute where the truncation occurred. This is due to the schema length of the attributes being set to eight digits.

Note: To find how many records (rows) there are in the input entity, open the Metadata Summaries for the entity and select About > Source Properties > Row Count.

The following table lists the output attributes that are truncated for 100,000,000 (100 million) or more records.

Output Attribute

Length (default)

Lev1 Matched

8

Lev1 Suspect

8

Lev2 Matched in Lev1 Matched

8

Lev2 Suspect in Lev1 Matched 8
Lev2 Matched in Lev1 Suspect 8
Lev2 Suspect in Lev1 Suspect 8
Record Number 8
Window Number 8

You can use the following procedures to increase the length of the attributes to avoid truncation.

To change the length of attributes in the Relationship Linker/Reference Matcher

  1. From the Navigation or Project View, right-click the Relationship Linker or Reference Matcher process and select Edit Process.
  2. Click the Schema Editor tab.
  3. In the Schema Editor, right-click one of the output attributes listed in the table above and select Edit.
  4. Specify the length of the attribute in the Max width box. For example, specify 9 or greater if the input entity contains 100,000,000 records.
  5. Repeat Step 3 and 4 for all other attributes in the table.
  6. Click OK to close the Edit Output Attribute window.
  7. Click Finish to save the schema changes.
  8. From the Navigation or Project View, right-click the process and select Apply Schema Changes.
  9. When the message box appears, click OK. The schema changes will be propagated to all downstream processes in the flow.
  10. Run the Relationship Linker or Reference Matcher.