Defining Performance Settings - trillium_discovery - trillium_quality - Latest

Trillium DQ Repository Administrator Guide

Product type
Software
Portfolio
Verify
Product family
Trillium
Product
Trillium > Trillium Discovery
Version
Latest
Language
English
Product name
Trillium Quality and Discovery
Title
Trillium DQ Repository Administrator Guide
Copyright
2024
First publish date
2008
Last updated
2024-10-18
Published on
2024-10-18T15:31:00.219841

The following settings in the Repository Manager can have an impact on data import performance:

  • Cache Size

  • Dependency Analysis

  • Key Analysis

  • Statistics Analysis

  • Duplicate Key Discovery

  • Copy Standard Attribute Business Rules (ABRs)

  • Standard ABRs Analysis

Note: They apply to all loader connections in the repository server.

To define or modify performance settings

  1. In the Repository Manager’s Navigation View, right-click Loader Connections and select Edit Loader Settings from the drop-down list. The Edit Loader Settings window opens.

    Edit Loader Settings Window
  2. Review the current cache size setting and make the necessary change. You can set the loader connection cache size between 128 and 2000 MB (2 GB).

  3. Click the other tabs, and modify as necessary using the instructions in the table given below.

    Table 1. Performance Setting Descriptions

    Setting

    Description

    Cache Size

    Specifies the maximum memory, in megabytes, to be used when loading data, analyzing attributes, and creating indexes. Define this setting to be as high as your hardware allows (cache size between 128 and 2000 MB) without relying on virtual memory.

    Note: On multiple CPU systems, the load process can run multiple times in parallel. In these situations, memory is shared across all processes to ensure that it does not exceed the limit you define.

    Run Dependencies at Load Time

    By default, Trillium automatically performs dependency analysis on a sample of your data (10,000 rows) during data import to find possible dependencies.

    Disabling the dependency analysis improves the load performance. To disable this option, clear the Discover Dependencies at Load Time selection.

    Note: If you choose to not run the dependency analysis during  data import, remember to manually run the analysis at a later time from the Control Center.

    Run Keys at Load Time

    By default, Trillium automatically performs key analysis on a sample of your data (10,000 rows) during data import to find potential keys.

    Disabling the key analysis improves the load performance. To disable this option, clear the Discover Keys at Load Time selection.

    Note: If you choose to not run the key analysis during data import, remember to manually run the analysis at a later time from the Control Center.

    Run Statistics at Load Time

    By default, Trillium automatically performs statistical analysis of numeric attributes during data import to calculate the standard deviation value.

    Disabling statistical analysis improves load performance. To disable this option, clear the Run Statistics at Load Time? selection.

    Note: If you choose to not run the statistical analysis during  data import, remember to manually run the analysis at a later time from the Control Center.

    To find other attribute statistics like percentile value, run percentile analysis of the attribute with numeric data in the Control Center.

    Find Duplicate Rows at Load Time

    By default, Trillium identifies duplicate rows discovered during the data import process.

    To improve system performance, you can disable this feature or you can limit the number of duplicates you want found.

    To disable this option, clear the Discover duplicated rows selection.

    To set a processing limit, enter a number in the Potential duplicates field. The default value is 10,000.

    Standard ABRs Analysis

    During entity creation process, this option allows the Trillium to copy standard ABRs to the new entity.

    Option is disabled by default to speed up the entity creation process and also to control the size of the business rules table.

    To enable copy of standard ABRs, select the Copy standard attribute business rules to attributes option. Now if you create a new entity in the Control Center, all standard ABRs are copied.

    To disable the copy function, remove the check from the Copy standard attribute business rules to attributes option.

  4. Click OK to save your changes.