Configuring environment retention - Data360_DQ+ - Latest

Data360 DQ+ Help

Product type
Software
Portfolio
Verify
Product family
Data360
Product
Data360 DQ+
Version
Latest
Language
English
Product name
Data360 DQ+
Title
Data360 DQ+ Help
Copyright
2024
First publish date
2016
ft:lastEdition
2024-07-09
ft:lastPublication
2024-07-09T15:09:58.774265

The Default Retention Period of an environment is the length of time for which data stages in the environment will store data. This setting is always applied uniquely to each data stage, each time a data stage executes with time being calculated from the time data is loaded into the data stage.

Note: The environment retention period is applied to all data stages within the environment, however, you can optionally set data store and data view retention settings to override environment settings if required.

Data retention is applied once per day and is calculated from the end of the most recent successful load or execution. If a run fails, data retention will be calculated from the failed run's start time.

Environment retention period

The Execution History Retention Period of an environment is the length of time that execution history information will be retained in an environment's executions section.

 

  1. From the menu at the top of the screen, select Admin > Environments.
  2. To edit the retention settings of an existing environment, select the environment that you want to edit from the list on the left of the screen.
  3. Ensure that the Details tab is selected, then scroll down to the Retention section.
  4. Edit the Retention settings as required:
    • Default Retention Period - The length of time for which data stages in the environment store data.
    • Period Precision - The precision at which a retention period is applied. For example, if an environment's retention period was set to 1 Year and the Period Precision was set to Month, the retention period would span one year back from the current day, plus the remaining days to get to the beginning of the month. For additional examples, see: Examples.
    • Execution History Retention Period - The length of time that execution history information is retained in an environment's executions section.
    • Execution Period Precision - The precision at which a retention period is applied.
    • Move S3 files to Glacier when retention deletes data - Select this option to move data store files that are stored in Amazon S3 to Amazon Glacier for backup when the system's retention processing deletes data from these data stores.
  5. Click Save Changes.
Tip: If you are working with static data, it is recommended that you set the environment retention period to something large and arbitrary so that users will be able to work with the data for a long period of time.

Examples

Retention period example one

In this example, the current date is 6/5/2020.

The environment Default Retention Period is set to 1 Year and Period Precision is set to Month.

The system first looks back one year to 6/5/2019. Then, since precision is set to Month, the system would go to the beginning of the month, to 6/1/2019.

Any data that was loaded into the data stages before 6/1/2019 is removed.

Retention period example two

The current date is 6/5/2020.

The environment Default Retention Period is set to 1 Year and Period Precision is set to Quarter.

The system first looks back one year to 6/5/2019. Then, since precision is set to Quarter, the system would go to the beginning of the quarter, to 4/1/2019.

Any data that was loaded into the data stages before 4/1/2019 is removed.

For quarter date information, see Quarter dates.

Retention period example three

The current date is 6/5/2020.

The environment Default Retention Period is set to 1 Year and Period Precision is set to Year.

The system first looks back one year to 6/5/2019. Then, since precision is set to Year, the system would go to the beginning of the year, to 1/1/2019.

Any data that was loaded into the data stages before 1/1/2019 is removed.

Quarter dates

Quarter

Start date

End date

1

January 1st

March 31st

2

April 1st

June 30th

3

July 1st

September 30th

4

October 1st

December 31st

Note: Note these ranges also apply to other features within Data360 DQ+, such as the Quarter function and date filters that filter by quarter.

Retention processing for comments, attachments and annotations

  • Data retention for a dashboard's comments, attachments, and annotations is based on the data retention settings of the dashboard's data view that has the shortest retention period. If none of the dashboard's data views have retention settings, the data retention settings of the environment are used.
  • Data retention for comments and attachments on other data stages is based on the retention settings of the environment.
  • Retention settings are evaluated against the most recent comment in a thread. If the most recent comment in a thread is older than the retention period, then the entire thread is removed.
  • If an attachment is made with a comment, retention settings are evaluated against the most recent comment in a thread. If an attachment is made separately from a comment, retention settings are evaluated against the attachment's upload date.