Handle external data store contents - Data360_DQ+ - Latest

After creating and saving an external data store, you can view the data store, upload a new file or delete all data.

1. Go to the pipeline and path that contains your data store.
2. Click the menu button to the right of the data store and select Edit > Edit Stage.

   A number of options are displayed across the screen above the data store tabs:

   • View - Allows you to look at the files that were used to create the data store, or its contents. The content can also be downloaded for local use. When viewing contents, files are grouped by data store load events, and data store load events are labeled by system generated Work Ids.

     When viewing data store content, you can also view each load event's execution history by selecting a Work Id and then clicking the View Execution button.

     Note: The following permissions are required:Permission needed to View: ReadPermission needed to Download files: Read DataPermission needed to view Load Event Execution History: Administer

     To download a file when you are viewing data store content, select the file in the Details panel at the bottom of the screen, then click Download. Data stores created from file are available in the compressed .bz2 format. These files must be extracted before you view them on your local machine.

   • Upload File - Allows you to upload files to the data store. After an external data store has been created and saved, you can continue to upload files to it. Assuming a file has the same field structure as the file that was used to create the external data store, an upload will add new data to the data store. However, if you attempt to upload a file with a different field structure than the file that was used to create the data store, errors may occur.

     When manually uploading files to an existing data store, it is important to consider how the upload will affect other data stages that use that data store. After upload, you might need to run or rebuild executable data stages that are connected to the data store. Depending on your permissions, such executions could be carried out manually, scheduled, or orchestrated via process models.

   • Delete All Data - Removes all data that is currently loaded to the store. You might want to perform this action in between analysis runs or rebuilds, or in cases where you want to upload a new file with the same field structure to an external data store and are not interested in the data contained in the old files.
     Note: Note that deleting all data can have an impact on other data stages that use the data store. To determine if this will be an issue, right-click the data store when viewing it in a pipeline, click More from the toolbar and select Find Usages.
     Note: Deleting all data from a data store while data stages that use that data store are executing will cause the execution to fail. Therefore, you should always check to see which data stages use the data store before deleting its data. If you need help with this, contact your Data360 DQ+ Administrator.
   Note: The View Content and Upload File buttons are not available for some data store types, for example for internal data stores with the delta lake repository type.