Execute data stages - Data360_DQ+ - Latest

Manually executing a data stage

If you have Execute permission to an executable data stage, or if you are an administrator, you can run or rebuild it at any time.

For simple pipelines with only a small number of execution dependencies, you can manually execute data stages:

1. Select Pipelines from the top of the screen.
2. Click the menu button to the right of the data stage that you want to execute, then select Execute.
3. Choose Run, Rebuild or Executions:
   • Run - Loads the data stage with new data based on its data load configurations.
   • Rebuild - Loads the data stage with all data, overriding its data load configurations. For example, you can rebuild a data stage that executes on a scheduled basis after you've modified it.
   • Executions - Opens the Executions History, listing all executions of the selected data stage. Depending on which stage is being viewed, different execution details will be provided. From this page, you can select a specific execution and perform the following additional options:
     • Terminate - Stops a run or rebuild mid-process.
     • Rollback - Reverts a data stage to the state it was in just after its last run or rebuild.
     • Rerun - Allows you to perform a run of a specific execution.
     • Refresh - Updates the list of executions.
     • Filter - If you have a long list of executions, you can filter to search for a specific one.
     • Filter By Process Id - If you have selected a process model, you can filter by the process ID of the selected process model to see a list of data stages that were executed during a specific run of that process model.
     • Download Log - Allows you to download log information for a selected execution. You cannot download log information for process models, data store load triggers, or case store execute triggers. Note that you must have Execute permission to the associated data stage to download log information.

Scheduling executions

Administrators and users with Administer permission to an executable data stage can schedule it to load or execute at any time.

If you are working with data stages that must execute on a continual basis, you can create an execution schedule to automate the process. This option is useful when working with data stores that load routinely and that are connected to data views or analyses.

1. Select Pipelines from the top of the screen.
2. Click the menu button to the right of the data stage that you want to execute then select Edit > Edit Settings.
3. Click the Schedule tab and select Enable Scheduling checkbox.
4. Configure the execution settings:
   1. Choose a Recurrence pattern (for example, daily, weekly, or monthly).
      Note: For increased flexibility, you have the option to enter a cron expression. This allows you to have multiple schedules on a data stage. For information on how to write a cron expression, see http://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html
   2. Set the start time and, if needed, the end time or duration.
5. Optionally, configure Exclusions to prevent execution on specific dates:
   Note: Not available for One Time, Monthly, and Yearly.
   1. Select the Exclusions from the drop-down. The available default exclusion calendar for the environment is preselected.

      Use Environment Default - Selects the available default exclusion environment calendar. If the default calendar is not available, it displays calendar value as Not Set.

      No exclusions - No exclusion calendar is selected, which means no dates are execution dates are excluded.

      Use Calendar - Allows you to select the available exclusion calendar for the environment.

      Tip: To create or manage exclusion calendars, see Configure environment calendars. When you import or promote a stage that references an exclusion calendar into a new environment, the system checks whether the calendar exists in the target environment. If the calendar is not present, the system creates a new, empty calendar with the same name as the one in the source environment. You will receive a notification that this calendar has been created.
6. Click Save to apply the schedule.

The data stage will now execute automatically according to the defined schedule, except on dates specified in the selected exclusion calendar. You can view, edit, or disable the schedule at any time from the Schedule tab.