You can move a pipeline that you have built in one environment to another environment to facilitate the process of development. Within different environments, a pipeline may have different use cases, different users who are able to access it, and even different contents or structure.
If your organization is using Data360 DQ+ across multiple environments, you can use the Promote functionality to move data stages or pipelines from one environment to another:
- Select the Pipelines menu at the top of the page.
- Select the pipelines or data stages that you want to move.
- Click the menu button at the top of the Pipelines browser:
- Select Promote.
- Select the target environment.
Selecting Reset promoted items' dirty states in target environment after promotion ensures the promoted items in the target environment is not-dirty.
Environment security settings
Promoting a data stage to another environment places a copy of the item in the target environment.
Access to promoted data stages is then managed through the use of environment groups. Depending on how an environment is configured, environment groups and environment group permissions set in one environment may or may not transfer during promotion.
|
Environment Security Setting |
When items are promoted to that Environment... |
|---|---|
|
Never Create or Update |
Associated environment groups will not be created in that environment. Since they will not be created, they will not be updated either. |
|
Always Create or Update |
Associated environment groups will be created in that environment and updated if they are ever changed in the source environment. This setting makes environment group membership uniform between environments. |
|
Create but never Update |
Associated environment groups will be created in that environment, but they will not be updated if they are ever changed in the source environment. |
Dirty state when promoting
When working with pipelines, you may notice that some items have an asterisk to the right of their name. This marking signifies the item's "dirty state", which is a flag that exists to help you track whether the item's definition has changed since the last time it was promoted or exported.
After creation, you can either manually mark it has Dirty or you can clear the dirty state if it already exists.
Changing environment group membership per environment
Environment group membership can also be changed per environment, so that one environment group has different members in different environments.
| Environment and pipeline | Environment group |
|---|---|
| Development environment - Pipeline A | Environment group A - Developers |
| Testing environment - Pipeline A | Environment group A - Testers |
| Production environment - Pipeline A | Environment group A - End users |
When promoting data stages, this environment group malleability can be extremely useful because it allows the administrator to associate a single environment group with multiple instances of a pipeline - each of which may need different permissions in different environments.
Example - Pipeline promotion
This example illustrates how environment groups, data stages, and promotion can be used to create a multi-environment pipeline structure. It is assumed that the environment group, environments, and data stages (with the exception of step 5) have already been created.
The focus of this example is to show what happens to pipelines when you promote them - and that the mechanics of promotion need to be considered whenever building pipelines that will exist in multiple environments.
In this example, there are two environments:
- A pipeline in a development environment where one set of users can create, save, and modify data stores and data views.
- A pipeline in a testing environment where another set of users can create, save, and modify dashboards using data views created by the first set of users.
There is a restriction that the users creating dashboards in the testing environment cannot create data stores or data views, or modify any existing ones.
(1) - Environment group A is given access to the development environment. If you want to grant an environment group access to a data stage, you must first grant the environment group access to the environment in which the data stage resides.
(2) - Environment group A is granted Write, Create Data Store, and Create Data View permissions to pipeline A. These permissions satisfy the requirement that there is a pipeline in a development environment where one set of users can create, save, and modify data stores and data views. The permissions are applied to the entire pipeline, and will apply to the entire pipeline until rights are delegated at a lower level.
(3) - The administrator sets the testing environment security detail to Create but never Update. This setting will ensure that environment group A is automatically created in the testing environment when pipeline A is promoted.
(4) - Pipeline A is promoted to the testing environment.
When promoting data stages, it is important to identify any dependencies. Although the users creating dashboards in the testing environment only need the data view to do so, the data view is dependent on the data stores in pipeline A. For this reason, we must promote the entire pipeline.
Due to the setting made in step 3, environment group A is created in the testing environment once pipeline A is promoted. By default, this new environment group A will have the same permissions to pipeline A that it did in the development environment.
(5) - The administrator builds a path for dashboards. This step is not required, however it will help to organize pipeline A in the new environment. Alternatively, the administrator could just re-delegate environment group A's rights to path A in the testing environment.
(6) - The administrator changes environment group membership with respect to the testing environment. Environment groups can have different members in different environments. This helps satisfy the second requirement that a pipeline in a testing environment where another set of users can create, save, and modify dashboards using data views created by the first set of users, since the dashboard testers are different users than those responsible for the data stores and data views.
The administrator will now only need to remember that "environment group A" should be associated with this set of data stages, regardless of environment.
(7) - The administrator removes environment group A's Create Data Store, Create Data View, and Write permissions to pipeline A in the testing environment and adds Read and Read Data permissions.
These actions satisfy the requirement that there is a restriction that the users creating dashboards in the testing environment cannot create data stores or data views, or modify any existing ones, and prevents the dashboard builders from breaking any data stores or data views.
Read and Read Data permissions to pipeline A are also added. This allows the dashboard builders to build dashboards with the data view.
(8) - Environment group A is granted Create Dashboard and Write permissions to path B in the testing environment.
This action satisfies the requirement that a Pipeline in a testing environment where another set of users can create, save, and modify dashboards using data views created by the first set of users. It will also allow dashboard builders to create more paths within path B.