Controls are Thresholds that are built in the Collaboration panel, while viewing a Dashboard in a Pipeline. Thresholds applied at this level are called Controls because they have additional functionality that can help you operationalize your data processing Pipelines.
If you have already read the Thresholds documentation or built a Threshold, you already have a basic understanding of how Controls work.
The content of this section therefore focuses on what Controls have to offer that ordinary Thresholds do not; and, how to determine when you should use a Control instead of simply building a Threshold into your Dashboard during initial creation.
Notifications
Controls allow you to notify specific groups of users when a Threshold Rule is broken. Like Dashboard Annotations, this notification is sent via email and it will contain a link to the Dashboard containing the broken Control.
The links sent in Control notifications are designed to take the recipient directly to the "point" on the Dashboard where the Control's Threshold rule was broken. With dynamic data, this "point" will represent the state of data at some time in the past. Also like Annotations, these points are recorded as Comments; so, every time a Control's Threshold rule is broken, a Comment highlighting the violation is made. Such comments allow users to return to points in time where Controlled Threshold rules were broken.
A comment created by the failure of a Control will be made by: Data360 DQ+ System.
Control parameters
- Display Name - The name that will be used to identify the Control in the Collaboration panel.
- Comment Text - This is the text that will be used when the Control is triggered and a Comment is created in the Collaboration panel, to show where the Control's Threshold rule was broken. Comment Text should therefore include information that will help the person who has been notified. For example, notes about the Threshold rule, why it is an issue that it has been broken, and what steps to take next.
- Visible To Group - This parameter defines which environment groups can see this control listed in the Collaboration panel. It also defines who is emailed when the Control's Threshold rule is broken and who will be able to see the comments made by the Control.
- Threshold Rules - When building Controls, you have the ability to build a Threshold for any chart dashlet in a Dashboard whose source is a Data View stage. Once a dashlet is selected, Thresholds are built the same way they are built within Dashboards. Note: If your Dashboard has multiple sheets, you must add the Control while viewing the sheet of interest to have that sheet's dashlets listed in the Threshold Rules parameter.
When are controls applied?
If you've ever built a Threshold inside of a Dashboard, you probably noticed that it was immediately applied - and that any points that broke the Threshold rule were immediately highlighted. This is not the case with Thresholds built inside of Controls.
Control Thresholds are evaluated each time a Dashboard's Data View is Run*, i.e. each time new data is brought to the Dashboard. Depending on how you have scheduled your Data View, this functionality can allow for real time monitoring of the Dashboard's Controls.
To see this in action after building your Control, you will need to Rebuild the Data View used by the Dashboard. If any points that break the Control are found during the Rebuild, users will be notified. Additionally, after the Rebuild, any time the Data View loads new data, the Control will be evaluated.
Viewing control comments
Whenever a Control rule is broken, a new Comment will appear in the Comments section of the Dashboard's Collaboration panel. Like a Dashboard Annotation, this comment represents a snapshot of the point in time when a Control rule was broken. Clicking on this comment will highlight the relevant points that broke the Control.
Additionally, a broken Control will trigger an email notification containing a direct link to the newly made Comment. This link will bring the recipient of the email directly to the points on a chart where the Control threshold rule was broken.
How many comments are made?
Each time a Data View loads new data, one comment is made for all points that broke the Control at the time of the load. Accordingly, just one email per user is sent for that Comment.
Basic control flow
Once a Data Store to Data View to Dashboard relationship has been established, it is likely that the Data View will be loading whenever the Data Store receives new data. In the simplest of scenarios, this would mean that whenever you loaded new data into the Data Store and then ran the Data View, points that broke a Control would be highlighted on the Dashboard, accompanied by a Comment and user notification.
Controls' interaction with other Data Stage configurations
There are, however, a number of configurations at other Data Stage levels that can prevent newly loaded data from making it to a Dashboard and from being considered by a Control.
One simple example is Dashboard Query Criteria. If the Dashboard is only configured to display 1000 records, and your newly loaded data is the 1001th record in its Data Store, this record will not trigger the Control - even if the record breaks the Control's Threshold rule.
There are a number of Data Stage configurations that can prevent data from being considered by a Control, including:
Data View Load Filters, Query Filters, and other settings made in the Other tab
Dashboard Query Criteria and Filters
An Analysis that alters a Data Sore that is used by a Data View
It is therefore important to consider all of these possibilities when writing a Control.
Control permissions
You must have Write permission to a Dashboard in order to Add or Edit a Control.
Only users who are within the Control's Visible to Group will be able to see the Comments the system has made when the Control has failed.
When to use a Control
In sum, Controls extend the functionality of Thresholds by making Thresholds more collaborative and automated.
A use case for a Threshold instead of a Control could be if you are looking at a large set of data just once - and you want to be able to quickly identify points that break some rule.*
Conversely, a use case for a Control could be any type of Threshold use case where it might also help to have multiple people monitoring the Control - and where all of those people need to receive automated notifications when the Control fails.
Controls enable record keeping
In addition, the Comments created when a Control is broken allow you to keep a record of what your data looked like when failures occurred. When working with dynamic data, using a simple Threshold alone will not necessarily allow you to do this because - depending on Retention settings - newly loaded data can clear out older data and remove points on a Dashboard where Thresholds were broken.
Using a custom notification process
If you would like to create a message of your own, instead of using the system generated message that is typically sent when a control is violated, you can use the Custom Notification Process parameter that is available when building your control. To use this parameter, you will need to set up a Process Model to perform the notification process ahead of time.
Building a custom notification Process Model
Below is an example of a typical design for a Custom Notification Process Model to be used with controls.
External Start EventWithin this node, you will need to create two String variables, each of the External Property Type: thresholdResults and commentLink. These are system level variables that will allow you to include data about the Dashboard where the threshold violation occurred in your custom message.
Script Task Within this node, you will need to create Property Reference variables to pass along the thresholdResults and commentLink variables created in the External Start Event. You should also create any other variables you may need within your script.A script such as the following could then be used to process the data associated with your incoming variables.output.thresholdResults = input.thresholdResults;output.commentLink = input.commentLink;var results = JSON.parse(input.thresholdResults);output.dashletId = '"';for (var ps in results) { output.dashletId += ps; output.dashletId += '"';}
Notify Task Within this node, you will once again need to create Property Reference variables for the variables you want to pass over from the preceding Script Task. You may then use those variables in the Notify Task's Message Details, using the ${variableName} syntax.For example:Click here: ${commentLink}The dashlet id was: '${dashletId}'The threshold rule violation results are:${thresholdResults}
Assigning a custom notification Process Model to a dashboard control
Once you have created a Process Model similar to the example described above, you can assign it to the Custom Notification Process parameter in your Dashboard control. After doing so, the next time the Dashboard's Data View runs, it will use the Process Model and its custom message to notify users, instead of using the standard message.