The Request accordion includes the details of how to make the request and what to collect from the RESTful API. The Data Connector includes the Request Type and Request Body that are required to make the request you only need to specify the URL and Headers that are specific to your instance of the RESTful API.
URL
The URL is the entry point into the RESTful API and is different for each instance of a RESTful API.
-
Modify the URL to the point to your instance of the RESTful API
Note: There is no validation applied to the URL entered, if an incorrect URL is defined, the connection will simply fail and data will not be retrieved.
If a start and/or end date macro has been specified in the URL with the {start} or {end} placeholder, when editing the URL these macros need to be retained as they determine the time range of the data being collected for each request. When running the request, they are replaced by the contents of the Start Date and End Date controls in the selected Date Format that are defined in the Date/Time tab within the Placeholders accordion.
Headers
Headers are configuration parameters that contain extra information about the request that are not specified in the URL or the Request Body. The Names of the Headers will be defined as part of the Data Connector, but the values of some of them may need to be updated to reflect your instance of the RESTful API. For example: some RESTful APIs require a Region to be specified and this can be different for each RESTful API instance.
-
Modify the Value of any row that needs to be specific to your instance of the RESTful API by clicking the Edit button on the table in the Headers accordion.
Placeholders
Within the Request section there is a collapsible Placeholders accordion. This section allows you to define parameters that are fed into the Request (URL or Request Body) one at a time, via the use of placeholder macros ({start}, {end}, {metric} and {instance}).
There are three distinct categories of Placeholder and these are described in the sections below.
Date/Time
The Date/Time accordion in the placeholder’s accordion is used to define the Date Format when requesting the time range of the data required.
The Start and End Dates are only used by the Portal to assist in definition of the Template.
The Data Connector includes the Date Format that will be used for all dates and times included in the request to your RESTful API instance and this should not need to be changed.
We need to a choose a time range that includes data within the RESTful API instance, as one or more samples need to be returned from the RESTful API to test the connection and map data into Syncsort™ Capacity Management.
-
Set the Start Date and End Date to a time range for which your RESTful API instance can return data
Metrics
The Data Connector includes a set of metrics recommended for collection from this type of RESTful API.
You can edit the list using the “Add”, “Edit” and “Delete” icons if you require different metrics to those included I the Data Connector. If you do want to add additional metrics you will need to know the unique identifier the RESTful API to reference it.
Instances
Depending on the Data Connector you have chosen instances may or may not need to be specified in the Template. This can be done in four different ways depending on RESTful API. The table below explains the different types of instance specification:
|
List Specific Instances
If the Data Connector has the Use Instance Collector option unchecked within the Instances tab it expects you to type in a list of instances.
To add specific Instances, you need to know the Identifier the RESTful API uses for each instance.
-
Click the Add, Edit, or Delete icon to define the instances.
As with the Metrics if the identifier of the instance is not a user-friendly name, Syncsort™ Capacity Management allows you to type in a Name for the Metric so that you can clearly identify each one in Syncsort™ Capacity Management Reporting applications
A Data Collector Template defined in this way will only ever retrieve information for the Instances specified within the Template. This list will need to be kept up to date manually.
Automatically Collect Instances
If the Data Connector includes two XML files and the Data Collector has the Use Instance Collector option checked then:
-
Select the Instance Collector that you have already defined for this instance of the RESTful API from the dropdown box.
If the Template is not listed, then it has not yet been defined and made live. Follow the instructions in the Instance Collector Template section above before proceeding.
If a Data Collector Template uses an Instance Collector Template this automatically retrieves the list of available instances to collect data from.
Run Request
Once the input fields have been updated with your RESTful API details then the next step is to test authentication and request details.
Click the Run Request button.
When the ‘Run Request’ button is pressed, Integrator will try to establish a connection to the specified URL, using any specified Authentication Script. If the connection fails, an error message describing the reason for failure will be displayed, otherwise the result of the API call will be displayed in a ‘Request Results’ text box which appears below the ‘Run Request’ button. This text box is read-only but can be copied to the clipboard if you want.
It might take a few minutes for the Run Request to complete, but once complete, the results of the API call will be displayed in whatever format the API delivers it, typically XML or JSON. These results are not stored as part of the Template Definition, so will not be displayed when the Template is re-loaded.
Note that the number of test API calls, and therefore the results displayed, will be limited to the first N defined Instances, where N is defaulted to 15, refer to the online help if you want to know how to change the value of N. This is purely to provide quicker feedback on the nature of the API results. When the Control Center runs the RESTful API request the limit is not applied.
If the URL or Request Body is updated once the Request has been run a warning message will be displayed to show the results are out of sync with the inputs.
Conversion
The Conversion accordion includes all the details required to convert the text returned by the RESTful API into a Data Table that can be used for mapping fields and processing via Control Center.
You should not need to change script for a Data Collector, however you may need to change the Interval Length.
Interval Length
Interval Lengths are important in Syncsort™ Capacity Management as they allow Control Center to determine the time span of a sample (i.e., Start Date Time + Interval Length = End Date Time). The Interval Length allows Control Center to determine if a sample overlaps another sample already stored in the database. For each instance being stored in a Capacity Database there must not be samples that overlap, as this would mean that any counts stored for a time range were in the database twice and would give an elevated and incorrect representation of the metrics. The Interval Length also allows easy conversion between rates and counts which is very helpful when reporting.
If the Specify Interval Length option is not selected within the Data Connector, then the RESTful API request should return the Interval Length and it does not need to be specified explicitly.
If the Specify Interval Length option is selected within the Data Connector then any Interval length the RESTful API request is not expected to be used and it must be specified as part of the Template.
Choose an Interval Length.
Ensure it matches the Interval length in the data being collected as it can vary between instances of a RESTful API. If you do not know the Interval Length you will need to follow the instructions below to return results before setting the value. This will enable you to see the samples that are returned by the request and determine the Interval Length from them.
Run Request and Convert
To test the conversion script and check the mappings:
Click the Run Request and Convert button.
When the Run Request and Convert button is pressed, Integrator will try to establish a connection to the specified URL, using any specified Authentication Script. If the connection fails, an error message describing the reason for failure will be displayed, otherwise the result of the API call will be run through the Conversion Script. If the Conversion Script generates an error, the cause of the error will be displayed allowing you to correct it and retry. If the Conversion Script is successful then the resultant data set will be displayed in the Data accordion, where it can be mapped to Components, Metrics, etc. for processing into the Capacity Database. Refer to Data for further details about this.
Data
The Data accordion includes a table that displays the column mappings defined in the Data Connector. Once the table is populated with the results of clicking the Run Request and Convert button, it also displays the results of running the Request.
You should not need to change any of these mapping for a Data Collector based on a Data Connector unless you have changed details in the other accordions that affect the columns returned by the request (e.g., added a metric).
However, any changes you have made to the Template may affect the columns displayed in data table. New metrics may need to be mapped, unwanted metrics may need to be unmapped and any unavailable metrics need to be ignored.
Data Collector mapping options:
-
Unused: Any column that is returned as part of the Request but is not used by the Template.
-
Start Date: One column that includes the Start Date for each record.
-
Interval Length: A single column that includes the time in seconds between each sample.
-
Component: Columns that make up the unique identifier of each instance being collected.
-
Metric: Columns that contain the values Syncsort Capacity Management will be used to report on for each instance.
If you need to change the column mapping properties, refer to the online help for details.
Save and Make Data Collector Live
Once the Data Collector has been defined you can save the Template and Make It Live so that it is available for use by Control Center to collect data.
- There are up to date results in the Data Grid and a column i.e., there have been no edits since the data Grid was last populated with results. If changes have been made an Out of Sync message will appear next to the Run Request and Convert button and you will need to click Run Request and Convert again before the Make Live button is enabled.
- There is one Date/Time column
- There is one Interval Length column
- There at least one Metric column
- Click Save.
- Click Make Live.
- The metric and component details are added to the Syncsort Capacity Management reporting applications
- The Template is added to the list of Templates that can be associated with a new Target in System Manager. So that Collection can be started via Control Center.
- A Target Group is created that will include all targets that are associated with this Template
Now that the Data Collector Template is Live you can associate a Target with it and start collecting and reporting on the data.
Creating a Target
We now need to create a Target that will use the Template and allow Control Center can collect and store data.
- Open System Manager, making sure the appropriate database is selected.
- Click the New Target menu item or the Add Target
button in the toolbar
to open the Add Target Wizard.Figure 8. Target Wizard – Acquire Type
- Select Integrator from the Acquire Target drop down.
- Choose the Live Data Collector Template, that we defined in the previous section, from the dropdown in the Integrator Template frame.
- Click Next.Figure 9. Target Wizard – Control Center
- Choose the Control Center that you want to collect data for this Target.
- Click Next.Figure 10. Target Wizard – Name and Description
- Type in the Name you want to use for the Target and any Descriptions you required.
- Click Next.Figure 11. Target Wizard – Collection Profile
- Choose the Collection Profile and ensure the Collection Enabled option is checked.
- Click Next.Figure 12. Target Wizard – Time Period Settings
- Choose Time Period Settings. See the table below for explanations of
each option.
Frame
Description
Start
Process samples after
Selecting this option means that data is collected continuously from a fixed point.
Once a Target has been created the Process samples after value cannot be changed to be a date/time earlier than the last data sample processed into the Capacity Database. If you want to collect and process data from an earlier period, create a new Target and set the Process samples after appropriately.
Collect data newer than
Selecting this option means that only data newer than the chosen time interval will be collected.
For example: this option can be used when there is a break in collection and you would rather not collect all the data since collection stopped and just have the data collection start from a recent time interval.
If you have been running your RESTful API Instance for some time, you can set the start date to be in the past, to capture all available historical data.
End
This defines a buffer period for collection. Control Center will only collect samples that are this old or older.
This ensures that Control Center does not try to collect the most recent time range, in case the RESTful API’s capture mechanism takes a significant amount of time to store the data for the most recent time period in its repository.
The default value is 60 minutes; that is, we do not collect any data that was captured in the last 60 minutes.
Batch Size
This defines the time range of data collected within a single collection event.
Batching allows data to be broken down into small sets of data samples in a single request or time period. This may be needed for performance reasons (e.g., helping to prevent timeouts) or due to a limit in the volume of the data returned from the RESTful API itself.
Syncsort™ Capacity Management allows you to change the number of minutes of data that each collection request will contain. The value should be greater than the interval length as collection cannot be split into batches smaller than a single interval.
The default value is 15 minutes; i.e., we do not collect more than 15 minutes of data in a single collection request.
Move to next collection interval on non-authentication errors
Selecting this option means that if requests fail for specific reasons Control Center will move on to the next collection interval rather than ‘getting stuck’ at a certain collection interval.
If an authentication error occurs, we don’t’ want to skip any data as when the connection is reestablished we would expect that we will be able to collect the data. That is, we don’t want to lose data due to connection issues.
However, when other issues occur (e.g., no data for this interval, due to a gap in the data being captured) some RESTful APIs return an error rather than just returning no data. Therefore, if this is the case we want the option to ignore the error and skip the missing data.
If this option is turned on, by default the non-authentication error codes that allow data to be skipped are: “400,402-499”. If there are additional error codes not in this list that your RESTful API produces that need to be treated as non-authentication errors, they can be added to a configuration file. Refer to the online help for further details.
- Click Finish.
The Target is added to the Capacity Database.
The Integrator Target Group for this Data Collector Template will include the new Target.
The Edit Page for the new Target is displayed.