HTTP - Data360_Analyze - Latest

Properties

Url

Specify the URL to which to send the request, for example, "http://www.example.com". The URL must be correctly formatted, or the node will fail.

A value is required for this property.

Choose the (from Field) variant of this property to look up the value from an input field with the name specified.

Method

Specify the type of HTTP request the node will make. If, at any time, the value of this Method property is not in the list below, then the node will fail at that point, and processing will not continue. Choose from:

• GET - Retrieves the entity (essentially, a file) from the specified URL. No body, or input file, is passed to the server.
• PUT - Places the body on the server at the specified location (i.e., the URL).
• POST - Similar to PUT, however, a body is also returned in the response.
• DELETE - Removes the entity at the specified URL. There is no response body.
• PATCH - Partially modify the data in the request Body at the URL specified. This method can return a body but does not have to.
• HEAD - Identical to GET, but unlike GET, no body, or file, is returned.
• TRACE - A testing method commonly used in debugging.
• CONNECT - Used in setting up proxy servers. Not generally needed for this node.
• OPTIONS - A method to poll a server for features that it supports. This is useful if you are exploring a server that you have never used before.

Choose the (from Field) variant of this property to look up the value from an input field with the name specified.

Body

Optionally specify the body of the HTTP request. The body is essentially a file that is sent to the server as part of the input into your HTTP call. Most often, this property will be set to the name of a field on the input pin that contains the data to send to the server.

The Body property should only be specified for POST, PUT, and PATCH requests.

The content-type of the body is defined via the ContentType property.

Choose the (from Field) variant of this property to look up the value from an input field with the name specified.

DataOutputMode

Optionally specify whether data is written to a file on disk or to an output field.

This property also determines how DataOutputField and DataOutputDirectory behave.

The default value is Field.

DataOutputField

Specify the output field that contains either the response body or the filenames where the response body has been written.

The behavior of this property depends upon the DataOutputMode. If the DataOutputMode is Field, it names the field where the response body is output. If the response has no body, then this field contains NULL.

If the DataOutputMode is File, the output field named in the property contains the full path of the files that contain the response bodies.

The data type of this field is set by the DataOutputFieldType property. The DataOutputFieldType property is useful if the response body has Unicode text data and DataOutputMode is set to Field.

DataOutputDirectory

Optionally specify the directory where response bodies are written when DataOutputMode is set to File. When DataOutputDirectory is blank, files are written to the Data360 Analyze temporary directory. Otherwise, the files are written to the specified directory. This node will not overwrite existing files.

If the response has no body, then no file will be written to disk.

This property should only be filled in when DataOutputMode is set to File.

StatusCodeOutputField

Optionally specify the field that will contain the HTTP response status code. This status code is a numeric value that indicates whether the HTTP request succeeded, failed, or needs more information.

See Python Scripting for Custom Error, Paging and Retry scripts for more details about HTTP requests and responses.

Common status codes are the following:

• 2xx - The request succeeded, but there could be qualifications depending on the specific code.
• 3xx - The server needs more information in order to fulfill your request. Typically, the server responds with a list of URL's for you to choose from for your next step.
• 4xx - Error. The request reached the server, but the server could not fulfill it.

The status code is most commonly used to group HTTP responses by these three types.

If not set, the status code will not be output.

StatusMessageOutputField

Specify the field that will contain the HTTP response status message. The status message explains the status code in plain language. The status message is most often used to provide a reason for a failure indicated by a HTTP status code of 4xx.

See Python Scripting for Custom Error, Paging and Retry scripts for more details about HTTP requests and responses.

If not set, the response status message will not be output.

ContentTypeOutputField

This optional property provides an easy way to output the Content-Type header of the HTTP response separately from the rest of the HTTP headers. Since the Content-Type identifies the type of file that is returned with the response--it functions very similarly to a file extension--this header is most commonly the only header that the user needs to output. Placing a field name in this property will cause the value of the Content-Type header to be output separately in a field of this type.

See Python Scripting for Custom Error, Paging and Retry scripts for more details about HTTP requests and responses.

If this property is left blank, then the content type is not output via this property.

If the HeaderOutputField is specified, then the content type is output in that field along with the rest of the headers regardless of the value of this property.

HeaderOutputField

Optionally specify the field where all of the response headers are output. This field will include the Content-Type header even if that header is output separately via its own the ContentTypeOutputField property.

If this property is left blank, then the header is not output.

The Content-Type header can be output separately via the ContentTypeOutputField property regardless of the setting in this property.

DataOutputFieldEncoding

Optionally specify when to encode the HTTP response data when writing to the field specified by DataOutputField.

See Python Scripting for Custom Error, Paging and Retry scripts for more details about HTTP requests and responses.

The data returned from an HTTP call can be either ASCII text, Unicode text, or binary. Since Data360 Analyze does not support binary data in records, this data must be encoded, or put in a valid data format, before being output to a pin. In addition, if DataOutputFieldType is set to "string", then any Unicode data will also have to be encoded in order to avoid errors. Choose from:

• Auto - Determines whether to Base64 encode the data based on the Content-Type of the HTTP response. This setting will encode all data types except for text, html, and xml.
• Base64 - Encodes all DataOutputField values using Base64 encoding. This is the safest option.
• None - Do not encode any of the output data. If binary data comes in an HTTP response, then an error will be thrown, and the node will stop processing. This option should only be used when the user can guarantee that the returned data is not binary and is of the same type as the DataOutputFieldType.

The default value is Auto.

DataOutputFieldType

Optionally specify the type of the field named in DataOutputField. Choose from:

• String - The data output field will be a Data360 Analyze string type.
• Unicode - The data output field will be a Data360 Analyze Unicode string type.

The default value is Unicode.

NetworkErrorOutputField

Optionally specify the output field name that network errors will be written to. A network error is any low level network error that might occur. This could be a socket or connection timeout, TLS error or similar. If this is blank, then network errors are not output.

PageNumberOutputField

Optionally specify the output field name that the page number will be written to. If a value is specified but paging is not enabled, this field will not be present in the output. In the case where paging is driven by the input pin and a record does not have paging enabled, the value of the output will be null.

RetryNumberOutputField

Optionally specify the output field name that the retry counter will be written to. If the counter is 0, then the request was never retried. If this is blank then the retry counter is not output.

ContentType

Optionally specify the "type" of the body, or file, being sent to the server. This "type" acts similarly to a file extension.

While the content type is necessary for PUT, POST, and other HTTP methods that send data to the server, this property does not always have to be filled in. If the input data has already been base64-encoded--this could happen if the data came from another HTTP node with DataOutputFieldEncoding set to Base64 or from any other Data360 Analyze node that encodes its output data--then this node will use the content type that the predecessor node defined for the input data. However, if the body data has not been base64-encoded, then this property must be filled in.

If this property is specified, and the content-type specified by the property is different than the one specified in the Body, this node will stop processing at that point with an error. No succeeding calls will be made.

In technical parlance, the ContentType is a "MIME type". This term could be useful in searching for additional content types.

Examples:

If the body contains HTML, then the ContentType property should be "text/html".

If the body contains Unicode XML, then the ContentType property should be "text/xml; charset=UTF-8".

If the body contains a JPEG, then the ContentType property is set to "image/jpeg".

The default value is "text/plain".

If the content type property is set, but the Header property already contains a content type header, this will not be overridden with the new content type set in the content type field.

Choose the (from Field) variant of this property to look up the value from an input field with the name specified.

Header

Optionally define additional HTTP headers to send with the request. Badly formatted headers will cause an error to occur.

In the case of multiple headers, each header should be separated by a new line. If the headers are input from a field then each header in the field should also be separated by a new line.

The HTTP node itself automatically places the headers listed below into each request. Therefore, the following headers should NOT be set in the Header property:

• Content-Type
• Content-Length
• Content-Transfer-Encoding

The default is to provide no additional headers.

Choose the (from Field) variant of this property to look up the value from an input field with the name specified.

PassThroughFields

Optionally specify which input fields will "pass through" the node unchanged from the input to the output, assuming that the input exists. The input fields specified will appear on those output records which were produced as a result of the input fields. Choose from:

• All - Passes through all the input data fields to the output.
• None - Passes none of the input data fields to the output; as such, only the fields created by the node appear on the output.
• Used - Passes through all the fields that the node used to create the output. Used fields include any input field referenced by a property, be it explicitly (i.e., via a 'field1' reference) or via a field pattern (i.e., '1:foo*').
• Unused - Passes through all the fields that the node did not use to create the output.

The default value is Used.

If a naming conflict exists between a pass-through field and an explicitly named output field, an error will occur.

ServerUsername

Specify the username to authenticate on the remote server. May contain name of domain if necessary in format "Domain\Username".

Choose the (from Field) variant of this property to look up the value from an input field with the name specified.

ServerPassword

Specify the password to authenticate the ServerUsername on the remote server.

Choose the (from Field) variant of this property to look up the value from an input field with the name specified.

ServerAuthenticationMethod

Optionally specify the authentication method used to verify the user on the remote server. Choose from:

• Basic
• Digest
• NTLM

If not set the system will try to automatically detect the best one. Not set by default.

Choose the (from Field) variant of this property to look up the value from an input field with the name specified.

ServerPreemptiveAuthentication

Optionally specify whether the basic authentication response is sent before the server gives an unauthorized response. The server returns immediately the resource (if the credentials are valid)

The default value is False.

Choose the (from Field) variant of this property to look up the value from an input field with the name specified.

ConnectionTimeout

Optionally specify the maximum period (in seconds) to wait before the connection times out.

By default, a connection timeout period does not apply.

MaxDataFieldSize

Optionally specify the maximum size of response content in bytes. This property only applies when DataOutputMode is set to Field.

By default the response size is 52428800 bytes (50 Megabytes).

TrustStoreFile

Optionally specify the name of the truststore file.

The default value is the Java Runtime Environment (JRE) truststore file.

Choose the (from Field) variant of this property to look up the value from an input field with the name specified.

TrustStoreFilePassword

Optionally specify the password relating to the TrustStoreFile.

A value is required for this property if the TrustStoreFile property is specified.

The default value is the Java Runtime Environment (JRE) truststore file password.

Choose the (from Field) variant of this property to look up the value from an input field with the name specified.

PagingModel

Specifies the paging model to be used:

• Analyze
• Govern
• Jira
• Zendesk
• Custom - A custom model can be set either through the PagingSetupRequest and PagingProcessResponses parameters, or by setting properties. See Python Scripting for Custom Error, Paging and Retry scripts for more information.
• Undefined - No paging model is set.

The default is Undefined.

PagingSize

Optionally specify the size of the pages to retrieve. Specifying the value in this parameter field, will override the default set for the paging model. The defaults vary for each model and are as follows:

• Analyze: 200
• Govern: 200
• Jira: 1000
• Zendesk: 100
• Custom: Unlimited

PagingSetupRequest

Optionally specify a custom python script for initializing an HTTP request for paging, prior to any requests being executed. This script is executed once per record.

See Python Scripting for Custom Error, Paging and Retry scripts for more details about HTTP requests and responses.

Example:

This example adds paging parameters to the URL (http://mysampleurl.com) to initialize paging:

maxPage = 10 # Initialize variable for use in PagingProcessResponses script

currentPage = 1

request.url.args['page'] = currentPage

# The resulting url would be http://mysampleurl.com?page=1

PagingProcessResponses

Optionally specify a custom python script for parsing an HTTP response and updating the subsequent HTTP request, after the request has been executed. This script is executed after every HTTP request has been executed.

See Python Scripting for Custom Error, Paging and Retry scripts for more details about HTTP requests and responses.

To stop paging set:

request = None

See below for API information.

Example:

This example updates the paging URL and stops paging when the `maxPage` limit has been reached:

if currentPage == maxPage:

request = None

else:

currentPage = currentPage + 1

request.url.args['page'] = currentPage

Custom models can be specified through the PagingSetupRequest or PagingProcessResponses parameters, or by specifying properties in <site-dir>/conf/site.prop, which point to python files located in the <site-dir>/conf directory.

This is achieved by doing the following:

1. Specify your model name. For the purposes of this example we will call the model exampleModel.

   Model names must adhere to the `[a-zA-Z][-_a-zA-Z0-9]*` naming convention:

   The first character can be:

   • A lowercase letter.
   • An uppercase letter.

   Subsequent characters can be:

   • Lowercase letters.
   • Uppercase letters.
   • Digits (0-9).
   • Underscore (_).
   • Hyphen (-).

   If the first character does not comply with convention, an error will occur.

   If subsequent characters do not comply with convention, they will be replaced with an underscore (_) and no error will occur. For example, `JIRA (v3)` will be changed to `JIRA__v3_`.

2. Put the PagingSetupRequest and PagingProcessResponses scripts in the <site-dir>/conf directory.
3. Specify the the following properties in <site-dir>/conf/site.prop, using the template
   ls.brain.node.http.paging.<modelName>.<propertyName>:

   • ls.brain.node.http.paging.exampleModel.pagingModelImpl=python.

     This property is required, python is currently the only supported language.

   • ls.brain.node.http.paging.exampleModel.pagingSetupRequest="${ls.appDataDir}/lib/nodes/http/<exampleModelPagingSetupRequest>.py".

     This property is not required. If it is not specified, no modifications will be made to the URL prior to executing the first HTTP request.

   • ls.brain.node.http.paging.exampleModel.pagingProcessResponses="${ls.appDataDir}/lib/nodes/http/<exampleModelPagingProcessResponses>.py".

     This property is required.

   • ls.brain.node.http.paging.exampleModel.pageSize=<pageSize>.

     This property is not required. If the PagingSize parameter is set, this will override any value provided in this property.

4. Custom properties can also be specified in <site-dir>/conf/site.prop by following the template above.

   For example, ls.brain.node.http.paging.exampleModel.myCustomProperty=myValue. Refer to the Analyze Python API documentation, for more information on how to read properties.

5. The model can be specified in the node in multiple ways:
   • By passing the model name into the PagingModel parameter field.
   • By using the input pin and specifying the input field in the PagingModel parameter field.

RetryCondition

Optionally specify under what conditions to retry a request.

Values:

• Never - Never retry a request.
• Http Status Error - Retry the request if the response status code is in the 4xx client error or 5xx server error categories.
• Network Error - Retry the request if a network error occurs. A network error is any low level network error that might occur. This could be a socket or connection timeout, TLS error and so on.
• Any Error - Retry the request if the response contains a http status error or a network error.
• Custom - Using the RetryProcessResponses field, provide a python script that checks responses for custom errors to retry.

The default value is Never.

RetryProcessResponses

Optionally specify a custom python script for parsing an HTTP response and determining whether the HTTP request should be retried. This script is executed after every HTTP request has been executed.

See Python Scripting for Custom Error, Paging and Retry scripts for more details about HTTP requests and responses.

To retry a request set the following:

request.retry = True # request.retry is False by default

Example:

The following example will retry a request if the response contains a 504, any other HTTP status will not be retried

if response.status == 504:

request.retry = True

OutputFailedRetriedResponses

Optionally specify whether or not to write failed retried requests to the output pin. If set to false, only the last failed or succeeded request will be written to the output pin.

The default value is False.

MaxRetries

Optionally specify the number of times to attempt an unsuccessful retry before error.

The default value is unlimited

RetryTimeout

Optionally specify the maximum total amount of time to attempt to retry a failed request regardless of the number of attempts.

The default value is unlimited, the node will not timeout.

RetryDelay

Optionally specify the delay between retry attempts, in seconds.

The default value is 0

MaxRetryDelay

Optionally specify the maximum delay between retry attempts, in seconds.

The default value is 3600 seconds

RetryDelayFalloff

Optionally specify the floating point fall-off factor that is used to calculate the next retry delay. A fall-off of 1.0 would keep the delay the same for every retry. A fall-off greater than 1.0 will increase the delay and a fall-off less than 1 will decrease the delay.

Negative numbers are invalid.

The default value is 1.0

ProxyUrl

Optionally specify the URL of the proxy server to use for the HTTP requests. The URL should contain the protocol (http or https), the host, and the port. All other parts of the URL will be ignored.

The URL should be in a format similar to the following:

• HTTP proxy: http://proxy.example.com:1234
• HTTPS proxy: https://proxy.example.com:443

The node supports automatic detection of system proxy settings To override system proxy settings see the IgnoreSystemProxy property.

ProxyUsername

Optionally specify the proxy username, where the proxy requires authentication.

This property should be left blank if ProxyUrl is not set.

ProxyPassword

Optionally specify the proxy password, where the proxy requires authentication.

This property should be left blank if ProxyUrl is not set.

ProxyAuthenticationMethod

Optionally specify the authentication method used to verify the user of the proxy server. Choose from:

• Basic
• Digest
• NTLM

If not set the system will try to automatically detect the best one. Not set by default.

ProxyPreemptiveAuthentication

Optionally specify whether the proxy basic authentication response is sent before the server gives an unauthorized response. The server returns immediately the resource (if the credentials are valid)

The default value is False.

IgnoreSystemProxy

If proxy settings are not defined then the system proxy is automatically used. This optional property tells Data360 Analyze to ignore the system proxy when set to true.

The default value is False.

Cookies

Optionally specify request cookies to be used with every HTTP call. These cookies can be either typed in by hand using the format below or passed in from a CookiesOutputField. The format of the cookies field is a series of Set-Cookie headers separated by newlines. For example,

Set-Cookie: foo=bar

Set-Cookie: baz=qux; Max-Age=3600

The Set-Cookie header is used in place of the Cookie header in order to preserve the age, expiration, and other information that are returned in an HTTP response.

Choose the (from Field) variant of this property to look up the value from an input field with the name specified.

CookieAccumulation

Optionally specify how response cookies are retained, or accumulated, across multiple HTTP requests made from a single HTTP node. These accumulated response cookies are used in the request of the next HTTP call. Choose from:

• All - Emulate a web browser by accumulating cookies from the input and the responses. All subsequent requests will include all cookies received from any previous request plus the input cookies.
• Input - Accumulate only cookies that come from the Cookie property.
• Response - Accumulate only cookies that come from the response of a previous request.
• None - Do not accumulate cookies. The only cookies that will be included with a request are specified in the Cookies property.

The default value is All.

Most commonly, only All and None will be used.

CookiesOutputField

Optionally specify the field where cookies are output. These output cookies can be passed to a subsequent HTTP node so that the next HTTP call can have all cookies that resulted from the last HTTP call.

The output cookies will contain, at a minimum, the cookies returned with the response. In addition, the output cookies will also contain whatever request cookies had accumulated for the request. If CookieAccumulation is set to All, then all cookies used to make the request will be added to the output. If CookieAccumulation is set to Input, then only the input cookies will be added to the response cookies. If CookieAccumulation is set to Response, then the accumulated response cookies will be added to the cookies from this response. Finally, if CookieAccumulation is set to None, then only the response cookies are output.

Normally, HeaderOutputField will contain any Set-Cookie headers sent back with the response. However, if CookiesOutputField is also set, then these headers are removed from HeaderOutputField and output only in the CookiesOutputField.

ErrorCondition

Optionally specify under what conditions to set a response as an error.

Values:

• Never - No response will be considered an error.
• Http Status Error - Error if the response status code is in the 4xx client error / 5xx server error categories. Network errors are caught as errors here too.
• Network Error - Error if a network error occurs. A network error is any low level network error that might occur. This could be a socket/connection timeout, TLS error etc.
• Custom - Using the ErrorProcessResponses field, provide a python script that checks responses for custom errors.

The default value is Network Error.

ErrorProcessResponses

Optionally specify a custom python script for parsing an HTTP response and determining whether the HTTP response contains an error. This script is executed after every HTTP request has been executed.

See Python Scripting for Custom Error, Paging and Retry scripts for more details about HTTP requests and responses.

To flag a response as an error set the following:

response.error = True # response.error is False by default

Example:

The following example will flag an error if the response contains a 503 HTTP status, or if the response contains a network error. Any other HTTP status will not be considered an error.

if response.status == 503 or response.networkError == True:

response.error = True

ErrorBehavior

Optionally specify the behavior of the node if an error occurs.

Options:

• Error - Logs an error to the Errors panel.
• Log - Logs a warning to the Errors panel.
• Ignore - Ignores the error.

The default value is Error.

StopAtFirstFailure

Optionally specify if the node should terminate execution on the first failure it encounters. Note this is a failure on a per record input basis, and if Retry is configured, then the node will retry until the max number of retries has taken place.

The default value is False.

Enabled

Optionally specify whether the node is enabled or disabled. You can either choose True or False, or reference another property (see Using derived property values) which will be evaluated to a true/false value.

Disabled nodes are not executed, even if they are selected to run.

The default value is True.

LogLevel

Optionally specify the level at which non-fatal messages are logged. The lower the level, the more information will be recorded in the log file. Choose from:

• 0 - Information
• 1 - Low
• 2 - Medium
• 3 - High

Inputs and outputs

Inputs: 1 optional.

Outputs: response and optional error. If an error pin is not specified, error responses will be written to the response pin.

Example data flows

A number of sample Data Flows are available from the Samples workspace, found in the Analyze Directory page.

In the Directory under the /Data360 Samples/Node Examples/ folder, you will find "HTTP Integration with other Services", which shows examples of how to use this node.

Python Scripting for Custom Error, Paging and Retry scripts

API for Custom Error, Paging and Retry scripts

The following are available for use in paging, error and retry scripts:

HttpRequest

The HttpRequest object contains the following fields:

• method - Type: HttpMethod. The request method. HttpMethod is an enumeration and contains the following values:

  HttpMethod.DELETE

  HttpMethod.GET

  HttpMethod.HEAD

  HttpMethod.OPTIONS

  HttpMethod.PATCH

  HttpMethod.POST

  HttpMethod.PUT

  HttpMethod.TRACE

HttpMethod can be set by setting the value to HttpMethod.<value>, for example:

request.method = HttpMethod.GET

• url - Type: furl object (Refer to furl documentation for usage: https://github.com/gruns/furl). The request URL.
• contentType - Type: String. The content type of the request.
• headers - Type: Dictionary. The request headers.
• Dictionary key is the header name, and value is a list of header values, for example: { <header-name> : [ <value>, <value>] }
• body - Type: String. The request body.
• json - If the response content type is application/json, the response body is parsed as JSON and set here.
• retry - Type: Boolean. Retry flag used to flag if a request is to be retried.

The HttpRequest object can be referenced in custom scripts using request.

HttpResponse

The HttpResponse object contains the following fields:

• status - Type: Integer. The HTTP status code of the response.
• statusMessage - Type: String. The associated status message to the status code.
• headers - Type: Dictionary. The response headers.

  Dictionary key is the header name, and value is a list of header values, for example: { <header-name> : [ <value>, <value>] }

• body - Type: File. The a file containing response body.
• error - Type: Boolean. Error flag used to flag if a request contains an error.
• networkError - Type: Boolean. Error flag used to flag if a request contains a network error.

The HttpResponse object can be referenced in custom scripts using response.

Example Parsing Response body

If the content type of the response is application/json, the request body will automatically be parsed into the json parameter of the response object. This can then be used as follows:

Response:

{

'metadata': {

'page': 10

'items': [...]

}

page = response.json['metadata']['page']

PagingException

If an error occurs in a script, throwing a PagingException will allow the node to catch the exception and log the output to the error console. For example, if a response has no body when a body is expected, the following can be done:

if response.body is None:

raise PagingException("No response body found for request: %s", request.url.url)