Realtime Data Metrics - ironstream_for_splunk - ironstream_for_kafka - Ironstream_Hub - ironstream_for_elastic - 1.3

Ironstream Hub Administration

Product type
Software
Portfolio
Integrate
Product family
Ironstream
Product
Ironstream > Ironstream Hub
Ironstream > Ironstream for Kafka®
Ironstream > Ironstream for Elastic®
Ironstream > Ironstream for Splunk®
Version
1.3
Language
English
ContentType
Administration
Product name
Ironstream Hub
Title
Ironstream Hub Administration
First publish date
2022
ft:lastEdition
2024-04-15
ft:lastPublication
2024-04-15T10:44:03.897025

One of the core design principles within Hub is that the user should be able to monitor the status and performance of each Data Flow. This has been achieved by adding real time and status information into the trace logs on a regular basis.

Realtime metrics are collected at the level of individual pipeline elements and at the level of complete pipelines. The element monitor will write an entry into the log whenever the element it is monitoring is invoked within the Pipeline. The pipeline monitor will write an entry to the log once the flow of data has ceased for a period of time, defined in the Source configuration file.

This is a list of metrics that are written by the pipeline monitor and element monitor:

Monitor Type Metric Description Units
Element Throughput A measure of the speed with which data flows through the individual element. MB/s
Element Bytes handled The number of bytes handled by the element. Used to calculate the throughout. Bytes
Element Elapsed time The amount of time for which the element was active. Used to calculate the throughput. Seconds
Element Records handled The number of records handled by the element within the current payload. Number of records
Element Total records handled The total number of records handled by the element since it was created. Number of records
Pipeline Throughput A measure of the speed with which data flows through the entire pipeline. MB/s
Pipeline Bytes handled The number of bytes handled by the pipeline. Used to calculate the throughout. Bytes
Pipeline Records handled The number of records handled by the pipeline. Number of records
Pipeline Elapsed time The amount of time for which the pipeline was active. Used to calculate the throughput. Seconds
Pipeline Latency first record The time taken for the first record to enter and leave the system. Microseconds
Pipeline Latency last record The time taken for the last record to enter and leave the system. Microseconds