Realtime Data Metrics - ironstream_for_splunk - ironstream_for_kafka - Ironstream_Hub - ironstream_for_elastic - 1.3

One of the core design principles within Hub is that the user should be able to monitor the status and performance of each Data Flow. This has been achieved by adding real time and status information into the trace logs on a regular basis.

Realtime metrics are collected at the level of individual pipeline elements and at the level of complete pipelines. The element monitor will write an entry into the log whenever the element it is monitoring is invoked within the Pipeline. The pipeline monitor will write an entry to the log once the flow of data has ceased for a period of time, defined in the Source configuration file.

This is a list of metrics that are written by the pipeline monitor and element monitor:

Monitor Type Metric Description Units Element Throughput A measure of the speed with which data flows through the individual element. MB/s Element Bytes handled The number of bytes handled by the element. Used to calculate the throughout. Bytes Element Elapsed time The amount of time for which the element was active. Used to calculate the throughput. Seconds Element Records handled The number of records handled by the element within the current payload. Number of records Element Total records handled The total number of records handled by the element since it was created. Number of records Pipeline Throughput A measure of the speed with which data flows through the entire pipeline. MB/s Pipeline Bytes handled The number of bytes handled by the pipeline. Used to calculate the throughout. Bytes Pipeline Records handled The number of records handled by the pipeline. Number of records Pipeline Elapsed time The amount of time for which the pipeline was active. Used to calculate the throughput. Seconds Pipeline Latency first record The time taken for the first record to enter and leave the system. Microseconds Pipeline Latency last record The time taken for the last record to enter and leave the system. Microseconds