Ironstream™ Software for Splunk®/Kafka®/Elastic® for IBM Z® Configuration and User Guide - ironstream_for_splunk - ironstream_for_elastic - ironstream_for_kafka - 2.1
Product type
Software
Portfolio
Integrate
Product family
Ironstream™ software
Product
Ironstream™ software > Ironstream™ software for Splunk®
Ironstream™ software > Ironstream™ software for Elastic®
Ironstream™ software > Ironstream™ software for Kafka®
Version
2.1
Language
English
Product name
Ironstream for Splunk®/Kafka®/Elastic® for IBM Z®
Title
Ironstream™ Software for Splunk®/Kafka®/Elastic® for IBM Z® Configuration and User Guide
First publish date
2014
Last updated
2025-06-25
Published on
2025-06-25T06:34:13.730000
About this publication
Understanding Ironstream™ Software
What is Ironstream™ software?
What Data Does Ironstream™ software Collect from z/OS?
What Happens to z/OS Data in a Destination?
What Happens If Ironstream™ Software Is Unable to Deliver Data to a Destination?
Fast and Efficient Performance
Ironstream™ software Components
Data Collection Extension (DCE)
The Ironstream™ Software Desktop (IDT)
Network Monitoring Components Alerts
Supported Data Sources
Ironstream™ Software Design Roadmap
Where is your data going?
What type of z/OS environment?
What type of security?
What data types are going to be forwarded?
Does your environment require minimal data loss?
Should I manually configure Ironstream™ software or use the Configuration Tool?
Integration with Splunk Premium Applications
Configuring Ironstream™ Software Target Destinations
Setting Up Splunk for Ironstream™ Software
Overview of Setting Up Splunk
Setting Up a Non-SSL Port
Setting Up an SSL Port
Splunk Platform
Mainframe Platform
Setting Up a Splunk Index
Next Steps
Setting Up Elastic for Ironstream
Overview of Elastic Support in Ironstream™ Software
Elastic Limitations with Ironstream™ Software
Forwarding Data to Logstash
Sending Data to Logstash
Logstash Configuration
Notes
Receiving Ironstream™ Software Data in Elastic Search
Displaying Ironstream™ Software Data in Kibana
Field Mappings and Elastic Defaults
Next Steps
Setting Up Kafka for Ironstream™ Software
Overview of Apache Kafka Support in Ironstream™ Software
How does Kafka work with Ironstream™ Software?
Kafka Requirements and Limitations with Ironstream™ Software
Instructions for Downloading and Configuring Kafka on Ironstream™ Software
Downloading and Installing Kafka to z/OS OMVS Systems
Converting Binary Kafka Files from ASCII to EBCDIC (Optional)
Applying the Kafka Function to Ironstream™ Software
Providing APF Authorization for Ironstream™ Software Programs
Authorizing the Ironstream™ Software Kafka Modules
Authorizing the Java Libraries
Configuring Ironstream™ Software to Use the Kafka Producer
Setting the Ironstream™ Software DESTINATION
Example Ironstream™ Software JCL for Sending Data to Kafka Brokers
Configuring Ironstream™ Software to Use Other Kafka Producer Configurations
Using the TLS protocol with Kafka Brokers
Keystore Requirements in producer.properties for SSL client.auth Type
Kafka Delivery Guarantees Recommended by Ironstream™ Software
At-most-once Delivery Guarantee
Expected Kafka Consumer Behavior
At-least-once Delivery Guarantee
Expected Kafka Consumer Behavior
Sample Kafka Consumer Using the At-least-once Guarantee
Confirming Kafka Activity Status in Ironstream™ Software
Dynamically set Topic and Key when using Ironstream™ Software API and Kafka feature
Next Steps
Setting Up the Ironstream™ Hub
Setting Up a Generic Destination
Setting Up Cribl Stream for Ironstream™ Software
Configuring and Running Ironstream™ Software
Configuring Ironstream™ Software Components
Overview
Manually Configuring Data Sources vs. Using the Configurator Utility
Installation Verification Process (IVP)
Manually Configuring Ironstream™ Software
Set Up Your Configuration File
Set Up the Ironstream Tasks
Configure Selected Data Sources
Next Steps
About the Ironstream™ Software Configurator Utility
How the Configurator Utility Works
Manual Post-Configurator Steps
DCE, IDT, and XCF Configuration Considerations
Ironstream™ Software XCF Instances
Ironstream™ Software Forwarder(s)
DCE Tasks
Ironstream™ Software Desktop Task
Running the Ironstream™ Software Configurator Utility
Launch the Ironstream™ Software Configurator
Select the Ironstream™ Software Components to Configure
Specify the Ironstream™ Software Forwarder Parameters
Additional Actions for Ironstream™ Software Started Task Members
Specify the DCE Parameters
Additional Actions for DCE Started Task Members
Specify the Ironstream™ Software Desktop (IDT) Parameters
Additional Actions for IDT Started Task Members
Controlling Authority Levels within IDT
Configuring IDT to Use Secure HTTPS and AT-TLS
Specify the Log4j Parameters
Additional Actions for Log4j Started Task Members
Configure the Network Monitoring Components
Configure the EE Monitor
Configure FTP Control
Configure the IP Monitor
Configure the OSA Monitor
Configure the UNIX Server
Create the NMC Run-time Objects
Post Configurator: Additional Actions
About the Forwarder Tasks and Additional Tasks
Next Steps
Manually Setting Ironstream™ Software Parameters
Overview of the Configuration File
Static versus Dynamic Modification of an Ironstream™ Software Configuration
General Syntax Rules
Keywords
Comments and Columns
Parameter List Delimiters and Continuation Methods
Section Identifiers and Parameters
Configuration Records Echoed to SYSPRINT
Configuration Parameters
KEYS Section
KEYS Parameters
SYMBOLS Section
SYSTEM Section
SYSTEM Parameters
DESTINATION Section
General DESTINATION Parameters
Data Loss Prevention Parameters
Connection Parameters
SSL Enabled Parameters
SOURCE Section
SOURCE Parameters
Unconverted EBCDIC Hex Values When Using “Remove ASCII Control Characters” Format Module
Subparameter Definitions for SOURCE Data Types
Typical Ironstream™ Software Parameters
Configuration File Examples
Forwarding SMF Data (Single Indexer)
Forwarding Syslog Data using SSL (Two Indexers)
Forwarding Log4j Data (Single Indexer)
Offline Ingestion of SMF Data (Single Indexer)
Offline Ingestion of Log4j Data (Single Indexer)
Offline Ingestion of Log4j Data from a z/OS Data Set
Offline Ingestion of Log4j Data using PATTERN
Forwarding Syslog Data using SSL and Translate Table
Using KEEP ALIVE at the SYSTEM and PORT Level
Controlling Ironstream™ Software Components
Controlling Ironstream™ Software Forwarders
Starting Ironstream™ Software Forwarders
Starting an Ironstream™ Software API Forwarder
Stopping Ironstream™ Software Forwarders
Checking Ironstream™ Software Forwarder Status
Controlling the Ironstream™ Software Desktop (IDT)
Starting the IDT
Stopping the IDT
Deploying Multiple IDT Instances
Controlling the Data Collection Extension (DCE)
Starting DCE
Starting RMF for the First Time
Starting USS for the First Time
Stopping DCE
Controlling the Network Monitoring Components (NMC)
EE Monitor
FTP Control
IP Monitor
OSA Monitor
UNIX Server
Managing 64-bit COMMON storage
Dynamically Modifying a Running Ironstream™ Software Configuration
Overview
Dynamic Reconfiguration Limitations
How Ironstream™ Software Performs a Dynamic Change in the Current Configuration
Dynamic Reconfiguration Commands
Command Notes:
Dynamic Reconfiguration Procedure
Messages Issued by Dynamic Reconfiguration
Configuring Data Loss Prevention
Overview
Excluded Functionality
Ironstream™ Software System Requirements for Using DLP
Coupling Facility Log Stream
System Authorization Facility
Modifying the SSDFAUX Procedure
Configuring Ironstream™ Software DLP Parameters
When to Configuring SSDFAUX for DLP
Configuring Splunk Parameters
Best Practices When Using DLP
Configuring SMF Record Collection When Using DLP
Messages Issued by DLP
Setting Up Ironstream™ Software Data Sources
Syslog Message Filtering
Overview of Filter Modules
Syslog Message Filtering
Overview of Filter by Configuration Keywords
Importance of Mixed INCLUDE and EXCLUDE Order
Ironstream™ Software SYSLOG Continuous Offload Reporter (ISCOR)
Enabling of ISCOR
ISCOR Message Forwarding Timeline
IPL Date and Time Message
Validating SYSLOG Contains Messages Created at the IPL Date and Time
Searching for a Previous Instance of Ironstream™ Software
Ironstream™ Software SYSLOG Shutdown Message
Error Messages
Message Counting
SMF Record Filtering
Overview of SMF Record Filtering
Using the Ironstream™ Software Configuration File to Create SMF FilterConfigurations
Using IDT to Create SMF Filter Configurations
Using the READ Command
Gathering SMF Data
Supported SMF Record Types
Manually Defining SMF Filtering Configurations
Defining Custom SMF Numbers for ISV Products
PRODUCT Statement Syntax
Limiting SMF Record Selection with WHERE Search Conditions
Overview of WHERE Statements
Example WHERE Statements:
Understanding the WHERE Syntax
WHERE Search Conditions
Comparison Operators
Comparison Operands
Parenthetical WHERE Clauses
NULL Processing Command
Validating Command Syntax with a PARM Option
Configuring the SYNTAXONLY Parameter
Using the SMF Filter Configuration Builder in IDT
Required JCL to Run the SMF Filter Configuration Builder
SMFDICT DD
SMFOUT DD
Using the SMFOUT Data Set in a READ Statement
SMP/E Updates to the SMF Dictionary
Using the SMF Filter Configuration Panels
About the SMF Filter Configuration Panel
Adding a Custom Filter Configuration
Editing a Filter Configuration
Viewing a Filter Configuration
Using the READ Command to Share SMF Filter Configurations
Implementing a Custom CICS Monitor Dictionary in Ironstream™ Software
The Need for CICS Monitor Dictionary Processing
Process Flowchart: Steps to Implement
Step 1: Run DFHMNDUP
Step 2: Run SSDFGDIC (STEP010 in JCL)
Step 3: Run Assembler and Linker (STEP020, STEP030, STEP040,and STEP050 in JCL)
Using the CICS DFHMNDUP Utility to Create SMF 110 DictionaryRecords
JCL MEMBER for DFHMNDUP
Using the SSDFGDIC Utility to Process CICS Monitor Records
STEP010
STEP020, STEP030, STEP040, and STEP050
JCL MEMBER for SSDFGDIC
Understanding the SSDFGDIC Report
Assemble and Link the Statements in the Ironstream™ Software Load Library
Using The SSDFGDIC SYSIN Commands
Using SYSIN to Modify the Formatting of Monitor Fields
Using SYSIN to Duplicate MCTs
General Comments on the Control Statement Format for SSDFGDIC
System Messages for a Custom CICS Monitoring Dictionary
Sample SMF Filter Configurations
Sample Filtering for All Fields in SMF Records
Sample Filtering for Specific Fields in SMF Records
Sample Filtering for All SMF Records Using Control Statements
SYSOUT Forwarding
Using the SYSOUT Forwarding Function
Configuring Ironstream™ Software for SYSOUT Forwarding
The Selection and Forwarding Process
Format of Forwarded Spool Data
Controlling the Job and Output Scan Wait Time
Job Scan Wait Time Parameter
Output Scan Wait Time Parameter
Advanced Spool Data Forwarder Options
Fields Forwarded from SYSOUT to Ironstream™ Software Destinations
SYSOUT Selection and Filtering
Job and Data Set Selection
Filtering Criteria
Job and Data Set Class Exclusion
Selection and Filter Keywords and Rules
Job Selection Keywords
Data Set Selection Keywords
Job Filtering Selection Keywords
Values When Using the PHASE Keyword for Filtering Jobs
Using the Advanced PRINT Data Block Parameters
Using Advanced Options to Process Log4j Data
Preserving SYSOUT State Across Restarts
Configuring SYSOUT RESTART
Step 1: Allocate and Initialize a Data Set
Step 2: Add a DD SYSOFILE Statement
Step 3: Optionally Add SYSOUT_RESTART to the SSDFCONF File
Syntax Rules for Adding SYSOUT_RESTART to the Configuration File
Changing SYSOUT Parameters When Stopping and Restarting Ironstream™ Software
Restart Limitations
SYSOUT Forwarding Parameter Examples and Sample Output
SYSOUT Data Forwarding Examples
Select Output from Two Separate DD Names of a Job
Select MSGUSR from all Jobnames Beginning with ‘CICS’
Select JESMSGLG from a Production IMS Message Processor
Select JESYSMSG from jobs DFHSM and TM1
Select Submitting JCL from Started Tasks
JSON Output Using PRINT_SEND
Alerts and SyslogD Forwarding
Overview of Syncsort Network Management Components
Configuring ZEN for Ironstream™ Software
ZEN Component Alert Generation
The OSA MONITOR (ZOM)
The LINUX MONITOR (ZLM)
FTP CONTROL (ZFC)
The EE MONITOR (ZEM)
The IP MONITOR (ZIM)
Routing SyslogD Messages to Ironstream
Configuring ZEN to Route SyslogD Messages to Ironstream™ Software
More Information about Alerts and SyslogD Forwarding
DB2 Data Forwarding
Overview of DB2 Tables
Configuration for DB2 Table Data
DB2 Definitions
Sample SSDFTRIG
Sample SSDFPROC
Ironstream™ Software DB2 Data Definitions
Enabling Data Loss Prevention in Splunk for DB2 Forwarding
Configuration File Example for DB2 Data with DLP
Command to Run Ironstream™ Software for DB2 Data with DLP
Sequential File Forwarding
Capturing Sequential Data
Sequential File Forwarding Example
Choosing the Data Output Format
Data Translation
Using a Translation Table
Batching FILELOAD Data
FILELOAD Batching Control Statements
BATCH_COUNTER Usage Notes
BATCH_RECORDS Usage Notes
FILELOAD Batching Example
System State Forwarding
Overview
Configuring Ironstream™ Software for System State Forwarding
System-level Data Fields Forwarded to Destinations
Configuring and Using the Ironstream™ Software API
Overview of the Ironstream™ Software API
Single-send versus Multi-send API
System Requirements
Defining the IRONSTREAM_API Data Type
Data Type Parameters
CLASS, TYPE, and SUBTYPE Parameters
CLASS Configuration Behavior
CLASS Configuration Example
Ironstream™ Software API Configuration Example
Using the Single-send API
Single-send API Parameters
RACF Authorization for the Single-send API
Using the Single-send SSDFAPI Routine
Single-send API Environment
Register Conventions
Single-send API Parameter List Format
Performance and Maintenance Considerations
Linking SSDFAPI Into a Load Module
Starting a Single-end API Instance
Using the Single-send API in CICS
Define the Ironstream™ Software API Parameters in a CICS Program
Calling the Single-send API in CICS
Using the Multi-send API
Multi-send API Request Types
INIT Request
SEND Request
TERM Request
Multi-send API Parameters
RACF Authorization for the Multi-send API
Using the Multi-send SSDFPAPI Routine
Multi-send API Parameter List Format
Performance and Maintenance Considerations
Linking SSDFPAPI Into a Load Module
Starting a Multi-send API Instance
Troubleshooting the Ironstream™ Software API
Return Codes and Reason Codes Generated by the Ironstream™ Software API
Handling Data Store Full Conditions
Ironstream™ Software API Coding Examples
Single-Send API Examples
Assembler Single-send API Examples
C Single-send API Example
COBOL Single-send API Example
REXX Single-send API Example
COBOL on CICS Single-send API Example
Multi-send API Coding Examples
Assembler Multi-send API Examples
C Multi-send API Example
COBOL Multi-send API Example
REXX Multi-send API Example
Ironstream™ Software API and KAFKA - Dynamic Topic and Key Support Feature Details
How the feature works?
Transient API Details
API Samples
Transient API samples
Persistent API samples
31-bit Sample Ironstream™ Software Configuration for using the API Source and a Kafka Target
64-bit Sample Ironstream™ Software Configuration for using the API Source and a Kafka Target
Setting Up Log4j
Overview of Log4j
Defining the Log4j Parameters
Sample Log4j Configurations
SDFAppender Sample in log4j.xml
SDFAppender Sample in log4j.properties
SDF2Appender Samples for Log4j 2.x
How to use PATTERN in the Log4j Reader Facility
IMS Log Record Forwarding
Overview of IMS Log Record Forwarding
Excluded Functionality
Synchronous versus Asynchronous IMS Log Record Capture
Synchronous IMS Log Gathering
Activating the log write Exit
Forwarder Task JCL
Asynchronous IMS Log Gathering
IMS Log Record Extraction Process
Using the Category Keyword
IMS Log Record Processing
IMS Log Record Field Descriptions
Messages Issued by IMS Log Records
LOGREC Forwarding
Overview
Configuring Ironstream™ Software for LOGREC Forwarding
Status report by LOGREC Type
Data Fields Forwarded by LOGREC Type to Destinations
Logstream Forwarding
Understanding the Logstream as used by Ironstream™ Software
Restart Caveats
How to Define the Forwarding of a Logstream
Console Commands
Logstream Forwarding Processing Configuration Keywords and Parameters
Messages Issued by Logstream Forwarding
Setting Up the Data Collection Extension Data Types
Configuring the DCE Parameters
Overview of DCE
DCE Configuration Files
Global Parameters
Ironstream™ Software Cluster Parameters
Include Parameter Group
Syntax for DCE Configuration Parameters
Ironstream™ Software Forwarder Configuration Files
Setting Up USS File Collection
Overview of USS File Collection
Adjust File Monitoring and Offloading
Scan for Duplicate Files
Tail Volatile Files
Detect Multi-line Files
Track Rolled (Archived) Log Files
Dynamic Administration of USS Processing Using IDT
Flexible Start Types
USS File Offload Operational Diagram
Summary of the DCE USS Offload Functions
Configuring DCE for USS File Offload
USS Defaults Parameters
USS Filter Parameters
Notes on USS File Filtering
USS Directory Parameters
Duplicate USS File Detection
How Duplicate USS File Detection Works
Modifying the Duplicate File Detection Behavior
USS File Tailing Process
How It Works
Tailing Volatile Files
Tracking Rolled USS Log Files
How It Works
Establishing the Rolled USS File Tracking Behavior
Filtering Rolled Log Files
Verifying Checksum Lengths
Constraints When Using Tracking Rolled Log Files
Detecting and Controlling Multi-line Log Records
How It Works
Recognized Formats of log4j-type Records
Recognized Formats of JavaTrace-type Records
Dynamically Modifying USS Processing
Accessing the Ironstream™ Software Desktop USS panels
Displaying the USS Files Status Panel
Dynamically Modifying USS Settings
Setting Up the RMF Data Forwarder
Overview of the RMF Data Forwarder
Configuring the RMF Data Forwarder
Configuring DCE RMF Parameters
Define RMFSettings
Setting the ScanFrequency
Defining Security Settings
Changing the RMF User ID or Password
Setting the RMF Filters in IDT
About the RMF Filters Panel
Sample Scenario for Setting RMF Filters
Step 1: Open the RMF Filters Panel
Step 2: Activate Filtering for Volumes
Step 3: Specify the Metrics Collected for Selected Volumes
Step 4: Specify the Metrics Collected per LPAR
Accessing RMF Enclave Attributes
Workload Manager Active Policy
Integration with Splunk Premium Applications
Splunk Enterprise Security and Ironstream™ Software
About Splunk Enterprise Security and Ironstream™ Software
Ironstream™ Software Enterprise Security Technology Add-on (TA)
Intrusion Detection
Splunk ES Visibility
TSO Log-on Activity
Splunk ES Visibility
TSO Account Activity
Splunk ES Visibility
FTP Sessions
Splunk ES Visibility
FTP Change Analysis
Splunk ES Visibility
IP Traffic Analysis
Splunk ES Visibility
Network Management/User-Defined Notification
Splunk ES Visibility
Troubleshooting Ironstream™ Software
Ironstream™ Software Commands
Overview
Management Commands
MODIFY Commands
BLOCKPRINT
DEBUG
DUMP
LIST
CAPTURE
CBS
MODULES
QUEUES
SYSOUT DATASETS
TRACE
RECONFIGURE
VALIDATE
EXECUTE
RECORDPRINT
RESTART
STATUS
Auxiliary Commands
STATUS
DEBUG
SMF Real-time INMEM Commands
REFRESH
DISCONNECT
CONNECT
STATUS
Operational Considerations
Message Flood Automation and Syslog Message Collection
Message Forwarding with SDFLOG and MPF Integration on z/OS
Network Contention
Data Store Filling or Full Condition
Recommended Data Store Configuration Guidelines
Ironstream™ Software Messages
Overview of Ironstream™ Software Messages
Ironstream™ Software Messages
Data Collection Extension Messages
Ironstream™ Software SYSLOG Continuous Offload Reporter (ISCOR) Messages
Diagnostics and Contacting Precisely Support
Before Calling Precisely Support
Searching the Precisely Knowledge Base
Contacting Precisely Support
Ironstream™ Software Audit Reporting
Using the Ironstream™ Software Data Usage Reporter
Overview
Configuring the Report Parameters
Basic JCL to Produce a Printed Report
SYSIN Parameters
SYSIN Syntax
SELECT Parameters
REPORT Parameters
Using the Report TRACE Facility
CSV File Report Format
Overriding the Default SMF Record Number
Ironstream™ Software Configuration File
Copying and Renaming the Default SMF Modules
System Messages for the Data Usage Reporter
Forwarded Data Formats
Syslog Format
FILELOAD Format
SYSOUT Format
Log4j Format
Alert Format
SyslogD Format
The SSDFCPR Utility
Overview of SSDFCPR
Executing SSDFCPR
Notices
Trademarks