Managing malformed input records - 23.1

Spectrum Dataflow Designer Guide
Version
23.1
Language
English
Product name
Spectrum Technology Platform
Title
Spectrum Dataflow Designer Guide
First publish date
2007
Last updated
2024-05-09
Published on
2024-05-09T23:01:03.226155

A malformed record is one that Spectrum Technology Platform cannot parse. When Spectrum Technology Platform encounters a malformed record, it can do one or more of these tasks:

  • Stop the job
  • Continue processing
  • Continue processing until a certain number of bad records are encountered
  • Continue processing but write bad records to a log file (via an optional sink stage)
Note: Malformed records functionality is limited to sources configured to read from files local to the server and that do not have sorting configured. When a source is configured with either a remote file or with sort fields and the source encounters a malformed record, the job will stop regardless of the configuration for malformed records.

To manage malformed records,

  1. Open the flow on the canvas.
  2. Add a malformed records sink in your flow.
    1. Create your job by defining your input file and source stage and adding services and subflows to your flow.
    2. You can:
      • Connect a sink stage to the optional output port on the source stage in your flow. The optional port is the clear output port just beneath the black output port on your source stage. If you mouse over this port, you will see a tool tip that says, "error_port." Malformed records go to this sink.
      • Connect nothing to the optional output port on the source stage in your flow, ignoring all malformed records.
  3. By default, processing stops at malformed records. This default behavior can be changed in your Advanced configuration options or in Spectrum Management Console. Regardless of your system's default behavior, you can override the default behavior for a job by following these steps:
    1. Open the job in Spectrum Flow Designer.
    2. Within an open job, go to Edit > Job Options.
    3. Select either Do not terminate the job on a malformed record or select Terminate the job after encountering this many malformed records and enter the number of malformed records you will allow a job to encounter before terminating.