RunLargeFileLoader Job initiator implementation strategy - Connect_ETL - Latest

Connect ETL Vertica Load Guide
Product type
Software
Portfolio
Integrate
Product family
Connect
Product
Connect > Connect (ETL, Sort, AppMod, Big Data)
Version
Latest
ft:locale
en-US
Product name
Connect ETL
ft:title
Connect ETL Vertica Load Guide
Copyright
2025
First publish date
2003
ft:lastEdition
2025-05-21
ft:lastPublication
2025-05-21T18:34:10.033000

When implementing the runLargeFileLoader job initiator script strategy, the partition, transformation, and load workflow is as follows:

  1. The runLargeFileLoader job initiator script, runs the configuration file, creates the required named pipes, and submits the Connect ETL partition job, which reads the input file set and initiates the Connect ETL partition task.
  2. The Connect ETL partition task partitions the input file set into the applicable named pipes.

  3. The runLargeFileLoader job initiator script submits one Connect ETL transformation and load job for each partition.

    Each parallel transformation and load job reads the named pipe and calls the Connect ETL transformation task and the custom load task.

  4. Using the named pipes as input, the Connect ETL task transforms the data.

    The following example reformats the EBCDIC-encoded data on the named pipe into an ASCII-encoded standard input.

  5. Using the standard input file as input, the Connect ETL custom load task, calls the database load script.
  6. Using the parameter and environment variable values available to the system after the runLargeFileLoader job initiator script runs the configuration file. The database load script builds the vsql command that loads the data from the standard input file directly into initiator nodes of the Vertica database.

When implementing the runMultiFileLoader job initiator script strategy, the transformation and load workflow is as follows:

  1. The runMultiFileLoader job initiator script, runs the configuration file, creates the required named pipes based upon the number of standard input files, and submits an instance of the Connect ETL transformation and load job for each named pipe.

    Each parallel transformation and load job reads the named pipe and calls the Connect ETL transformation task and the custom load task.

  2. Using the named pipes as input, the Connect ETL task transforms the data.

    The following example reformats the EBCDIC-encoded data on the named pipe into ASCII-encoded standard input.

  3. Using the standard input file as input, the Connect ETL custom load task, calls the database load script.
  4. Using environment variables, the database load script builds the vsql command that loads the data from the standard input file directly into initiator nodes of the Vertica database.