When implementing the runMultiFileLoader job initiator script strategy, the transformation and load workflow is as follows:
-
The runMultiFileLoader job initiator script, runs the configuration file, creates the
required named pipes based upon the number of standard input files, and submits an
instance of the Connect ETL transformation and load job for each named pipe.
Each parallel transformation and load job reads the named pipe and calls the Connect ETL transformation task and the custom load task
-
Using the named pipes as input, the Connect ETL task transforms the data.
The following example reformats the EBCDIC-encoded data on the named pipe into ASCII-encoded standard input.
- Using the standard input file as input, the Connect ETL custom load task, calls the database load script.
- Using environment variables, the database load script builds the vsql command that loads the data from the standard input file directly into initiator nodes of the Vertica database.