There are two basic types of unload engines:
- A special version of the an Apply Engine that reads the source datastore locally and writes records that can be loaded to the same or different type of target database, using a database load utility. Often this engine can use unmodified versions of the "mapping" PROCS used by the Apply Engine and only the target datastore type will be modified to output comma separated records to one or more individual files that then are used as input to the chosen database load utility. The engine will often have to be parametrized to allow specification of the particular source descriptions (tables, segments, records, etc). Then the parameter driven engine will be run as many times as needed to generate files for each of the specified descriptions.
- The second type of unload engine typically runs on the same platform as the Apply Engine. Instead of connecting to a remote Publisher it connects remotely to the source database, reads the source from top to bottom and then using either the same mapping PROCS as the Apply engine or a simple REPLICATE script, writes directly to the Target datastore, be it a traditional RDBMS or Kafka or HDFS.
Contact Support to discuss both the benefits and techniques for implementing and perhaps more importantly maintaining, a load/refresh Engine solution.