Amazon S3 Datastores - connect_cdc_sqdata - Latest

Connect CDC (SQData) Apply engine
Product type
Software
Portfolio
Integrate
Product family
Connect
Product
Connect > Connect CDC (SQData)
Version
Latest
ft:locale
en-US
Product name
Connect CDC (SQData)
ft:title
Connect CDC (SQData) Apply engine
Copyright
2026
First publish date
2000
ft:lastEdition
2026-03-26
ft:lastPublication
2026-03-26T20:24:24.831000
L1_Product_Gateway
Integrate
L2_Product_Segment
Data Integration
L3_Product_Brand
Precisely Connect
L4_Investment_Segment
Application Data Integration
L5_Product_Group
ADI - Connect
L6_Product_Name
Connect CDC

Amazon Simple Storage Service (Amazon S3) is a highly scalable, secure, and performant object storage service. It supports a wide range of use cases including backup and restore, archiving, and enterprise applications. Amazon S3 also provides robust management features to help you optimize, organize, and control access to your data in alignment with business and compliance requirements.

The S3 Target Datastore is identified by a url composed of the bucket name and object key.

S3 supports file rotation, enabling analytics on static file sets while avoiding conflicts with files being updated using replication. Rotation can be triggered based on time intervals, file size, or record count.

Environmental Requirements

To use Amazon S3 as a target datastore, the SQData Apply Engine requires the AWS SDK for C++ shared libraries.

If building the AWS SDK for C++ from source, follow the official AWS setup guide. Only the S3 service package is required.

Example build command:
BASE_DIR="/usr/local"

sudo yum update

sudo yum install -y cmake3 ninja-build clang perl golang libpsl-devel

# Clean up previous directories
rm -rf \
    "$BASE_DIR/aws-lc" "$BASE_DIR/aws-lc-install" \
    "$BASE_DIR/curl" "$BASE_DIR/curl-install" \
    "$BASE_DIR/aws-sdk-cpp" "$BASE_DIR/aws-sdk-cpp-install"

# Clone and build aws-lc
git clone --depth 1 -b fips-2022-11-02 https://github.com/aws/aws-lc "$BASE_DIR/aws-lc"
cd "$BASE_DIR/aws-lc"
mkdir build && cd build
cmake -G Ninja \
    -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_INSTALL_LIBDIR=lib \
    -DBUILD_SHARED_LIBS=ON \
    -DCMAKE_INSTALL_PREFIX="$BASE_DIR/aws-lc-install" ..
cmake --build .
cmake --install .
cd "$BASE_DIR"

# Clone and build curl
git clone --depth 1 -b curl-8_5_0 https://github.com/curl/curl "$BASE_DIR/curl"
cd "$BASE_DIR/curl"
autoreconf -fi
mkdir build && cd build
../configure \
    --enable-warnings \
    --enable-werror \
    --enable-shared \
    --disable-static \
    --with-openssl="$BASE_DIR/aws-lc-install" \
    --prefix="$BASE_DIR/curl-install"
make
make install
cd "$BASE_DIR"


# Clone and build aws-sdk-cpp
git clone --depth 1 -b 1.11.603 --recurse-submodules https://github.com/aws/aws-sdk-cpp "$BASE_DIR/aws-sdk-cpp"
cd "$BASE_DIR/aws-sdk-cpp"
mkdir build && cd build
cmake -G Ninja \
    -DCMAKE_PREFIX_PATH="$BASE_DIR/curl-install;$BASE_DIR/aws-lc-install" \
    -DBUILD_ONLY="s3" \
    -DBUILD_SHARED_LIBS=ON \
    -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_INSTALL_PREFIX="$BASE_DIR/aws-sdk-cpp-install" \
    -DAUTORUN_UNIT_TESTS=OFF ..
cmake --build . 
cmake --install . 
cd "$BASE_DIR"

After building, make the libraries accessible to the SQData Apply Engine:

export LD_LIBRARY_PATH=$BASE_DIR/aws-sdk-cpp-install;$BASE_DIR/aws-lc-install;$BASE_DIR/curl-install:<SQDATA_INSTALL_DIR>/lib:$LD_LIBRARY_PATH

Authentication

Amazon S3 authentication for the SQData Apply Engine is handled using the AWS SDK for C++ mechanisms. Refer to the AWS credentials guide. The recommended setup is to use AWS CLI.