Google Cloud Storage (GCS) is a highly durable, scalable, and secure object storage service provided by Google Cloud. It supports a wide range of use cases including analytics, backup, archival, and serving static content. GCS offers fine-grained access control, lifecycle management, and strong consistency for all operations.
The GCS Target Datastore is identified by a URL composed of the bucket name and object key.
GCS supports file rotation, enabling analytics on static file sets while avoiding conflicts with files being updated. Rotation can be triggered based on time intervals or file size.
Environmental Requirements
To use Google Cloud Storage as a target datastore, the SQData Apply Engine requires the Google Cloud C++ Client Libraries.
If building from source, follow the official Google Cloud C++ setup guide. Only the storage package is required.
cmake .. -DCMAKE_BUILD_TYPE=Release
-DBUILD_SHARED_LIBS=ON
-DGOOGLE_CLOUD_CPP_ENABLE=storage
-DBUILD_TESTING=OFF
-DGOOGLE_CLOUD_CPP_ENABLE_EXAMPLES=OFF
-DCMAKE_INSTALL_PREFIX=/home/jsmith/gcp-sdk-cpp-install
-DCMAKE_TOOLCHAIN_FILE=$HOME/vcpkg/scripts/buildsystems/vcpkg.cmakeexport LD_LIBRARY_PATH=<GCS_SDK_INSTALL_PATH>/lib:<SQDATA_INSTALL_DIR>/lib:$LD_LIBRARY_PATHAuthentication
Google Cloud Storage authentication for the SQData Apply Engine is handled via Google Application Default Credentials (ADC). Refer to the Google Cloud authentication guide. The simplest setup is through the gcloud CLI.