The data gathered by business applications found new life with the advent of large scale data analysis. Connect CDC SQData had already solved the change data capture challenge. Two new technologies have led the way to platforms and data storage techniques that support all sorts of data analytics:
- Kafka is a robust Open Source clustered distributed streaming platform. It has become the acknowledged leader for real-time streaming data due to it's performance, reliability and low cost. Many other products in this space now support the Open Source librdkafka API in order to position themselves as viable proprietary or managed cloud based alternatives.
- HDFS is one component of the framework called Hadoop, is a distributed Java-based file system typically used to store very large volumes of data.
Connect CDC (SQData) supports both as Targets and automates some of the maintenance formerly associated with management of the Schemas that describe the structure of the data.