Integration of data of multiple types (structured, unstructured, semistructured) and formats is one of the major challenges of big data, particularly when the requirement is for real-time data analysis, writes Wikibon Principal Research Contributor and Big data Analyst Jeff Kelly in “Attunity Adapts Change Data Capture for Big Data Analytics”. To truly be “real time” analytic databases must provide data that is complete and up-to-date and not just interactive query on data that is hours old. Traditional ETL (extract, transform and load) technology, basically a batch process, is not adequate, and many analytic database vendors claiming real-time analytics capabilities really provide something less than real time.

Burlington, Mass.-based Attunity  has adapted change data capture (CDC), which detects changes in source data and updates target analytic databases with only those changes. Its TurboStream CDC is enhanced to provide limited amounts of pre-processing including data consolidation of data from source systems before updating the target databases. The result is linear real time updates to target analytic databases. It also is data source and database target agnostic, allowing it to partner with multiple database vendors including Actian Vectorwise, HP Vertica, IBM Netezza, Teradata, and AWS RedShift. It also has acquired Hayes Technology Group, whose data replication technology is optimized for SAP HANA.

Jeff’s full Alert is available without charge on the Wikibon Web site. IT professionals are invited to register for membership in the Wikibon community, which allows them to participate in Wikibon research, comment on Alerts, and post their own tips, questions, and research.

Graphic courtesy Attunity.