Data observability startup Acceldata Inc. today added artificial intelligence features to its platform with a “copilot” that it says lets DataOps teams tailor AI assistants to their unique technology and business environment.

That includes the ability to set guardrails that ensure that business context, regulatory requirements, and human oversight are factored into the equation.

Incorporating technology from its acquisition last fall of Bewgle Inc., a technology developer for analyzing unstructured data, Acceldata’s co-pilot analyzes and generates alerts on anomalies in data freshness, data profiling and data quality to improve data trustworthiness. Using large language model technology, it automatically generates human-readable descriptions of data assets, policies and rules to improve communication between technical and business constituents.

Tuned with retrieval-augmented generation, the assistant can determine “what data sets are being used, what is being included in reports and what is not included,” said Rohit Choudhary, co-founder and chief executive of Acceldata. “People often don’t know where their data is being used. This tells you about the data that’s most important to you.”

Learned behavior

The company said the AI model learns patterns in how data is used to customize alerts and reports. For example, the failure of a report to arrive at its usual time or changes in the demographic profile of a customer cohort can indicate data drift. “All of these are now being done automatically, not figuring out after outcomes have been seen,” Choudhary said.

The assistant can also learn cost consumption patterns to help companies better control their cloud costs and forecast consumption based on learned behavior. In a cloud data warehousing scenario, “the user community never had cost considerations before, but those costs add up because Snowflake is consumption-based,” Choudhary said, referring to one popular cloud data warehousing vendor. “We can find out which SQL queries are poorly formed with machine analysis instead of scrounging through lines of SQL.” Acceldata also analyzes intermediate data assets, notebooks and programmatic access.

LLM features streamline policy creation and reduce the risk of errors. “We’re looking at the pattern of data usage to learn how different data sets are being put together,” Choudhary said. “An employment contract has a set of fields and documents that you should look at. Any data set that represents that data can automatically be verified. If somebody has made a wrong entry into an upstream employment database, we can automatically discover that it is not following the pattern without creating a manual rule. We then have the ability to create a policy.”

Acceldata has raised more than $105 million from several prominent investors. It open-sourced its platform in late 2022 and makes money by selling proprietary extensions aimed at enterprise customers.

Image: Unsplash