Databricks Inc. today introduced Genie Code, an artificial intelligence agent designed to automate complex data engineering and analytics tasks. The move extends the rapid evolution of agents from software engineering into enterprise data workflows.

The data science platform maker also announced the acquisition of Quotient AI Inc., an early-stage startup focused on evaluating and diagnosing failures in AI agents.

Genie Code aims to move data teams beyond code assistance toward systems that autonomously plan and execute data tasks under human supervision. It addresses fundamental differences between traditional coding assistants and systems designed specifically for data work, said Ken Wong, senior director of product management at Databricks.

“Coding agents are focused on this problem of completing code,” he said, “and in many ways, code is a means to the end in data.”

Though large language models and coding assistants have improved dramatically in the past year, Wong said they often struggle with data engineering tasks because they lack access to the contextual information that data systems require.

“Data context is not captured in source files,” he said. “It’s a different type of problem. Data context is more dynamic and kind of messy.”

Interpreting intent

Genie Code addresses that challenge by integrating deeply with enterprise data systems and governance layers, Databricks said. The system interprets organizational data context, historical query patterns and business definitions to translate user intent into definitions needs for production data workflows.

“If you’re trying to understand what annual recurring revenue means in an organization, it may be in source files, but it’s more likely to be in historical query patterns,” Wong said. “We bubble that up to the LLM so the agent can properly translate user intent into how it’s manifested in the data system the enterprise is working on.”

Databricks’ Unity Catalog provides the governance layer, which ensures that agents operate within enterprise security and compliance boundaries. Genie Code is designed to work primarily within the Databricks platform, although organizations can connect external data sources through Unity Catalog.

Databricks said agents are changing the role of data professionals by shifting their work from writing code to supervising and orchestrating AI agents. “We see that as the future,” Wong said.

The biggest productivity gains come not only from development but also from operational maintenance of data systems, he said.

“A huge part of most data practitioners’ work is operational,” Wong said. “It’s not just creating a pipeline, but keeping it running and troubleshooting issues and upstream changes.” He expects agents like Genie Code to absorb much of that operational burden over time.

Hanlin Tang, chief technology officer for neural networks at Databricks, said the system has already begun reshaping his own workflow as a data scientist.

“I used to write a bunch of code to clean up tables and data, find missing values, impute them, and then do a transformation,” he said. “It’s grungy work.” Genie Code has automated much of that preparation, “so I can do the core machine learning I’m good at.”

Reinforcement learning

Quotient AI’s technology will help Databricks improve the reliability and performance of agent-based systems, Tang said. The company, which was founded by the developers of GitHub Inc.’s Copilot, uses reinforcement-learning models that analyze agent behavior and identify where processes break down.

“Understanding why agents fail is a hard problem,” Tang said. “It’s a complex system that can call tools and has a memory. There might be two models talking to each other. Quotient has done a good job of using reinforcement learning to train custom models that can look at an agent’s activity and say, ‘This agent made the wrong tool call.’”

Databricks plans to integrate Quotient’s technology into Genie Code and also its broader agent platform for use in production scenarios.

“Even after you deploy an agent, you want to keep monitoring,” Tang said. “You want to know what mistakes it makes, especially as the environment changes over time.”

Image: Adobe Stock