Databricks Inc. today introduced two new products, LakeFlow and AI/BI, that promise to ease several of the tasks involved in analyzing business information for useful patterns.

LakeFlow is designed to simplify the process of building data pipelines. AI/BI, in turn, is a business intelligence platform with a built-in artificial intelligence chatbot. Databricks unveiled the offerings alongside a new release of its Mosaic AI toolkit for building AI models, which is based on assets it obtained through a $1.3 billion startup acquisition last year.

Real-time data pipelines

LakeFlow, one of several new products that Databricks debuted today at its Data + AI Summit in San Francisco, also incorporates technology from an acquisition. Databricks spent more than $100 million last year to buy a startup called Arcion Inc. The startup developed a tool for moving information between applications that is now one of LakeFlow’s core building blocks.

Companies can use LakeFlow to build data pipelines, software workflows that can automatically transfer information from one system to another. Often, such pipelines also modify the information to make it easier to process. Databricks says that LakeFlow provides features for both those tasks.

The platform includes connectors that can pull information from databases as well as popular business applications such as Salesforce, Google Analytics and SharePoint. The connectors are integrated with an existing Databricks tool called Unity Catalog. It allow administrators to track how employees use the information processed by LakeFlow.

After extracting data from a system, LakFlow can enrich it with information from external sources, change its format and apply other modifications. Developers can write the code that performs those modifications in SQL or Python. The feature is powered by Delta Live Tables, Databricks’ ETL, or extract, transform and load, tool.

Companies with advanced requirements may configure LakeFlow data pipelines to process information in near-real-time. According to Databricks, this is facilitated by a feature called Real Time Mode that can be enabled without major code changes.

Customers can monitor the health of their data pipelines using a set of features dubbed LakeFlow Jobs. When a technical issue is detected,  data about the incident is sent to alerting platforms such as PagerDuty that can notify administrators. LakeFlow Jobs also promises to ease several other tasks including the process of deploying newly created data pipelines in production.

AI-powered dashboards 

The second product that Databricks introduced today, AI/BI, has two main components. AI/BI Dashboards is a module that makes it possible to turn business data into visualizations through a low-code interface. The platform’s second component, Genie, is a chatbot that allows users to ask natural language questions about those visualizations.

“A truly intelligent BI solution needs to understand the unique semantics and nuances of a business to effectively answer questions for business users,” said Databricks co-founder and Chief Executive Officer Ali Ghodsi (pictured). “We believe this requires a different approach than how BI software has been designed in the past — one that places an AI system at the center of the architecture and is designed to take advantage of the AI systems’ strengths as well as complementing their weaknesses to tackle the challenges of understanding and learning these nuances.”

Under the hood, AI/BI is powered by a collection of AI agents that are each optimized for a different set of tasks. One is responsible for visualizing users’ data, while another helps explain the information. Databricks has also created an AI agent that can explain the results of an analytics project.

The company says that its agents learn from the information they process. For example, if a user specifies how AI/BI should interpret the term “churn” when analyzing customer retention data, the platform can reuse that knowledge in future analytics projects. Databricks says its AI agents also learn from other information and technical assets in a company’s analytics environment.

Like Dataflow, AI/BI integrates with the company’s Unity Catalog product. Administrators can use the latter tool to monitor how AI/BI users interact with business information. According to Databricks, Unity Catalog doubles as a data reliability tool: it allows workers to check that information they’re using in an analytics project was processed correctly.

Alongside data visualizations, AI/BI provides access to a built-in chatbot called Genie. It enables workers to query data in a natural language format. Databricks says that the chatbot can learn from user feedback improve the quality of its output.

Photo: Databricks