Data applications are now being built on different platforms for different use cases, and artificial intelligence is one of the key drivers behind which compute engines for AI an enterprise may want to use.

Databricks Inc. and Snowflake Inc. are in the hurricane’s eye of this change as the point of control shifts from the database management system to the governance layer, a transformation driven by increased use of open table formats.

Rob Strechay and Sanjeev Mohan of theCUBE Research discuss compute engines for AI during Supercloud 7.

“You couldn’t buy Snowflake compute and [only] some of the storage, but now with table formats that game has changed,” said Sanjeev Mohan (pictured, second from right), principal at SanjMo and analyst for theCUBE Collective. “If I can build reports and dashboards very easily in Snowflake with an amazing user experience, great. But if I want to do AI and machine learning, maybe I should use Databricks. If I want to do my data cleansing and data prep, I should use Apache Spark. It’s not making copies of the data and moving it around, but it’s changing the compute engine.”

Mohan spoke during an AnalystANGLE Panel segment on the Supercloud 7: Get Ready for the Next Data Platform event. He was joined on the panel by Sarbjeet Johal (far right), founder and chief executive officer of Stackpane, along with theCUBE Research’s John Furrier (far left) and Rob Strechay (second from left), and they discussed key emerging trends in enterprise use of database services.

Selecting the right compute engines for enterprise use

Behind this changing landscape for database vendors such as Snowflake and Databricks is an evolving strategy for how both will meet customer needs for AI application development and business intelligence. This will involve serving a client base that may pick and choose services within a newly formed ecosystem, as seen in recent industry survey data provided by Enterprise Technology Research and SiliconANGLE.

“What they’re building is several apps, and those apps are coming together as part of an ecosystem, the same way that AI is coming together with collections of agents,” Strechay said. “When you start to look at it, I think Databricks has been really excelling in the AI space with data science crews. Snowflake has always been in the business intelligence analytics realm. I think there is some crossover between the two…and we see that in the data where 81% are using Snowflake for the BI analytics and data warehousing storage, but you have 61% of the people who responded also using Databricks for that.”

There is also a shift with data now residing outside of the traditional data stores managed by giants such as Databricks and Snowflake. Data’s value will now be weighed based on location and time, according to Johal.

“That data was sitting in some relational database earlier, but now it can be sitting in the small language models and combinations of those small language models will be woven into our next generation of applications,” he explained. “Data has life span, and, as data ages, it loses value, but some data loses value more than other data. So that demarcation of time, what data loses how much value…will determine what goes into the language models and what stays in the traditional databases.”

Shifting landscape for governance layer

ETR’s survey data highlighted how security and governance have become primary decision points for customers, with 70% indicating that no platform decisions are made without consideration for governance. As AI continues to exert influence in enterprise IT, open protocols such as table formats will likely become key factors in how data governance is structured.

“The trend is here, open data formats are driving the landscape governance layer shifting and that’s enabling AI intelligent data applications,” Furrier said. “That’s the trajectory.”

The influence of AI is also reshaping how data products are defined and used in compute engines according to Mohan.

“We talk about all the compute engines, where you’ve got Star, Dios, Pandas, Spark, you’ve got all of that,” he said. “But one of the compute engines is…the LLM. And that LLM is now driving more of the semantic search kinds of applications. The AIs, such as conversational chatbots and assistance co-pilots, are sitting on top of the LLM, and reports, dashboards, machine learning algorithms are all sitting on top of the rest of compute. They’re all data products on top.”

Here’s the complete video interview, part of SiliconANGLE’s and theCUBE Research’s coverage of the Supercloud 7: Get Ready for the Next Data Platform event:

Photo: SiliconANGLE