Universal data storage platform provider Vast Data Inc. has outlined ambitious plans to power a new generation of artificial intelligence models that do more than simply parrot responses when prompted.

Instead, it envisions a future where AI is able to automatically drive new discoveries and understandings that were formerly unknown. To deliver on this goal, the company today announced a new, global data infrastructure for AI called the Vast Data Platform, which unifies data storage, database and virtualized compute engine services in a scalable system.

Built with the entire spectrum of natural data in mind – including unstructured and structured data in the form of video, imagery, free text, data streams and instrument data – the Vast Data Platform will help enterprises transform their AI initiatives, the company said. The platform will enable them to access and process data from any private or public cloud data center, understand natural data through a queryable semantic layer that’s embedded directly into it, and continuously compute data in real time.

Four moving parts

The secret sauce behind the Vast Data Platform is its novel components, which consist of a unified and global datastore, a new kind of semantic database layer and a specialized AI computing engine.

According to the company, the Vast DataStore will serve as the foundation of the Vast Data Platform. Vast DataStore is a scalable storage architecture for unstructured information that eliminates the need for storage tiering.

By exposing file storage and object storage interfaces, it is uniquely able to bring archive economics to flash storage, making it suitable for archive applications for the first time. “Resolving the cost of flash storage has been critical to laying the foundation for deep learning for enterprise customers as they look to train models on their proprietary data assets,” the company said.

Another innovation is the Vast DataBase, a semantic database layer that combines the capabilities of a traditional database, data warehouse and data lake into one platform. It’s a distributed and unified database management system that supports the capture and cataloging of data plus the ability to analyze and correlate that information, in real-time, the company said. In other words, it’s built for fast data capture and rapid querying at scale, making it possible to refine and enrich raw unstructured data into structured and queryable information.

As for the Vast DataEngine, it serves as a global function execution engine that consolidates geographically distributed on-premises data centers and cloud regions into a single computational framework, the company explained. It supports programming languages such as Python and Structured Query Language, introduces event notifications and materialized and reproducible model training. The net result is that it becomes much simpler to manage AI pipelines, according to Vast.

Finally, Vast DataSpace is a global namespace that permits any location to store, retrieve and access information from any other location. This can be done with high performance while enforcing security policies across every access point, the company said. With Vast DataSpace, the Vast Data Platform can be deployed in any environment, including on-premises data centers, cloud platforms such as Amazon Web Services and Microsoft Azure, and edge platforms.

The company explained how these components will slot together. Vast DataSpace provides direct access to natural world data, eliminating the need for slow and inaccurate analog translators, while Vast DataStore makes it possible to store large amounts of unstructured data in an accessible way. Vast DataEngine makes it possible to transform unstructured data and understand it, while Vast DataBase serves as the main tool for querying this information.

The true promise of AI?

Vast Data says this new architecture is necessary because, although generative AI and large language models have gotten lots of people excited, these technologies remain limited to performing routine tasks. For example, most generative AI models are only capable of business reporting and reciting information that’s already known. However, the true promise of AI is that it will be able to “recreate the process of discovery by capturing, synthesizing and learning from data as humanity does.”

The company said DataStore, DataBase and DataSpace are all generally available from today, with DataEngine set to arrive next year.

Market Strategy analyst Merv Adrian was impressed, saying the Vast Data Platform is radically different from previous platforms. “By bringing together structured and unstructured data in a high-performance, globally distributed namespace with real-time analysis, Vast is not only tacking fundamental DBMS challenges of data access and latency, but also offering genuinely disruptive data infrastructure that provides the foundation organizations need to solve the problems they haven’t yet attempted,” he said.

Image: VecStock/Freepik