Big data management startup Komprise Inc. said today it’s introducing a major update to its platform with the launch of Komprise Transparent File Tables, an entirely new feature that enables a “structured view” of messy unstructured data, making it more accessible to artificial intelligence models and applications.

In an interview with SiliconANGLE, co-founder and Chief Technology Officer Krishna Subramanian said the new capability is available in early access now. It makes it possible for organizations to use unstructured data as an Apache Iceberg table within their cloud data lakehouse of choice, without paying the steep costs associated with large-scale data movement.

The update is all about enhancing AI operations and agents, she said. There are more ways than one to improve the capabilities of AI. While frontier labs are focused on building newer, more powerful AI models, Subramanian believes it’s possible to significantly boost the performance of existing models simply by feeding them more data.

The messy data problem

Unstructured data is the bread and butter of most large language models, but the vast majority of it is currently not being used at all. “We all know that AI is nothing without high-quality data, and enterprise data is the differentiator when all enterprises have access to the same models,” Subramanian said. “However, over 80% of enterprise data is unstructured, and 99% of this data is dark to AI because there is no easy way to query unstructured data.”

There are multiple reasons why the bulk of unstructured data remains dark, Subramanian said. Most of it lacks a consistent schema, much of it is of extremely poor quality, and it also tends to be voluminous and difficult to move from one place to another. Moreover, existing data ingestion mechanisms are not well-suited to unstructured information, because they usually work by copying all of the raw data. But this means that it lacks the required structure to be useful for AIU models.

“Current techniques to ingest data like ETL are built for structured and semi-structured data, because they focus on copying all the raw data without any schema extraction,” Subramanian said. “For instance, with file-based batch ingestion from cloud storage, files are staged in cloud object storage and loaded into the platform either on a scheduled or incremental basis. Data lands as raw strings, binary blobs or in semi-structured formats like JSON and XML, and is then refined through processing layers.”

Another method is to query unstructured data in place using external table references provided by cloud data lakehouse platforms such as Databricks and Snowflake. But this is extremely cumbersome, Subramanian said. It takes many hours for users to preprocess the data manually to create a structured description of it that’s compatible with the data lakehouse.

Bringing order to the chaos

Komprise says Transparent File Tables provides enterprises with a much more convenient alternative. Subramanian explained that it leverages Komprise’s distributed scale-out architecture, which helps classify the nature of unstructured information so it can be formatted as a tabular schema. It begins by automatically indexing an enterprise’s unstructured data from across their information technology environments, including their cloud and on-premises servers. During this process, companies can add rich content to the files with content, header, sensitive data scanning and metadata tagging to make it more organized and searchable, he said.

Once done, the table displays the enriched metadata in an Iceberg table and uses the company’s Transparent Move Technology to point to the location of the data it refers to. That means AI models can quickly find the data they’re looking for by searching through the classified and enriched metadata, then go to the place where it’s stored. The data itself never has to be moved. Instead, it’s dynamically loaded whenever it’s needed.

“Komprise TFT provides a structured, query-ready view of global enterprise unstructured data,” Subramanian said. “Komprise maintains a Global Metadatabase containing system, content and custom metadata which provides classification and structure for all the data in the enterprise. It solves two major issues for data engineers and analysts by giving consistent schema to unstructured data and enabling its use in analytics and AI tools like Databricks and Snowflake, without moving petabytes of data.”

Komprise said TFT makes it possible to search for unstructured data using existing business intelligence and data analytics tools using Iceberg queries, with data governance based on each user’s access permissions.

Easier access means more value

Subramanian believes Komprise TFT will dramatically improve access to unstructured data for many new kinds of applications. For instance, a machine learning engineer at a healthcare organization can use it to curate the high-quality datasets needed to fine-tune a radiology LLM without exposing sensitive medical records.

“They can do this with AI-generated tags for the modality, body part, study type, findings and so on extracted from DICOM files and their associated reports,” she said. “The engineer can then join this with structured patient cohort data from the EHR to scope the right subset and export it as Parquet for ingestion into the RAG pipeline or fine-tuning workflow.”

Other use cases include building a unified data estate map to help financial organizations understand where their sensitive data lives, so they can enhance their compliance operations. Alternatively, an AI agent in media and entertainment can use the Iceberg table format to identify media archives and narrow down which scripts it needs to ingest when it’s given a specific task, Subramanian explained.

The main benefit for organizations is that they can leave their vast unstructured data estates intact, exactly where they live, while making it much more accessible to AI and analytics teams. It means they can avoid the expensive and complex process of moving large petabytes of data, lowering their costs and ensuring any sensitive information remains secure, while also getting much more value out of it.

“Data and AI teams get access to high-quality unstructured data through their familiar interface, without the cost and complexity of ingesting raw data and figuring out how to extract the schema,” Subramanian said.

Image: SiliconANGLE/Gemini AI