These days, businesses are awash with data ⁠— and the demand for that data is extremely high, with smart devices, domain experts, ecosystem partners and many others requiring access simultaneously. That creates some significant challenges for organizations.

It’s also creating significant demand in the market. The data fabric sector is expected to grow from $1.6 billion in 2022 to $6.2 billion by 2027, according to a recent forecast. How might organizations adapt to these modern data challenges, and how can smart data fabrics play a role in solving them?

“There’s tremendous pressure on digital transformation and modernization of our businesses and certainly in the face of volatility, whether it be a global pandemic, whether it be geopolitical volatility,” said Scott Gnau, global head of data platforms at InterSystems Corp., during a keynote presentation at Global Summit 2022. “The best way we can all be successful is to have all of the information that we need available to us at the right time to make those decisions, especially in the transformation zone.”

During the recent “Unlocking the True Potential of Data With Smart Data Fabrics” event, theCUBE industry analyst Dave Vellante spoke with InterSystems experts about the company’s plans to solve modern data challenges, including its plans to utilize smart data fabrics to help solve those challenges. TheCUBE is SiliconANGLE Media’s livestreaming studio. (* Disclosure below.)

Reducing, integrating and consolidating: A changing paradigm

Given the demand for data across industries — and the ongoing efforts to monetize that data — today’s landscape has grown increasingly complicated. Adding to the challenge is that data is often siloed and, therefore, is difficult to consume, share and make valuable.

For years, there has been a conversation around reducing silos while integrating and consolidating data, according to Gnau.

“But the sources are just too varied,” he said during today’s event. “The required agility for a business unit to operate and manage their customers is creating an enormous pressure.”

But, ultimately, silos aren’t going away, and there’s a realization that organizations will need to manage silos while taking advantage of data that may live across different parts of the business — while managing the all-time-high expectations of the consumer, according to Gnau.

“Pulling all of this together really means that our customers and businesses around the world are struggling to keep up, and it’s forcing a new paradigm shift in underlying data management,” he said, adding that what started many years ago with data marts shifted to data warehouses and then graduated to data lakes.

“I think those technologies are now just a piece of the puzzle that is really required for success — and this is really what’s leading to data fabrics and data meshes in the industry.”

So, what exactly are data fabrics — what do they solve, and how do they work? Keeping in mind that the technology does not exclude the other technologies mentioned above, data fabrics intend to take the best of those worlds while adding in a notion of being able to do data connectivity with provenance as a way to integrate data versus data consolidation.

“Data has gravity, right? It’s expensive to move data. It’s expensive in terms of human cost to do ETL processes where you don’t have known provenance of data,” Gnau said. “Being able to play data where it lies and connect the information from disparate systems to learn new things about your business is really the ultimate goal.”

Ongoing challenges around supply chain and logistics tie into these challenges, given that all companies in that field are data-driven, have lots of access to data, and have formalized and automated their processes.

“But being able to connect that information together without changing the underlying system is an important way to learn and optimize for supply and logistics, as an example,” Gnau said. “And that’s a key use case for data fabrics — being able to connect, have provenance, not interfere with the operational system but glean additional knowledge by combining multiple different operational systems data together.” 

Ultimately, the business benefit of the data fabric is to have a “single source of the truth” for a company, according to Gnau. Companies will want to be able to have any kind of algorithm, where appropriate, run against data without having to do a bunch of massive ETL processes or making another copy of the data and having to move it somewhere else.

“To that end, we have taken our award-winning engine — which provides traditional analytic capabilities and relational capabilities — we’ve now integrated machine learning,” Gnau said. “You basically can bring machine learning algorithms to the data without having to move data to the machine learning algorithm.”

Here’s theCUBE’s complete video interview with Scott Gnau:

Making a data fabric ‘smart’

How do smart data fabrics iactually work? By embedding a wide range of analytics capabilities — including data exploration, business intelligence, natural language processing and machine learning —  directly within the fabric, things become faster and easier for organizations to gain new insights and power intelligence, predictive and prescriptive services, and applications, according to InterSystems.

The company also says that smart data fabrics are accessible across many industries, including financial services, supply chain, healthcare and more.

As a part of a live demonstration during today’s event, Jess Jowdy (pictured), manager of healthcare sales engineering at InterSystems, used Postman to simulate a call from an external application, using the healthcare field as an example.

Jowdy showed how an organization might be pulling information from an electronic medical record, pulling clinical history or clinical notes from a medical transcription software, or pulling adverse reaction warnings from a clinical risk grouping application. Using the example of a patient logging on from their phone and retrieving the information through Postman, Jowdy hit “send” and showed how everything had already been preloaded.

“I’m going to be looking for information, where the last name of this patient is Simmons, and their medical record number, their patient identifier in the system is 32345,” Jowdy said. “So, as you can see, I have this single JSON payload that showed up here of just relevant clinical information for my patient … all within a single response.”

Typically, when organizations see responses such as this, there may be an assumption that the service is interacting with a single backend system and that single backend system is in charge of packing the information up and returning it back to the caller. That’s not the case here.

“In a smart data fabric architecture, we’re able to expand the scope to handle information across different, in this case, clinical applications,” Jowdy said. 

Here’s theCUBE’s complete video interview with Jess Jowdy:

How smart data fabrics could be applied to different industries

Given the increasing demands on businesses to provide value to customers while reducing risk and swiftly responding to data demands, organizations face some big challenges. InterSystems believes its next-gen approach to Customer 360 solution, along with how it leverages its smart data fabric across various industries, uniquely positions it to solve those challenges, according to Joe Lichtenberg, director of product and industry marketing at InterSystems.

“It’s a fundamentally different approach,” he said. “All of these prior approaches have added value, right? It’s not like they were bad. But there’s limitations, and the business still isn’t getting access to all the data that they need in the moment.”

Data warehouses are “terrific” if one knows the questions that need to be answered and they take the data and structure it in advance, Lichtenberg pointed out.

“Now, you’re serving the business with sort of pre-planned answers to pre-planned queries,” he said. “The data fabric, what we call a smart data fabric, is fundamentally different … in that rather than sort of in batch mode, taking the data and making it fit for purpose — with all the complexity and delays associated with it — with a data fabric we’re accessing the data on demand, as it’s needed, as it’s requested either by the business or by applications or by the data scientists directly from the source systems.”

Beyond healthcare, InterSystems sees applications for smart data fabrics in every industry. The company, which has been around now for 43 years, has tens of thousands of customers in every industry, each of which Lichtenberg has seen critical use cases. One example is issues being seen in the supply chain at present.

Organizations want a near real-time expansive view of what’s happening across the entire supply chain, from supply through distribution, posing a huge real-time data silos problem. If one considers an extended supply chain, it’s complicated enough with all the systems and silos inside a firewall, according to Lichtenberg.

“And then building on top of real-time visibility is what the industry calls a control tower, what we call the ultimate control tower,” he said. “It’s the built-in analytics to be able to sense disruptions and exceptions as they occur, and predict the likelihood of these disruptions occurring, and having data-driven and analytics-driven guidance in terms of the best way to deal with these disruptions.”

Here’s theCUBE’s complete video interview with Joe Lichtenberg:

Key takeways

1. Data silos aren’t going away — so organizations will need to juggle multiple needs at once.
2. Data fabrics aim to help combine multiple different operational systems data together.
3. Smart data fabrics are intended to function across multiple different industries.
4. Data fabrics are also intended to provide a fundamentally different approach to big challenges.
5. Supply chain challenges are a prime example of the solutions proposed.

You can watch the full event video here:

(* Disclosure: TheCUBE is a paid media partner for the “Unlocking the True Potential of Data With Smart Data Fabrics” event. Neither InterSystems Corp., the sponsor of theCUBE’s event coverage, nor other sponsors have editorial control over content on theCUBE or SiliconANGLE.)

Photo: SiliconANGLE