Imagine an intruder broke into your house and hid there, quietly observing your most intimate personal actions without you knowing. It’s the stuff of horror movies, but this is essentially happening across the globe as sophisticated cyberthieves infiltrate companies’ systems and stay undetected and able to access data resources at will for months.

Despite the millions being thrown at cybersecurity measures, the time taken to identify and contain a data breach is rising. According to IBM’s “Cost of a Data Breach” Report, the average time to detect and fix a breach was 279 days in 2019, 280 in 2020 and 287 in 2021. The average cost of a data breach is rising in parallel, from $3.86 million in 2020 to $4.24 million in 2021.

But what keeps security executives up at night is the fact that they’re working blind, unable to predict where an attack could come from or how to prevent it. It turns out, they need to be looking in the same place they’re finding other answers: their data observability tools.

“Most people have assumptions around observability as only an operational or an application support process. It’s also a security process,” said Ed Bailey (pictured), senior developer evangelist at Cribl Inc.

Bailey spoke with theCUBE industry analyst John Furrier during the AWS Startup Showcase: “Data as Code — The Future of Enterprise Data and Analytics” event, an exclusive broadcast on theCUBE, SiliconANGLE Media’s livestreaming studio. They discussed the importance of observability to not only provide visibility into a company’s data resources, but also enable breach investigators to track problems and recover faster. (* Disclosure below.)

Observability lake enables fast incident response and recovery

Taking advantage of observability solutions that are already in place to answer questions on application performance and user experience is one way companies can gain an advantage over cybercriminals and reduce their personal breach detection and recovery timeline.

Data observability solutions provider Cribl has been working on creating an observability architecture that combines an observability pipeline to route, reduce, collect and transform data with a queryable observability lake where the data is explored and refined.

“Technology professionals, especially security professionals, need data to make decisions. They need data to drive better decisions. They need data to achieve understanding. And that means they need everything,” Bailey said.

But data creation and replication are outpacing storage capacity, growing at a 23% compound annual growth rate, according to research by International Data Corp. This onslaught of data (not to mention that primary storage is expensive) means system administrators can’t keep all of the data accessible. Many have to make the tough choice of which data they keep accessible in their security information and event management store and which they don’t. Choose wrong, and the dollar clock is ticking on recovery from a breach.

“When you do detect a breach, you’re bringing in your instant response team. And typically without an observability lake, without Cribl solutions around observability pipeline, you’re going to have an incomplete picture,” Bailey said.

Establishing the scope of the breach, its impact, and what has been compromised requires building a picture from data, and “the more time it takes to get that data, the more time it takes for them to finish their analysis and contain the breach,” Bailey said. With its observability lake, Cribl provides the capability to retain all data in an accessible format and gives administrators the ability to reconstruct a complete picture for a pain-free breach investigation, regardless of if it occurs at the time of the crime or six months later.

One key feature of Cribl’s observability pipeline is that it is vendor-neutral. Data is stored in open formats, making it simple to interface with any vendor. It is also schema-agnostic, meaning data can be stored without a predefined schema, and format-agnostic so that it accepts any data, including configuration files and packet captures. And being able to store data in the cheapest store available saves costs.

“You have the choices to put your own solutions together and put your data where you need it to be. … We’re letting you pick and choose what suits your business,” Bailey stated.

Bailey’s first encounter with Cribl was in 2018, when as an enterprise architect for credit reporting agency TransUnion LLC., he demoed the company’s solution. Being used to using locked-in solutions that required him to spend hours creating custom code to configure them to TransUnion’s specification, he was dubious about Cribl’s claims to be completely vendor-neutral and agnostic. Then, he tried it.

“I cut about half a million dollars out of our license in the first 30 minutes in our first demo, and I was stunned,” he said. “It was just too easy and … it just struck me that my engineers can now spend their time on delivering value instead of integrations and moving data around.”

Digital reliance requires new ways of thinking about data

Finding new ways to handle data is essential for the future, according to Bailey, who describes operational security data as “the most volatile data in the enterprise,” as it is constantly being altered by developers and vendors who omit to notify anyone of the changes. Data needs to be treated in the same way that code is, with strict methodology around how it is handled.

“The same types of standards of disciplines that database administrators have done for years has to filter down into the operational areas, and you need tooling that’s going to give you the ability to manage that data, manage it in flight in real time in order to drive detections, in order to drive response,” he said.

Eliminating the complexity and confusion around data is the goal of observability, with the market gathering security under its wing as it expands to make sense of and protect the data that drives digital business. With its observability architecture, Cribl is part of this drive, and data as code is an important component.

“It’s so easy to be confused. It’s so easy to have complexity get in the way of progress,” Bailey said. “Being able to represent your data as code is a step forward, because the amount of data and the complexity of data is not getting simpler … so we need to come up with better ways to handle it.”

Watch the complete video interview below, part of SiliconANGLE’s and theCUBE’s coverage of the AWS Startup Showcase: “Data as Code — The Future of Enterprise Data and Analytics” event:

(* Disclosure: TheCUBE is a paid media partner for the AWS Startup Showcase: “Data as Code — The Future of Enterprise Data and Analytics” event. Neither Cribl Inc., a sponsor for theCUBE’s event coverage, nor other sponsors have editorial control over content on theCUBE or SiliconANGLE.)

Photo: SiliconANGLE