The “unbearable” levels of latency when accessing analytics-related storage was among the drivers that forced one company to radically rethink how log analytics retrievals are performed,.

Log analytics platform company Coralogix Ltd. reckons that the ever-escalating quantities of log data are now becoming so unwieldly that companies are having difficulty keeping up with it. And cherry-picking the data that an organization thinks it might need, on the off-chance that they get the selection right, isn’t a viable alternative because one can’t get the full insight across the entire data set anyway.

“Instead of ingesting the data, then indexing and storing it, and then analyzing the stored data, we analyze everything. And then we only store what matters,” said Ariel Assaraf (pictured), chief executive officer of Coralogix. “We go from the insights backwards.”

Assaraf spoke with, John Furrier, host of theCUBE, SiliconANGLE Media’s livestreaming studio, during the AWS Startup Showcase: The Next Big Things in AI, Security & Life Sciences. They discussed how and why Coralogix is attempting to revolutionize: what it believes is an archaic reliance on traditional storage for log analysis. (* Disclosure below.)

Reducing digital exhaust

Lowering the amount of data and the “digital exhaust” that logs create not only improves time-to-analysis, in part because of the aforementioned latency improvements, but also optimizes costs, according to Assaraf.

“If you don’t have a lot of data, those legacy systems do the trick,” he said. The executive is referring to $1,000 a month-type solutions. But he pointed out that costs don’t stop there. That’s because log data is constantly growing — often faster than revenue. That kiboshes unit economics for a company.

Suddenly, as the business builds, revenue growth isn’t as fast-moving as storage-requirement growth. In other words, a company might quickly jump from the thousand-dollar-a-month traditional, logs storage requirement to perhaps $70,000 a month — all without a natural increase in revenue. That messes with the valuation because storage costs end up detracting from the possibly healthy revenue. The balance sheet becomes skewed.

The key to Assaraf’s system is implementation of a state-store rather than a log-store. The Coralogix product, called Streama, “analyzes everything in real time, also the stateful thing,” a kind of capturing of insights, according to Assaraf.

Instead of going to storage and looking up what happened last week, say, one compares states. A state can be what’s normal or abnormal, for example. It’s “a very small portion of data that we actually need to store and frequently search,” Assaraf added.

Real-world example

One of the largest telcos in India is using the system with successful results, as an example, according to Assaraf.

“They were cherry picking about 100 gigs of data per day and sending it to one of the legacy providers, which has a great solution that does give value,” he stated.

However, the entire data set wasn’t getting sent. That was, in part, because of cost and scale. Equally important, it was also too cumbersome due to “clutter.”

“Whenever you search, you have to sift through millions of records. Many of them are not that important. We helped them analyze their data and worked with them to understand [it]. These guys had over a terabyte of data that had incredible insights. It was like a goldmine of insights,” Assaraf said.

Coralogix was able to prioritize it all by use case, resulting in more efficient data storage and retrieval of the intelligence.

“When you go from the insights backwards, you understand your data before you have to store it, you understand the data before you have to analyze it or before you have to manually sift through it,” he added.

A ‘road of pain’

There wasn’t a lightbulb moment that got Corlalogix to where it is now: providing products for five of the globe’s 50 largest companies now, according to Assaraf.

“It was a road of pain for years and years,” he recalled. “We had a rough first three years, and the company was on the verge of closing at the end of 2017.”

Indeed, it was the laborious dependency on schema, pricing and other issues, like latency, that was part of what got the company shifting into this radical decoupling of data from storage, cloud-based direction Today, Coralogix is an Advanced Technology Partner within the AWS Partner Network.

“If I see a company that grew from 50 people to 500 in three years, specifically if it’s cloud native or internet company, I know that their data grew not 10 times, but 100 times. It’s all about how you consume the data, because it doesn’t make sense for me to pay the same amount or give the same amount of attention to a record that is completely useless,” Assaraf concluded.

Watch the complete video interview below, and be sure to check out more of SiliconANGLE’s and theCUBE’s coverage of the AWS Startup Showcase: The Next Big Things in AI, Security & Life Sciences. (* Disclosure: Coralogix Ltd., sponsored this segment of theCUBE. Neither Coralogix nor other sponsors have editorial control over content on theCUBE or SiliconANGLE.)

Photo: SiliconANGLE