As organizations continue to collect more and more data, the cost of analyzing that data can cripple margins. Companies, therefore, are cherry-picking data to analyze. That’s a mistake, according to a startup that specializes in real-time streaming analytics. Why?

Because you’re not getting all the intelligence, according to Ariel Assaraf (pictured), chief executive officer of log analytics startup Coralogix Ltd. More than 99% of data is untapped, or unanalyzed, Assaraf added, quoting MIT research. The main reason is that the sheer quantity of incoming information is overwhelming storage, and consequently storage costs are escalating. It’s “data growing exponentially, but it’s not just growing exponentially, it’s growing faster than revenue,” he said. To compensate, companies are choosing not to analyze some of the data.

In anticipation of the AWS Startup Showcase: The Next Big Thing in AI, Security, & Life Sciences event — set to kick off on June 16 – John Walls, host of theCUBE, SiliconANGLE Media’s livestreaming studio spoke with Assaraf for a special CUBE Conversation on how businesses can cope with escalating data costs yet still perform required analysis. (* Disclosure below.)

Too expensive to store

Organizations aren’t tapping their data resources fully because it’s simply too expensive to analyze everything. That’s despite the advantages of doing so, like monitoring observability, which is “probably the largest data producer inside any organization,” Assaraf said.

Observability is where companies check out how every user experiences an app, among other things. It helps a lot with code upgrades. However, real-time streaming analytics solves the problem to a certain extent, but only if you can create snapshots over time. Long-term analytics matters — you need to be able to look backwards too, according to Assaraf.

Simple real-time alerts aren’t enough in microservices, though. “You see an event, send me an alert. You see a metric, send me an alert,” but really, what one needs to know is: Did something happen more than it did last week? or “Let me know if something happened for the first time this month.” Assaraf says Coralogix’s product can achieve that.

The company’s technology reduces the storage requirements — the expensive part of long-term analytics — and uses real-time streaming to do it.

Storage is a can of worms, according to Assaraf, particularly if it’s high-performance storage, which is typically needed for speedy analytical access. Cheaper, lower storage tiers are too much of a compromise to really get into the data quickly, he added. Additionally, exponentially growing storage, kept over time, not only costs money, but loses performance.

“It’s actually slower, it’s more cluttered,” he stated.

The problem is exacerbated by the fact that storage is such an aged technology that’s there’s no longer any wiggle room to optimize compression more — it’s all been done, Assaraf explained. “Storage is one of the areas where you can least optimize because it is what it is,” he said. “There’s not a lot of ways to really save on storage.”

Assaraf also speculates that data growth is going to make it ultimately “impossible to write all the data to the disk” — there will just be too much data. Traditional scaling will become unattainable.

Real-time streaming

“We decided to go with the real-time streaming analytics approach,” said Assaraf, explaining that the machine learning engine “learns the data continuously throughout the entire history of time.” That allows improvements to code and microservices because all of the data gets looked at, not just specific servers and environments — cherry-picking is eliminated.

Indexing and then analyzing isn’t as economic as ingesting and analyzing in real time and then only storing what matters, according to Assaraf: “Streama is the technology that we created that does the real-time streaming but also involves components that store the state of the system at any given point in time.”

Coralogix customers can go back in time and compare versions or compare matrix or graphs and see how specific versions affected specific microservices and how specific microservices affected their entire production systems. So, the question “what is the version that broke something?” is answerable by performing analytics in the system. That’s a particularly difficult question as more versions get uploaded to services all the time — common in microservices and CI/CD, the software engineering practice where integration and deployment is continuous. But because automation builds applications continuously, organizations can use continuous software monitoring as well, according to Assaraf.

CI/CD along with microservices “causes a lot of uncertainty.” It gets much harder to triage and figure out which particular microservice is causing the problem. “On one hand that’s great because you have less decoupling. You can go faster, you can be faster to market, respond to the market faster,” he added. But a lot of software problems develop after upgrades.

Kubernetes is the infrastructure foundation for Coralogix’s engine, which is built entirely on AWS. Interestingly, Coralogix has also built an open-source CI/CD tool too.

“So now [organizations] can connect their entire environment and get full cloud observability and security within minutes and do it in an economic way,” Assaraf concluded.

Watch the complete video interview below, be sure to check out more of SiliconANGLE’s and theCUBE’s CUBE Conversations, and tune in to theCUBE’s live coverage of the AWS Startup Showcase: The Next Big Thing in Security, AI and Life Sciences event on June 16. (*Disclosure: Coralogix Ltd. sponsored this CUBE Conversation. Neither Coralogix nor other sponsors have editorial control over the content on theCUBE or SiliconANGLE.)

Photo: SiliconANGLE