Observability tools allow enterprises to monitor the operating state of their systems in real time.

However, value stands to be lost when operations teams rush through the initial telemetry pipelines and focus more on storing and analyzing the expanding data.

“Right now, there’s a huge problem where people are generating more data into their system, but the value doesn’t correlate and can even go down,” said Eduardo Silva (pictured, left), chief executive officer of Calyptia Inc. “We found that in the observability space, everybody was focusing on the storage of the data or the running analysis but nobody focused on the previous part, which we call the first mile.”

Silva and Mike Kelly (right), chief executive officer of observIQ Inc., spoke with theCUBE industry analyst John Furrier and guest analyst Joep Piscaer at the KubeCon + CloudNativeCon Europe event, during an exclusive broadcast on theCUBE, SiliconANGLE Media’s livestreaming studio. They discussed new open-source innovations in enterprise observability. (* Disclosure below.)

Evolving observability to suit expanding user demands

ObservIQ focuses on open-source observability solutions, with a strong connection with the larger user community. It’s catering to that community by contributing to several projects such as OpenTelemetry, which was recently merged with the Elastic Common Schema to facilitate ease and consistency, according to Kelly.

“Anything that can combine these standards makes it easier for everyone, and we’ve been seeing that across the board,” he said. “I’d say also that there’s been a shift over the last year that we’ve noticed with OpenTelemetry. The expanding vendor support is great, but now we’re seeing all the customers are also embracing it, and that’s been a huge benefit.”

The main areas driving the changing customer use cases are increased data sources, a proliferation of data formats and much bigger scale requirements, according to Silva. These have resulted in the need for a unified telemetry standard spread out with a multi-vendor approach.

“The other problem that exists right now is that with telemetry, you need to integrate and connect different systems,” Silva said. “Given that reality, most companies are on the journey to have a vendor-neutral approach to observability and avoid vendor lock-in.”

As the data being ingested by enterprises continues to soar, managing that data has become a critical point of failure. Standardizing the telemetry layer, as projects such as Fluentd, Fluent Bit and OpenTelemetry believe, will go a long way to remedy this pain point, Kelly added.

In the area of implementing artificial intelligence in the observability mix, Calyptia is already testing out scenarios where it’s charged with generating processing rules for data handling. However, it’ll take time for AI to perform those tasks in a consistent, trustworthy manner, according to Silva.

“What we did was an AI implementation where you write in human text, ‘Hey, take my data that looks like this and do x, y, z,’ and the AI will generate the processing rule for you,” he said. “I think that we’re in the phase of simplifying the control of the data, but it will take maybe a few years to get to the AI discovering everything.”

Here’s the complete video interview, part of SiliconANGLE’s and theCUBE’s coverage of the KubeCon + CloudNativeCon Europe event:

(* Disclosure: This is an unsponsored editorial segment. However, theCUBE is a paid media partner for KubeCon + CloudNativeCon. Neither Red Hat Inc. nor other sponsors of theCUBE’s event coverage have editorial control over content on theCUBE or SiliconANGLE.)

Photo: SiliconANGLE