The volume and velocity of data necessary for cloud monitoring can make the task a major challenge for any organization. Maintainers of tools such as Prometheus and OpenMetrics are working to make real-time monitoring as practical as possible, even as cloud operations scale.

Real-time analytics is also becoming increasingly important, as analysis after the fact may yield insights that are no longer useful or relevant.

“’If you just toss it into a data lake and do batch analysis like half a day later, no one cares about it anymore,” said Richard Hartmann (pictured), community director at Grafana Labs Inc. “It needs to be live … or at least the largest part of it needs to be live. You need to be able to alert right now if something’s imminently customer-facing.”

Hartmann spoke with John Furrier, host of theCUBE, SiliconANGLE Media’s livestreaming studio, during KubeCon + CloudNativeCon. They discussed Prometheus, OpenMetrics and the challenges that have come with the growth of cloud computing. (* Disclosure below.)

New standards for cloud monitoring

As more businesses move to the cloud native space, they are able to scale different and new axes, which enables growth, different operating models, and the ability to choose different or more modern engineering trade-offs, according to Hartmann.

“The underlying problems are still the same, but you just slice and dice your problems and compartmentalize your services differently,” he said. “But the problem is, it becomes more spread out and the more classic tooling tends to be built for those small classic setups and architectures. As your architecture becomes more malleable and as you can choose and pick how to grow it along which axis … that limits the ability of the humans actually operating that system to understand what is truly going on.“

Artificial intelligence, machine learning and other automation tools are helping to change the data equation by streamlining this new level of observability and a company’s ability to scale.

“A lot of this is about enabling this higher volume of data, this higher scale of data, this higher cardinality of what you actually attach as metadata on your data and then still be able to query all this and make sense of it at scale and at speed,” Hartmann said.

As cloud infrastructure scales up, it’s also essential that organizations develop a common language of open standards that will ensure communication across different systems and between various end users.

“Because you have this ‘lingua franca,’ you have these widely adopted open standards,” Hartmann said.

Here’s the complete video interview, part of SiliconANGLE’s and theCUBE’s coverage of KubeCon + CloudNativeCon. (* Disclosure: the Cloud Native Computing Foundation sponsored this segment of theCUBE. Neither the Cloud Native Computing Foundation nor other sponsors have editorial control over content on theCUBE or SiliconANGLE.)

Photo: SiliconANGLE