UPDATED 11:48 EST / DECEMBER 23 2023


The rise of observability and why it matters to your business

The rapid awareness and adoption of artificial intelligence has increased concerns around security, governance, compliance and business risk.

The digital business imperative to improve customer experience combined with a vastly more complex technology environment, comprising multiple clouds, the internet of things, containers, app integration and the popularity of open source, are driving need for better observability across technology stacks. But fragmented information technology environments, lack of talent and a spate of existing installed monitoring tools are preventing organizations from achieving their full-stack observability aspirations.

In this Breaking Analysis, we dig deeper into the trends in observability based on a recent custom study from our partners at Enterprise Technology Research. As always, we’ll provide our perspective on the market and players with a view toward trends to watch in 2024.

What is full-stack observability & O11Y?

Full-stack observability is the capability to discover the real-time status of each technology stack component in a distributed IT environment. It involves holistically viewing cloud-hosted applications, services, infrastructure, on-premises IT, K8s (there’s your O11Y hint) infrastructure and more. O11Y uses telemetry data such as metrics, logs, traces and other information from the entire IT environment to provide complete end-to-end visibility to understand the connection and dependencies among IT components.

FSO is more than just monitoring and reporting. It uses reasoning and often machine intelligence to draw conclusions about an environment to recommend change or even take action on behalf of humans.

New survey captures the customer mindset of FSO

FSO or O11Y has been a hot topic in the press and analyst community. That’s why we’re so excited to share the results of a recent study from ETR.

ETR has an amazing custom research capability that we’re privileged to share with you. This particular study was done in partnership with New Relic Inc., which we promise has zero influence on the commentary and conclusions of this Breaking Analysis or the ETR analysis.

This particular study had 1,700 respondents and, unlike most of the work we share, had a global respondent base, with U.S. customers only accounting for about a quarter of the survey. The companies spanned multiple industries, all organization sizes (80% more than 5,000 employees) and mostly technology practitioners.

Like security, observability is a crowded market

It has grown out of the world of application performance monitoring or APM, network monitoring, log analytics, AI operations or AIOps, database performance management, dashboarding, error tracking and literally dozens of legacy and fragmented market segments. There are literally dozens and dozens of vendors in this space.

Here’s a sample using network monitoring as the sector of choice. In this space “other” is the leading vendor, which underscores how crowded the market is. But the likes of Splunk, Grafana, Dynatrace, Elastic, AppDynamics, open-source tools such as OpenTelemetry, Prometheus and the ELK stack, New Relic, Sumo Logic and more play in this specific market.

On the right-hand side of this chart you can see the sectors ETR tracked in this study – about 18, including distributed tracing, serverless monitoring, mobile monitoring, MLOps and so forth. Selecting each one in the pulldown will shuffle the deck of players.

Splunk leads in most, but as the leader everyone wants to attack it and get a piece of its market. It’s why Cisco Systems Inc. paid $28 billion for Splunk, because it has the largest footprint. But there are many others. For example, in application performance monitoring, Datadog, Dynatrace, AppD (also owned by Cisco) and New Relic join Splunk at the top of the list. In synthetic monitoring, Dynatrace and Datadog pop up as most popular along with Splunk. In distributed tracing, Grafana, Dynatrace, Prometheus and OpenTelemetry are cited more often than Splunk. And “other” is always prominent.

You get the point. FSO involves a lot of different capabilities and there are many vendors playing in the market. For example, here are some names in the other category you may or may not know: Lucidworks, Exabeam, Sentry, Moogsoft, Honeycomb, ChaosSearch, Graylog, ScienceLogic, Cribl, Monte Carlo, Observe, InfluxData and dozens more I could name. It’s crowded.

Consolidation continues to be a theme, but it’s not easy

The market is so fragmented and the budget pressures of 2022 and 2023 have caused a consolidation of tools.

This chart shows the number of observability tools installed in 2023 (top) as compared with 2022 (bottom). And you can see the reduction in the percentage of customers with five, six, seven and eight tools – highlighted in red. And the shift to one, two and three tools installed is shown by the red arrows.

But it’s clear that the movement in the past year is just scratching the surface as around 60% of customers cite that they have four or more tools. So more work must be done to accommodate greater consolidation. Having said that, around 30% of customers in the survey lean toward having multiple tools as a means of tapping best-of-breed capabilities.

Like in security, the quest for excellence often results in fragmentation.

Which vendors benefit most from consolidation?

When asked which vendor platforms customers prefer to consolidate on, about13% say they don’t want to consolidate. But Splunk, Dynatrace and Datadog are most commonly cited.

Once again, “other” is in the mix pretty prominently, with Elastic, AppD and New Relic also showing.

Now some interesting things change when you filter the data. For example, executive management prefers to consolidate on Dynatrace and surprisingly a somewhat higher percentage, 16%, say they don’t want to consolidate.

For what it’s worth, the small percentage of females in the survey want to consolidate on AppDynamics, a company Cisco acquired in 2017. Services and consultancies prefer to consolidate on Datadog.

So again this just underscores the complexity and varied preferences of the market.

Humans are still in the loop

Another notable finding in the survey is the degree to which manual labor is still a dominant force.

Despite the AI hype, fewer than 10% of customers say their incident detection and remediation workflows in the future will be mostly AI-led. But a huge percentage – around 75% as shown on this chart – expect to have some type of AI-led automation in their identification and remediation workflows.

Automation is a two-edged sword. It’s one of those alluring concepts, but there’s real caution around trusting machines to judge what actions should and shouldn’t be taken and when. So given the sensitive nature of change management, we would expect this trend to continue to lean toward AI-led automation, but it will take some time before humans are mostly out of the loop. Moreover, while many vendors claim to have AI, there’s a wide spectrum of capabilities, and customers should be very cautious about vendor claims in this regard.

Now, not surprisingly, the regulated industries of financial services, healthcare and government see a much lower tendency to be mostly-AI led in this context over the next year (well under 5% say mostly AI-led in this chart), whereas industries such as energy and high tech are much more likely to adopt AI aggressively in this space.

Interestingly, the data show that senior managements are more likely to push for AI adoption whereas the practitioners, who literally have their jobs on the line – that is, machines replacing humans or getting fired for implementing rogue automation – are much less optimistic.

What does O11Y do for your business?

According to more than 1,500 practitioners in this survey, there are many benefits.

Better uptime, improved efficiency, more secure environment and a better customer experience help developers be more productive because they’re doing less troubleshooting and the like. And this ostensibly leads to greater business agility and more revenue. Only one-half of 1% cited no benefit. This is across all roles, all geographies and all industries, by the way.

But there are challenges to adopting FSO, including the siloed nature of data and IT, tools creep, lack of talent… but the No. 1 pressure is lack of budget – that is, it’s perceived as too expensive. So the industry has an opportunity to better sell the benefits.

Perhaps retiring goofy terms like O11Y would help!

FSO trends to watch

Security remains the No. 1 priority in the ETR survey data and that requires better visibility. So we don’t expect FSO to be a fad that fades in 2024. Just the opposite, actually. As digital businesses fight for customers and access to real time data becomes more critical, FSO will only grow in prominence in our opinion.

The latest ETR survey data on the spending macro is showing some positive momentum for 2024 – but really toward the second half. Amazon Web Services Inc. Chief Executive Adam Selipsky  said it best when he shared at the re:Invent in late November that we’ve seen better times, we’ve seen worse times, but we’ve never seen such uncertain times. True words of wisdom from the CEO of AWS.

Everyone is AI-crazed – including us – but your practitioners are cautious and probably skeptical that they have job security. So part of your challenge is to convince the doers that AI will bring new opportunities and relieve them of mundane and tedious tasks. And you’ll have to also put in processes to make sure that automation doesn’t create boo-boos because with the cloud and distributed computing, mistakes can travel fast.

The last point is that clearly the market generally and full-stack observability specifically are moving toward platforms. But the siloed nature of tech stacks, technical debt and sunk costs will mean a continued fragmented market. That said, those leaders with full-stack capabilities, cloud-native tooling and real AI will be in a good position to ride the consolidation wave in 2024 and beyond.

Keep in touch

Thanks to Alex Myerson and Ken Shifman on production, podcasts and media workflows for Breaking Analysis. Special thanks to Kristen Martin and Cheryl Knight, who help us keep our community informed and get the word out, and to Rob Hof, our EiC at SiliconANGLE.

Remember we publish each week on Wikibon and SiliconANGLE. These episodes are all available as podcasts wherever you listen.

Email david.vellante@siliconangle.com | DM @dvellante on Twitter | Comment on our LinkedIn posts.

Also, check out this ETR Tutorial we created, which explains the spending methodology in more detail. Note: ETR is a separate company from Wikibon and SiliconANGLE. If you would like to cite or republish any of the company’s data, or inquire about its services, please contact ETR at legal@etr.ai.

Here’s the full video analysis:

All statements made regarding companies or securities are strictly beliefs, points of view and opinions held by SiliconANGLE Media, Enterprise Technology Research, other guests on theCUBE and guest writers. Such statements are not recommendations by these individuals to buy, sell or hold any security. The content presented does not constitute investment advice and should not be used as the basis for any investment decision. You and only you are responsible for your investment decisions.

Disclosure: Many of the companies cited in Breaking Analysis are sponsors of theCUBE and/or clients of Wikibon. None of these firms or other companies have any editorial control over or advanced viewing of what’s published in Breaking Analysis.

Image:  Designpics/Adobe Stock

A message from John Furrier, co-founder of SiliconANGLE:

Your vote of support is important to us and it helps us keep the content FREE.

One click below supports our mission to provide free, deep, and relevant content.  

Join our community on YouTube

Join the community that includes more than 15,000 #CubeAlumni experts, including Amazon.com CEO Andy Jassy, Dell Technologies founder and CEO Michael Dell, Intel CEO Pat Gelsinger, and many more luminaries and experts.

“TheCUBE is an important partner to the industry. You guys really are a part of our events and we really appreciate you coming and I know people appreciate the content you create as well” – Andy Jassy