Big data analytics has emerged in recent years as the heart of the digital business, the basis for turning data into a business value that drives differentiating operations and customer experiences.

But even as big data analytics, or BDA, has become a boardroom conversation, too often the discussion comes down to one difficult question: “Why aren’t we generating the expected returns?”

The tech industry is finally making progress on providing an answer. User convergence on full-stack BDA architectures is a key that will unlock digital business success. However, technology alone is insufficient for driving analytic applications unless organizations accelerate digital transformation to structures and processes that leverage BDA’s disruptive potential.

Likewise, organizations will need to step up the empowerment of their digital workforces by investing in people, roles, skills, organizational centers of excellence and other pillars of data-driven culture. At the very least, more business analysts and other knowledge workers will need to shift toward more of a development role in the machine learning, deep learning and artificial intelligence initiatives that drive disruptive business applications.

The worldwide BDA market grew at 24.5 percent in 2017 from the year before, faster than Wikibon, the analyst group of SiliconANGLE Media, forecast in last year’s report thanks to better-than-expected public cloud deployment and utilization as well as progress in convergence of tools. Enterprises are moving more rapidly out of the experimentation and proof-of-concept phases to achieving higher levels of business value from their big-data deployments.

Looking forward, the overall the BDA market will grow at an 11 percent compound annual growth rate to reach $103 billion by 2027 (below). Edge computing – including streaming and ML app deployments on smart devices — will boost the market in the out years.

Trends overview

Going forward, the following digital business trends will accelerate solution convergence up and down the stack include:

Achieving strategic outcomes. Customers will assure strategic business outcomes by converging on strategic BDA solution providers who deliver prebuilt applications that incorporate best practices and are rapidly customizable to their unique requirements.

Reducing latencies. Customers will increasingly eliminate delays and bottlenecks throughout data, application and business infrastructures by converging on internal pipelines that incorporate fewer discrete BDA products.

Streamlining processes. Customers will reduce complexities by converging on fewer, simpler, more consistent, and more automated BDA development and operations processes that span disparate pipelines, platforms, tasks, and roles.

Tightening controls and safeguards. Customers will tighten oversight and compliance by converging on unified governance tools that enforce security, policy and other guardrails up and down the BDA stack and across disparate pipelines, platforms and tasks.

From Wikibon’s latest update to the annual BDA market study, what remains the same as in previous years are the following challenges:

Excessive complexity. BDA environments and applications are still too complex and need to simplified to put them within reach of mainstream user and developers, many of whom lack in-house IT staff with the requisite specialized skills.

Cumbersome overhead. BDA administration and governance processes are still too siloed, costly, and inefficient for many IT professionals.

Protracted pipelines. BDA application development, training, and deployment pipelines are still too time-consuming, manual, and inconsistent, and need to be automated, accelerated, and streamlined to a greater degree.

Bespoke applications. BDA professional services are still essential for developing, deploying, and managing the many bespoke applications that span across hybrid clouds, involve disparate platforms and tools, and incorporate unfathomably complex data processes.

Chief big data analytics market trends through 2030 (Source: Wikibon)

Wikibon’s latest BDA market study also found that several trends are driving the industry’s competitive landscape. Here are some of them:

Public cloud providers are expanding their sway over the BDA market. By 2020, the BDA industry will have converged around three principal public cloud providers — most likely, Amazon Web Services, Microsoft Azure and Google Cloud Platform — and most ISVs will build solutions that operate in all of them. These and other BDA public cloud providers — including such established BDA vendors as IBM Corp. and Oracle Corp. – offer managed IaaS and PaaS data lakes into which customers and partners are encouraged to develop new applications and into which they’re migrating legacy applications. As a consequence, the pure data platform/NoSQL vendors seem to be flatlining and becoming marginalized in a BDA space increasingly dominated by diversified public cloud providers vendors. Likewise, the old-line business analytics vendors are losing momentum and falling by the wayside as the market migrates to cloud-based solutions from diversified BDA for these capabilities.

Public cloud advantages over private clouds for BDA continue to widen. By 2021, public clouds will be the preferred BDA approach for every customer segment, including large enterprises. That’s because BDA public cloud solutions are maturing more rapidly than on-premises stacks, adding richer functionality, with increasingly competitive cost of ownership. Unlike on-premises alternatives, public cloud providers can offer BDA databases such as Google’s Spanner, Microsoft’s Cosmos DB and AWS’ DynamoDB that provide global consistency and availability, owing to the proprietary fiber networks linking their data centers.

Hybrid clouds are becoming an intermediate stop for enterprise BDA on the way to more complete deployment in public clouds. By 2022, hybrid clouds will figure into the plans of most large enterprise, but predominantly as a makeshift strategy. The balance is tipping toward enterprises putting more of their BDA assets in public clouds, thereby boosting the importance of hybrid public/private BDA clouds as transitional architectures. Traditional BDA vendors are adding developing their products for hybrid use cases, creating new options for traditional workloads. By the same token, more traditionally premises-based BDA platforms are being rearchitected to deploy primarily in public clouds.

Innovative BDA startups are bringing increasingly sophisticated cloud-facing solutions to market. By 2024, the dominant BDA application providers will be those that have disrupted the competitive landscape with AI-based solutions of astonishing sophistication. The threat from new market entrants is accelerating in every BDA niche, with most of the innovations being designed for public or hybrid cloud deployments. Many new analytic and application databases, stream processing, and data science startups have entered the BDA arena in the past several years. Many of these new vendors provide solutions that incorporate sophisticated ML/AI and are packaged to address disruptive business applications.

Disruptive BDA approaches are becoming viable alternatives to established platforms. By 2025, a new generation of “unicorn” BDA powerhouse solution providers will have emerged from startup status on the strength of a new technical approach of unprecedented flexibility and sophistication — probably based on some blend of IoT, blockchain and stream computing. The threat of substitute products and services in the BDA market is coming from startups contending with incumbents, but also from incumbents who are themselves shaking up the old order by introducing a steady stream of innovative solutions that address established and emerging business requirements in new ways using new technological approaches.

Databases are being deconstructed and reassembled into new approaches to address emerging requirements. By 2028, the database as we used to know it will be ancient history, from an architectural standpoint, in a world where streaming, in-memory, serverless infrastructures reign supreme. The BDA market continues to evolve through the rethinking of traditional database architectures to address new analytic use cases.

BDA packaged applications are becoming more widely available. By 2030, every customer — from large enterprises down to mom-and-pop shops — will acquire BDA solutions as prebuilt, pretrained templatized cloud offerings that continuously and automatically adapt and tune themselves to deliver desired business outcomes. Packaged business applications are driving mainstream adoption of BDA solutions, just as they did for transactional business applications. In that regard, a growing range of packaged BDA applications are coming to market, leveraging AI, DL, and ML as their core value added.

Action Items

All those trends suggest a number of actions companies should take:

• Over the coming 3-6 months, BDA professionals should begin to evaluate the growing range of commercial packaged solutions that accelerate time-to-value for specific industry or line-of-business analytic and transactional applications.
• Over the coming 3-12 months, you should explore moving more of your BDA applications to public cloud environments to take advantage of the rapidly maturing, low-cost options in this arena.
• Over the coming 3-24 months, you should consider building out your hybrid public/private multi-clouds to ensure a graceful long-term transition to the more completely public-cloud BDA computing environments of the future.
• As you work out your long-range public-cloud BDA architecture—including the underlying hardware platforms needed to process all these workloads at scale–you should evaluate cloud providers on the richness of their own solution portfolio, on the range of partner/ISV solutions built for those environments, and on APIs for mixing and matching open-source and proprietary solutions for diverse analytics, transactional, and other data-driven workloads.
• Starting immediately, you should engage professional services partners for assistance in planning and executing your move toward BDA in the public cloud.

This is an excerpt of more extensive research by Wikibon that’s available in full with a subscription. Wikibon also will unveil the results of this and other big-data research in detail at an 8 a.m. breakfast Thursday, March 8, at BigData SV, running March 7 and 8 at the Forager Tasting Room & Eatery in San Jose.

Image: geralt/Pixabay