Wikibon recently completed its 2017 Big Data Worldwide Forecast. Authored by Wikibon analyst George Gilbert, with help from Wikibon’s Ralph Finos and Peter Burris, the report covers market sizing, growth and trends.

This year, we focused specifically on the big data software technologies most important to establishing the strategic business capabilities necessary to achieve big data outcomes. Here are some of the key findings from the report:

• In 2016, the total worldwide market for big data hardware, software, and services grew 22 percent, to $28.1 billion, making big data one of the more attractive segments in the technology domain (pictured).
• Our models show that total spending on big data hardware, software, and services will experience a 12 percent compound annual growth rate through 2027 to $9.2 billion, largely driven by software.
• However, even as big data continued to capture the imagination of businesses everywhere, serious questions blemished the market’s glittering surface. In particular, failure rates with big data projects are high, even by information technology standards.
• Partly in response to high project failure rates, the big data market is rapidly evolving. As we predicted, the focus on infrastructure is giving way to a focus on use cases and applications, creating sustainable business value with big data capabilities.
• To avoid infrastructure issues, enterprises are especially interested in utilizing the cloud for big data applications.

A Peak At Big Data Software Segments

Like many technology sectors, much initial interest in big data focused first on hardware and infrastructure software. In the big data domain, these were the technologies required to capture, store and process large volumes — “big” — of data. Thus, users spoke of the need to build “data lakes” capable of handling high data arrival rates at scale. While this capability remains important, setting up the clusters, databases and data movement tooling is not enough. Today’s leaders are refocusing attention on tooling and business capabilities for building big data software and systems. Here’s a quick overview of trends within each big data software subsegment:

• Application databases accrue functionality of analytic databases. Analytics increasingly will inform human and machine decisions in real time. The category totaled $2.6 billion in 2016. Growth slowly tapers off from 30 percent CAGR to reach $7.7 billion in 2022.
• Analytic databases evolve beyond data lakes.  MPP SQL databases, the backbone for data lakes, continue to evolve and eventually will become the platform for large-scale, advanced, offline analytics. The sector totaled $2.5 billion in 2016. Growth in this sector will be a bit slower than the overall category, reaching $3.8 billion in 2022.
• Online options grow at the expense of batch infrastructure spending. This category, which includes offerings such as Spark, Splunk and Amazon Web Services EMR, totaled $1.7 billion in 2016. Growth will be strong through 2022, reaching $6.1 billion in 2022.
• Continuous processing infrastructure is boosted by IoT applications. The category will be the basis for emerging microservices-based big data applications, including much of intelligent systems of engagement. It totaled $200 million in 2016 but takes off to reach $800 million by 2022.
• The data science tool chain is evolving into models with APIs. Today, data science tool chains require dedicated specialists to architect, administer, and operate. However, complex data science toolchains, including those for machine learning, are transforming into live, pre-trained models accessible through developer application programming interfaces. The cottage industry of tools today totals $200 million, growing rapidly through 2022 to reach $1.8 billion.
• Analytics applications are mostly custom-built today. They will become more pervasive in existing enterprise applications in addition to spawning new specialized vendors. The market in 2016 totaled $900 million. It will outpace growth in all other big data markets, reaching $6.3 billion in 2022.

Action Item: Enterprises are accreting experience with big data quickly, but not fast enough to keep up with the imagination and demands of business leaders. In 2017 and beyond, chief information officers need to focus on the strategic business capabilities necessary to more rapidly create, cost-effectively administer and successfully integrate advanced analytic systems.

Image: Wikibon