The first two days of Spark Summit East 2016 played host to the exploration of a wide range of possibilities for further development of Spark-based systems, though this was just scratching the tip of the iceberg in terms of what analysts expect to emerge.

George Gilbert, Big Data and analytics analyst at Wikibon, spoke to Dave Vellante, cohost of theCUBE, from the SiliconANGLE Media team, about the first-ever Spark industry forecast. Gilbert and Vellante presented and examined some of the expectations for the implications of Spark’s use throughout the next decade, moving past Spark’s immediate influence to determine how the systems using it might alter the tech landscape and what sort of market reactions might arise.

Intelligence systems and insight production

One of the major points of the discussion focused on systems of intelligence in regards to their applications, development and challenges. Personalized real-time services, machine-learning models, back-office systems information, and systems of engagement were all identified as crucial parts of this emerging infrastructure, though for all of their complexity, the output being in an easily-decipherable form is essential.

“As our tools get better, we are shortening the amount of time to get the insight,” Gilbert stated.

Information from back-office systems, systems of engagement, and a simplification of the technology component of data management were other high-priority aspects. Examining systems that handle monetary transactions, Gilbert stated that deep integration and short latency are “where all the money is made.”

With intelligent systems having a shorter time to insight, and higher quality as a result of reduced batch operation dependencies, the integration of networked and interdependent processes was identified as a major goal for savvy enterprises. “When you tie all those systems together, that’s where you make the value,” he said.

Customers and connectivity

Engaging customers “in a back-and-forth conversation” was touched on as one of the ways in which brands could reinforce their proprietary value, as well as opening up new avenues for data collection. With these sorts of customer engagement efforts, “You never are done adding to your understanding of the customer,” Gilbert noted.

However, as modern tech continues to move forward, the market share of non-custom apps is dwindling. “It’s very, very difficult to build package systems for these [analytics systems],” Gilbert pointed out, mentioning the old days of hundreds of identical installation CDs being mailed out as an example of how drastically things have changed.

Looking to the future of Big Data, Vellante laid out three options: humans can get smarter, data services can improve or data management software can be scaled. Realistically, due to the volume of data and its increasing collection, only the last of these options has a chance of keeping up.

As such, Big Data apps are expected to rise significantly over the next decade, and there’s a recognized need to overcome conditioned expectations from the past few decades so as to anticipate market disruption. Similarly, acknowledging a data gap and working to address that omission will provide a boost to those enterprises that can manage it.

‘Web 2.0 and beyond’

Summing up some of the projections, Gilbert condensed things. “We’re going to still be driven, for the most part, by mostly custom apps. That’s going to put a bit of a lid on the growth rate. … Then there’s the issue we’re going to have to see, that this journey to systems of intelligence, it’s not like you can skip a step.” Though he felt that “Spark is really taking the industry by storm,” Gilbert recognized that there were still areas into which it had yet to make a significant entry.

Despite the turmoil expected from the upcoming decade as these systems and their effects begin to be felt, Gilbert presented a positive outlook. “People-processed technology has always been the foundation of how we evolve in technology,” he concluded.

Watch the full video interview below, and be sure to check out more of SiliconANGLE and theCUBE’s coverage of Spark Summit East 2016. And join in on the conversation by CrowdChatting with theCUBE hosts.

Photo by SiliconANGLE