The data analytics arms race continues to escalate, with major players battling to better predict what consumers want — sometimes before they even know themselves. The newest weapons in analytics arsenals are “Systems of Intelligence,” the natural evolution of traditional “Systems of Record” and more modern “Systems of Engagement.”

But it can be tough to separate the reality from the marketing hype. Wikibon analyst George Gilbert joined the SiliconANGLE team and guests from all over the world, including from Vancouver, New Delhi and Rio De Janeiro, to summarize the differences between the systems, share up-to-the-minute intelligence, and talk about how open-source projects like Apache Spark are taking the data analytics game to the next level.

What makes “Systems of Intelligence” so special?

Systems of Record (SoR), Systems of Engagement (SoE) and Systems of Intelligence (SoI) all arose from the needs of enterprise to store, manage and interpret data, but today they play vastly different roles. SoR was the earliest attempt (over 50 years ago) to manage and store data electronically, with airlines being among the first to take advantage of these systems to manage reservations.

While some define SoRs simply as databases, Gilbert suggested that they are systems that “automate business processes and report on historical performance” – the “equivalent to steering a ship by looking backwards at its wake.” They can pump out reports on demand, but only after the fact — and it’s still up to human beings to make changes based on historical trends.

Kirk Borne, principal data scientist at consulting firm Booz Allen Hamilton, Inc., went further and suggested that Enterprise Resource Planning (ERP) software, Customer Relationship Management (CRM) software and databases sometimes masquerade as next-gen analytics, even though they primarily store, synchronize and manage data.

CrowdChat participant Jasdeep Singh countered that “sophistication lies in the way they run,” especially as reports and UI have become more user-friendly. But Gilbert responded that Systems of Intelligence stand out by informing consumers “in real-time at the point of interaction” and anticipating their actions. This combination of semantic intelligence, behavior analytics and ability to influence consumers take SoIs even beyond their precursor, SoEs, which include pieces like Customer Experience Management. But “SoIntelligence is predictive, prescriptive, cognitive,” according to Borne, as well as context and advanced analytics.

Where are we on the “Systems of Intelligence” customer adoption curve?

One CrowdChat participant, Jen Cohen-Cheplick, senior manager, global marketing campaigns, at Syncsort, Inc., pointed out that “organizations with a tangible benefit will be the earliest adopters,” such as online gaming organizations who use data from players to change the experience in real-time.

Rodrigo Gazzaneo, EMC executive briefing senior manager for Rio de Janeiro, pointed out that other key adopters include proponents of the Internet of Things (IoT) who have a huge amount of raw data to work with. By adding machine learning and even automatic adjustments based on predictions, everything from smart grid systems to healthcare protocols can respond instantly and dynamically to changes in usage without human intervention.

But technological advances alone won’t push consumers and enterprise into the adoption of new systems. As Cheplick highlighted, “You need a shift in skills and in organizational mindset to make the transition.”

What are the best applications of Systems of Intelligence?

CrowdChat participants seemed divided between prediction and automation as the most useful feature of SoIs. On the one hand, as Wikibon cofounder and CTO David Floyer pointed out, automation increases ROI for businesses, especially when it’s system-to-system, not system-to-people. Prediction allows companies to stop problems before they start. Apps like Workday allow companies to predict which employees might be preparing to leave so they can intervene when necessary.

SiliconANGLE founder John Furrier highlighted several possible applications, from running datacenters, apps and IoT to other areas where “lots of things are connected and taking action on data is the big thing.”

Cheplick also emphasized the importance of real-time, actionable data, saying that “automating between systems means the data will actually be used in a timely fashion – and not left up to humans to incorporate.”

Kevin Petrie, technology evangelist with Attunity, Ltd., brought up healthcare as an industry ripe for SoI applications, saying, “IBM Watson can diagnose and treat conditions better than doctors. Healthcare professional roles will be more consultative, relationship based in the future as AI plays traditional doctor role.”

What is the impact of Spark on the trend toward SoI?

The pre-chat CrowdChat poll indicated that most respondents thought Spark would “turbocharge” Hadoop rather than disrupt it, which makes sense given that they were designed to complement each other. Gilbert pointed out that Spark “makes a lot of analytics easier and faster by running different workloads on the same engine,” although it needs some improvements in performance.

Floyer responded by saying, “Hadoop is batch and most efficient/greatest throughout,” adding that Spark is great for “microbatch” analytics. It can work more quickly, but is less efficient overall and lacks a distributive file system.

Some participants seemed tempted to relegate Hadoop to the status of the new “tape” or “spinning rust” of the storage world, expecting Spark to do all the analytical heavy lifting. But most agreed that there was room for both, with Borne even going so far as to call them “The Two Towers” of Lord of the Rings fame, with Spark being the fast, in-memory option and Hadoop continuing to rule over batch processing.

Petrie questioned why Spark is considered to be at the “slow” end of innovation as in the slide deck Gilbert shared. He answered that “Spark libraries must evolve to integrate with each other,” while the “Wild West ecosystem of databases” can evolve independently at its own pace. This restricts future innovation, as Petrie concluded.

How do Spark and Hadoop coexist?

How does this play out in practice? Hadoop still has plenty of room for growth, according to Gazzaneo, and as Cheplick added, it’s important “to use the right ‘tool’ for the job, depending on data type, batch vs. real-time, etc.” The two systems can definitely coexist and will likely continue to do so for some time.

What does the future look like? Gilbert concluded, “Hadoop is becoming a key part of enterprise infrastructure. Some part of it — at least HDFS and YARN should be [a] common foundation — for a while. Apache Spark will have its own take on storage integration at some time.”

Until that happens, the two systems will continue to work in tandem to empower enterprise in its quest to model and predict everything from consumer behavior to the movement of cars on the road — and anything else that can be visualized by the innovators of the future.

Check out the entire CrowdChat below: