Batch, continuous and interactive: Those are the three data ingredients needed to power tomorrow’s analytic and Internet of Things applications. But getting them to harmonize in an end-to-end model is taking Spark and other software developers through their paces.

At Spark Summit East 2017 Boston, George Gilbert (@ggilbert41) (pictured at left), Wikibon analyst and co-host of theCUBE, SiliconANGLE Media’s mobile live streaming studio, presented Part 1 of a Wikibon Big Data Market Update. (See part 2 here.) He said that for these applications to manifest, the technology will have to mature. (*Disclosure below.)

So where are the growing pains today?

“Right now you have some pretty tough trade-offs between integration, which provides simplicity, and choice and optimization, which give you fragmentation,” he said, adding that certain skillsets will also need to develop.

Indeed, Gilbert said that one of the main pieces needed in end-to-end data apps, particularly the IoT type, is still MIA: “We don’t even have a data store that can go all the way from ingest to serve,” he said.

He explained that this is crucial for industrial IoT apps, which have to ingest millions of events per second coming in from devices and serve them as real-time business intelligence updates.

Not even Spark, for all its popularity, can offer this yet, Gilbert stated.

All of which is not hard to understand given the labyrinth of processes it might require with the tools now on hand.

Gilbert illustrated this with an axis for developers: On the X axis, there are the processes: Ingest, explore, process, serve. On the Y axis, there are the “fistfuls” of products needed to execute them; the products effectively multiply the processes.

“It’s a mess,” Gilbert said.

The case for integration

Gilbert’s co-host Dave Vellante (@dvellante) (pictured at right) agreed, saying, “That’s why everybody’s pushing on this whole unified integration — that was a major theme that we heard throughout the day today.”

Gilbert and Vellante agreed that there is hope from big Internet companies with the wherewithal to piece processes together and cloud service providers who integrate Spark’s APIs to lessen complexity for end-users.

Watch the complete video interview below, and be sure to check out more of SiliconANGLE and theCUBE’s coverage of the Spark Summit East 2017 Boston. (*Disclosure: TheCUBE is a media partner at the conference. Neither Databricks nor other sponsors have editorial control over content on theCUBE or SiliconANGLE.)

Photo by SiliconANGLE