Data integration vendor SnapLogic Inc. is taking its platform to the cloud with an extension that nontechnical users can employ to create complex data processes.

Those processes can run on cloud-based Apache Hadoop platforms such as Amazon Web Services Inc.’s Elastic MapReduce, Microsoft Corp.’s HDInsight and Google Inc.’s Cloud Dataproc.

SnapLogic eXtreme can be used to build integration logic that’s automatically transformed into Apache Spark pipelines through a drag-and-drop process that replaces complex code, the company said. As a bonus, SnapLogic is throwing in provisioning services that can save customers money by launching cloud servers only when needed.

Big data whack-a-mole

Big data processing is increasingly moving to the cloud as the complexity of on-premises deployment makes projects “a high-tech whack-a-mole game,” said James Markarian (pictured), SnapLogic’s chief technology officer. “To make all of the parts work, we’re constantly fighting fires against the installation or the vendor themselves.”

Modern big data projects are incorporating more external data than was true in the past, and usage patterns are becoming less predictable, which makes cloud a particularly attractive option, the CTO said. “This is a demand-side pull into the cloud by our customers,” he said.

SnapLogic positions its platform-as-a-service as an easy-to-use, scalable integration layer that can connect almost any application to anything else. It comes with a library of more than 450 connectors, called “snaps,” that automate the process of weaving services and data from multiple sources into integrated applications.

Spark in a box

The core product remains unchanged with this release, but the extension of SnapLogic functionality into the cloud is a significant step for the company, as is the integration with the popular Spark analytics framework. “If you know SnapLogic, you’re already a Spark developer,” he said.

The eXtreme engine isn’t limited to Spark, however. “We have the same design environment we apply to everything,” Markarian said. He mentioned Apache Flink as a possible future execution target but said no decision has been made. Similarly, the “ephemeral plex” cluster management component can potentially also be used on-premises, but the company hasn’t committed to doing that.

Ephemeral plex can translate into significant savings compared with conventional server subscriptions, Markarian asserted. “We can dynamically provision the nodes as needed; we have our own spin-up and shutdown cloud cluster environments and can mange them very efficiently,” he said. “It’s all transparent to customers.” The feature should be particularly appealing to customers who want to run predictable workloads on low-cost reserved instances and use costlier metered processing for spikes in capacity.

The product is going into beta test this month and will be generally available over the summer. The company sells bundles of snaps by usage tiers on a subscription basis, but will probably switch to a fixed-fee model for eXtreme, Markarian said. However, no decision on pricing has been made.

Markarian was interviewed on theCUBE, SiliconANGLE’s livestreaming studio, at BigData NYC in December 2016. Here’s the full video for some background:

Image: SiliconANGLE