Mesosphere Inc. is partnering with Typesafe Inc. to provide support for an instance of Apache Spark that can be run atop of the Mesosphere Data Center Operating System (DCOS) on the Amazon Web Services cloud. The integration makes it much easier to deploy Spark in-memory computing clusters in production environments, the company’s said.

Matt Trifiro, senior vice president of marketing for Mesosphere, told SD Times the idea was to allow IT teams to deploy Spark in a matter of minutes. He added that the Mesosphere DCOS can itself be downloaded and installed in a similar amount of time.

“Using a single command, Spark can now be deployed on Docker containers on AWS,” Trifiro said. “We’re also working on an implementation that can be deployed on premise.”

Mesosphere said it’s teaming up with Typesafe because Spark is written in the Scala programming language, which derived from the Java language that Typesafe first developed. Databricks, the company behind Spark’s development, has also jumped on board, certifying the Mesosphere implementation. The development comes just a week after Mesosphere announced it was shipping the first commercially supported version of its DCOS alongside a community edition that runs on AWS.

Interest in using Spark as the foundation for cloud-based Big Data analytics applications is certainly high, but few organizations possess the expertise needed to get the most of it, Trifiro says. However, by employing DCOS on AWS, organizations will at least be able to begin developing Spark apps in the cloud, from where they can determining if they want to run them in production.

Spark is of course has a very close association with Hadoop, and is popular as a faster alternative to MapReduce. But Spark itself doesn’t store any data – rather, data is pulled from the Hadoop cluster and processed in-memory, before being returned to whence it came from.

One interesting tidbit is that both Mesosphere and Spark share a common heritage – they were both developed out of the University of Berkley, which has given birth to a range of open-source technologies aimed at solving challenges associated with deploying and managing applications at scale.

Platforms like Mesosphere’s DCOS first emerged to take advantage of advances in automation to simplify IT infrastructure provisioning and orchestration. But now they’re also helping to simplify application frameworks that invoke the APIs they expose. In other words, they’re helping to simplify the overall IT infrastructure environment at a time when the application frameworks themselves are becoming ever-more complicated.

Image credit: ClkerFreeVectorImages via Pixabay.com