Microsoft is making what it claims is an “extensive commitment” to the Apache Spark Big Data processing engine, launching several new offerings out of preview and into general release.

The move is the latest in Microsoft’s embrace of open-source technologies, a trend that only emerged in the last couple of years after CEO Satya Nadella took over the company. Microsoft made its announcements at the Spark Summit 2016 developer’s conference in San Francisco yesterday.

Microsoft launched Spark for Azure HDInsight in preview last year, promising it would be the “best environment to run Apache Spark” with a managed service in the cloud. Now, Spark for Azure HDInsight has been made generally available, together with a fully managed Spark service from Hortonworks Inc. that’s been “hardened for the enterprise and made simpler for you to use,” the company said.

In addition, Microsoft announced the general availability of R Server for HDInsight later this summer. Currently available in preview mode, R Server for HDInsight include Spark integration for both the on-premises and cloud versions. The “R” refers to the R programming language that’s used for statistical computing and predictive analytics. Microsoft is one of the leading proponents of R following its acquisition of Revolution Analytics in 2015.

Microsoft previously said it’s planning to integrate the commercial R distribution into SQL Server 2016 via SQL Server R Services.

Microsoft also announced that R Server for Hadoop on-premises installations will be made generally available in June. R Server for Hadoop on-premises will support both Microsoft R and native Spark execution frameworks, the company said.

“Combining R Server with Spark gives users the ability to run R functions over thousands of Spark nodes letting you train your models on data 1000x larger and 100x faster than was possible with open source R and nearly 2x faster than Spark’s own MLLib,” according to Microsoft’s blog post.

Last but not least, Microsoft said PowerBI now supports Spark Streaming. The previously announced Spark support in PowerBI has been expanded with support for Spark Streaming scenarios, the company said.

While many consider that Apache Spark is a competitor to Hadoop, Microsoft’s announcements indicate that the company believes the two platforms are more complimentary than competitive. However, despite its embrace of Spark, that hasn’t stopped Microsoft Research from pursuing a new, potentially competing project called Prajna, which is an open-source distributed analytics platform designed for building cloud services that make use of Big Data analytics.

Image credit: Patrick Foto ;) via Flickr.com