“If everyone is moving forward together, then success takes care of itself.” – Henry Ford

In no other industry is Henry Ford’s word more true than the technology industry. Open source has been a key factor in the innovation and adoption of new technology, enabling businesses to grow in remarkable ways. As Spark Summit 2016 began day two at the Hilton San Francisco, Union Square this morning we heard from collaborators and customers are who using Spark to advance the data-driven organization.

Transforming analytics and today’s data reality

Opening the keynote speech, Ali Ghodsi, CEO of Spark’s founding company Databricks, Inc., began by talking about how analytics is transforming industries, from predictive analytics to anomaly detection. He pointed out the benefits of predictive analytics by noting the ability to predict product revenue, make customer assessments and develop targeted advertising. He also explained how anomaly detection is aiding in fraud detection, risk assessment, and equipment failure. All of this being done through data-driven, real-time analytics applications.

Ghodsi also explained the moving data from data warehouses to data lakes to cloud storage as a necessity due to siloed environments, along with the rapidly growing amount of data becoming a costly and ineffective proposition.

Combating the gap in real-time data-driven analytics?

Three key issues, according to the CEO, are managing data infrastructure, empowering teams to be productive and establishing production ready applications. Databricks is battling these issues with the company’s Cloud-Hosted Platform. explains that the platform provides a ‘Just in Time’ data platform, and integrated workspace and automated Apache Spark management.

End-to-end security for analytics

The big announcement for the company today was the release of Databricks Enterprise Security (DBES). “This is a holistic end-to-end security solution for data analytics and it is the first end-to-end security solution for Apache Spark,” said Ghodsi.

The product protects files, tables, clusters, workflows, notebooks, dashboards and reports. The solution offers role-based access control, auditing and governance, integrated identity management and encryption on disk and on the wire.

Ghodsi discussed many of the Databricks use cases and was excited to give the stage to the upcoming partners and customers who discussed their experiences with Apache Spark.

 Microsoft and Spark creating the intelligence revolution

Joseph Sirosh, corporate vice president of the Data Group for Microsoft came on stage to talk about how far and how fast the age of intelligence is moving. He explained how Microsoft is working on three dimensions. The first being cloud, encompassing offerings such as IaaS PaaS and SaaS. The next dimension is data, where the company is working on operational analytics, in-memory, data lakes and more.

“The most exciting thing, when looking at the past to present is the intelligence dimension,” said Sirosh. He outlined tabular BI, stream analytics, R. Server, ML and deep learning and cognitive APIs as the things to be motivated about in the future.

Transforming data into intelligent action

Sirosh highlighted Microsoft’s Cortana Intelligence Suite that offers one platform eliminating the need for multiple vendors. He talked about how the platform encompasses information management, Big Data stores, machine learning and analytics and produces intelligent actionable information.

He also provided information about Spark for Azure. This solution provides a single execution model for multiple tasks and has 100 times faster performance. Spark for Azure is also developer friendly and offers a notebook experience.

Adding to the open source theme, he talked about Microsoft R Server, and open source Big Data analytics platform. “With Microsoft R Server you only have to write once and deploy anywhere,” Sirosh commented. He outlined the platform and the various integration options.

After talking about several use cases he ended by saying, “Intelligent cloud is here. May the intelligence revolution begin.”

Detecting credit card fraud using Spark

Chris D’Agostino, vice president of technology at Capital One, provided a demonstration of how his company is building a world class defense using a powerful credit card fraud detection platform using Spark. He revealed how working with Databricks and another vendor Visallo helps the company flag suspicious activity and reduce the incidences of credit card fraud and identity theft.

Commitment to the power of Spark

Also speaking about their work and projects with Apache Spark were several notable leaders of technology companies who are all part of the Apache Spark community. Doug Cutting, chief architect for Cloudera and co-founder, Apache Hadoop, Cloudera, along with Ziya Ma, vice president of Big Data for Intel and Rob Thomas vice president of  Product Development for IBM Analytics, IBM also outlined their projects and commitment to the Apache Spark open source community.

For more information about the keynote speakers and Spark Summit 2016, visit spark-summit.org/2016.

Photo by HPE Discover