Following the adage that two heads are better than one, speakers from Hortonworks Inc. and IBM Corp. unveiled more details today about their partnership to combine data management platforms with data science and machine learning capabilities. The companies’ presentations were delivered during the opening keynote session at the DataWorks Summit in San Jose, California.

The news released today was that Hortonworks will combine forces with IBM to make machine learning and data science technologies more readily available for Apache Hadoop users. (Hadoop is open-source-based software used for storing, processing and analyzing big data.) Specifically, the IBM Data Science Experience and Big SQL will be packaged together with the Hortonworks Data Platform to create one integrated solution.

“This will be great for the open-source community and enterprise customers. It will also help us get to the next generation of end-to-end analytics,” said Rob Bearden, chief executive officer of Hortonworks.

Collaboration necessary for competition

The need for collaboration was a central message in the remarks delivered by many of the speakers during the DataWorks Summit opening keynote. The competitive environment facing Hadoop has put pressure on IBM to strengthen its hand in machine learning, expertise that Hortonworks can provide, according to Rob Thomas, general manager of IBM Analytics.

“It’s becoming a team sport. Data science has to have collaboration,” Thomas said.

Today’s announcements are an important second step for IBM, Thomas explained. The company had previously deployed IBM Watson to the mainframe in February in an effort to help enterprises with transactional data processing.

“Now this will open up the world of data science directly onto your Hadoop data,” he said.

The integration of the IBM Data Science Experience is designed to make the development of new models an easier and faster process. Dinesh Nirmal, IBM’s vice president of analytics development, described how 80 percent of the time spent in data science is focused on cleaning the data.

“The deployment of the model is where enterprises struggle,” Nirmal said. “The aim is simplicity, and you need automation to pick the right algorithm.”

The IBM Data Science Experience will offer data visualization tools using Spark, and companies can test models once deployed.

The integration of HDP with IBM’s Big SQL engine for Hadoop is designed to provide Hortonworks clients with a recognizable way to better manage their data needs.

“We’ve come together to create and enable this modern data architecture,” said Scott Gnau, chief technology officer of Hortonworks Inc. “We want to take advantage of the commitment to open source and leverage our ecosystem to deliver real-time value for customers in the data science space.”

More Hortonworks news is coming

Gnau told the DataWorks gathering that there will be more announcements to come, dropping hints that centered around “creating a seamless experience” across the data life cycle and “leveraging containerized applications.”

The Hortonworks and IBM collaboration underscores the increasing importance for many industries to develop tools around machine learning and data science that will enable new technologies. The use of data to build and train software is leading to key advances in fields, such as the development of an autonomous car and healthcare, according to Jim McHugh, vice president and general manager with Nvidia Corp.

“I can now train a system to recognize an image better than a radiologist,” McHugh said.

John Pressley, the director of IT for Duke Energy Corp., highlighted the importance of data science for his industry, which is undergoing a rapid transformation. “We need data-driven insights to get us to this new energy revolution,” he said. “This isn’t a fad.”

Stay tuned for the complete keynote video, and be sure to check out more of SiliconANGLE’s and theCUBE’s independent editorial coverage of DataWorks Summit.

Photo: SiliconANGLE