IBM Corp. has spent years developing its artificial intelligence platform, Watson, looking to the open-source Apache Spark platform for innovative machine learning capabilities that can help push Watson into business verticals for manufacturing, healthcare, cybersecurity and retail. Having made the IBM Spark Technology Center a part of its AI-driven ecosystem, IBM has opened up the analytics capabilities of Watson in an effort to simplify the machine learning for progressive neural networking using Apache SystemML. The idea is to help businesses move quickly into the new domains of machine learning.
“We have an optimizer underneath [the system] that takes care of the details of how you get data into processing, how you get the processing moved to the data. … All those decisions are taken care of and … that lets us implement deep learning much more quickly,” said Frederick Reiss (pictured), chief architect at the IBM Spark Technology Center.
Reiss spoke with Jeff Frick (@JeffFrick) and George Gilbert (@ggilbert41), co-hosts of theCUBE, SiliconANGLE Media’s mobile live streaming studio, during the BigData SV event in San Jose, CA, about the complexity of Spark and how IBM is using the platform to enhance machine learning. (*Disclosure below.)
Developing simplicity within Spark
In a report from IBM, Apache Spark for the Enterprise: Setting the Business Free, 80 to 90 percent of companies have big data analytics listed as one of the top priorities. Nine out of 10 of the leading companies recognize the competitive advantage of the technology. However, in most cases, implementation in the enterprise falls to the data scientists.
The IBM Spark Technology Center is working on the Spark platform to build on it’s core technology to enhance its machine learning capabilities for new trends in neural networking. Reiss explained that neural network-based capabilities are not available with Spark, but they are accessible in SystemML.
SystemML removes many of the complications associated with Spark and machine learning, he explained.
“Just to be clear, linear algebra really is the language of machine learning. SystemML is scalable linear algebra on top of Spark. It is actually better than what’s built into Spark,” Reiss said.
He continued to explain how IBM had developed tight integration tools into Spark to provide capabilities that pass Spark dataframes directly into Syste ML and then move them back. “Once you have written your SystemML algorithm, it plugs into Spark like all the algorithms,” Reiss noted.
For the people working at the IBM Spark Technology Center, finding the parallelism in machine learning is what they enjoy doing. Outside of the building is a different story.
“A lot of people working in data science see parallelism as a tool. They want to solve the data science problem, and SystemML lets you focus on solving the data science problem because the system takes care of the parallelism,” said Reiss.
The real potential of the projects happening at the Spark Technology Center revolves around the value it brings to the industries such as automotive, banking, healthcare, manufacturing and retail. These industries are just a few that are making big investments in machine learning to monetize and reap the benefits of big data, Reiss concluded.
Watch the complete video interview below, and be sure to check out more of SiliconANGLE’s and theCUBE’s coverage of BigData SV 2017. (*Disclosure: Some segments on SiliconANGLE Media’s theCUBE are sponsored. Sponsors have no editorial control over content on theCUBE or SiliconANGLE.)