UPDATED 08:00 EDT / JULY 07 2021

AI

IBM open-sources CodeFlare framework for AI models that run on multicloud platforms

IBM Corp. announced a new open-source framework today called CodeFlare that it says helps simplify the integration and efficient scaling up of big data and artificial intelligence workflows on multicloud infrastructures.

The new framework is built atop an open-source distributed computing framework called Ray, extending that software’s capabilities by adding various elements that make it easier to scale up lots of different workloads, IBM said. CodeFlare was first demonstrated at the 2021 Ray Summit in June.

In a blog post, IBM explained that creating machine learning models these days is an intensively manual task. First, researchers have to train and optimize a model, which involves tasks such as data cleaning, feature extraction and then model optimization. IBM said CodeFlare helps to simplify this work.

It uses a Python programming language based interface to create a pipeline, through which it becomes easier to integrate, parallelize and share data. Then, CodeFlare can be used to unify pipeline workflows across multiple cloud computing platforms, without learning a new workflow language for each type of infrastructure.

IBM said CodeFlare pipelines can be deployed on any cloud infrastructure, including the new IBM Cloud Code Engine, which is a serverless platform, and Red Hat OpenShift. CodeFlare also provides adapters for event-triggers such as the arrival of a new file, which means the pipelines can integrate and bridge with other cloud-native ecosystems, IBM said. Further, it enables data to be loaded and partitioned from numerous sources such as cloud object stores, data lakes and distributed file systems.

The main benefit of using CodeFlare to set up new machine learning projects is speed. The company claimed that when one of its users applied CodeFlare to analyze and optimize 100,000 pipelines to train machine learning models, it reduced the time to execute each one from four hours to just 15 minutes.

Speed is important, IBM explained, because data sets are growing larger all the time, meaning that machine learning workflows become more involved and complex. As such, researchers find themselves spending more time on configuring their setups before they can get things done.

Holger Mueller, an analyst with Constellation Research Inc., told SiliconANGLE that AI and machine learning need to marry with multicloud if enterprises are going to be able to create next-generation applications that can run properly on any platform and in any location. So it makes sense that any project that tries to enable this is open source, he said.

“IBM is going after this by open-sourcing CodeFlare as a framework for data workers and developers to create AI models that can run on any cloud,” Mueller said. “CodeFlare runs on RedHat OpenShift and achieves its multicloud capability from that.”

IBM said CodeFlare is being made open-source today, available from the IBM GitHub repository. It’s also releasing various examples of CodeFlare pipelines it has created that run on IBM Cloud and Red Hat OpenShift.

The company said its work with CodeFlare is ongoing, and that over time it will work to enable it to support more complex pipelines. It also aims to improve CodeFlare’s fault tolerance and consistency and enhance data integration and management for external sources. Adding support for pipeline visualization is another long-term goal.

Image: geralt/Pixabay

A message from John Furrier, co-founder of SiliconANGLE:

Your vote of support is important to us and it helps us keep the content FREE.

One click below supports our mission to provide free, deep, and relevant content.  

Join our community on YouTube

Join the community that includes more than 15,000 #CubeAlumni experts, including Amazon.com CEO Andy Jassy, Dell Technologies founder and CEO Michael Dell, Intel CEO Pat Gelsinger, and many more luminaries and experts.

“TheCUBE is an important partner to the industry. You guys really are a part of our events and we really appreciate you coming and I know people appreciate the content you create as well” – Andy Jassy

THANK YOU