Lyft open-sources Flyte tool for managing machine learning workflows
Ride-sharing company Lyft Inc. today said it has open-sourced a new debugging tool for artificial intelligence data that its pricing, locations, estimate time of arrivals, mapping and self-driving developer teams have been using in-house for the last three years.
Flyte is described by Lyft as a “structured and distributed platform for concurrent, scalable and maintainable machine learning workflows.”
“With data now being a primary asset for companies, executing large-scale compute jobs is critical to the business, but problematic from an operational standpoint,” Lyft engineers Allyson Gale and Ketan Umare wrote in a blog post on Medium. “Scaling, monitoring, and managing compute clusters becomes a burden on each product team, slowing down iteration and subsequently product innovation. Moreover, these workflows often have complex data dependencies. Flyte’s mission is to increase development velocity for machine learning and data processing by abstracting this overhead.”
Gale and Umare explained that Flyte is a “multi-tenant” service, which means teams can work on separate data repositories and employ them without affecting the rest of the platform. Software code is versioned and containerized with its dependencies to ensure all executions are still reproducible. Developers can invoke different parameters, or variables internal to AI models that define its skill relating to a particular problem. Flyte can also use cached outputs from previous model executions to save time and memory.
Flyte also can link tasks together and pass data among them via a Python-based domain-specific programming language. Further, because every entity in Flyte is immutable, with each change captured as a new version, it’s possible to iterate or rollback workflows quickly and share versioned tasks across workflows.
“Flyte is built to power and accelerate machine learning and data orchestration at the scale required by modern products, companies, and applications,” the company said. “Together, Lyft and Flyte have grown to see the massive advantage a modern processing platform provides, and we hope that in open-sourcing Flyte you too can reap the benefits.”
Constellation Research Inc. analyst Holger Mueller said Flyte looks to be a useful and proven machine learning management tool that should be attractive to some companies as it spans multiple underlying compute frameworks.
“Executives will need to validate that the components Flyte has are the ones they want to build their next generation applications with, before considering it further,” Mueller said. “If there’s a fit, it’s well worth the detailed evaluation.”
Since you’re here …
Show your support for our mission with our one-click subscription to our YouTube channel (below). The more subscribers we have, the more YouTube will suggest relevant enterprise and emerging technology content to you. Thanks!
Support our mission: >>>>>> SUBSCRIBE NOW >>>>>> to our YouTube channel.
… We’d also like to tell you about our mission and how you can help us fulfill it. SiliconANGLE Media Inc.’s business model is based on the intrinsic value of the content, not advertising. Unlike many online publications, we don’t have a paywall or run banner advertising, because we want to keep our journalism open, without influence or the need to chase traffic.The journalism, reporting and commentary on SiliconANGLE — along with live, unscripted video from our Silicon Valley studio and globe-trotting video teams at theCUBE — take a lot of hard work, time and money. Keeping the quality high requires the support of sponsors who are aligned with our vision of ad-free journalism content.