Distributed file system developer Alluxio Inc. today added interfaces to Posix and Amazon Web Services Inc. S3 storage to its open-source Data Orchestration Platform, in a bid to make it easier for business intelligence and model training to be conducted using popular machine learning and analytics frameworks such as TensorFlow and PyTorch.

Specifically, Alluxio 2.5 adds Posix data access via a Fuse-enabled Java Native Interface, which is a programming framework that enables Java code running in Java virtual machines to interact with native applications. This improves the performance up to fivefold for high-performance and high-concurrency workloads such as machine learning training.

Support for Posix is significant, said Chief Executive Haoyuan Li, who co-developed the distributed file architecture while a graduate student at the University of California at Berkeley. The company previously focused on analytic workloads using open-source frameworks such as Apache Spark, Apache Presto and Apache Hive, which primarily work with the Hadoop HDFS format.

“HDFS’ day has come and gone,” Li said. “Data scientists are replacing it with cloud and object stores. The dominant frameworks like PyTorch make it easy to leverage the Posix API for two- to three-times performance improvement,” meaning that models that formerly took a year to train can now be completed in four months.

S3, HDFS and Posix comprise a majority of the application program interfaces used by data-driven applications and data management tools, the company said. “Often object stores have a performance downside for things like metadata,” Li said. “Having Alluxio in front of object stores reduces that performance downside.”

Administrators can now manage the entire Alluxio file system namespace across existing users using a standard object storage console, making it easier to integrate files into large-scale data pipelines.

New and improved storage connectors have been added to simplify onboarding with major cloud providers, including built-in authentication and improved performance. The new release also improves S3 API access for better compatibility with browsing software such as S3 Browser, which is used to manage S3 storage.

Founded in 2015, Alluxio has raised $23 million in funding and has good momentum, Li said. “We are seeing cloud providers increasingly using Alluxio as the default for their orchestration,” he said, and the company is collaborating with five of the top six cloud providers. The company said it became cash flow positive as its business grew 350% in the most recent fiscal year.

Image: Interwork/Facebook