Hewlett Packard Enterprise Co. today said that it has acquired Determined AI Inc., a startup with an open-source software platform capable of accelerating the speed at which machine learning algorithms can be trained by dozens of times.

The acquisition will boost HPE’s machine learning capabilities in a time when AI is becoming an increasing priority for buyers of its supercomputers and servers.  

San Francisco-based Determined AI launched in 2017 and has raised more than $13 million in funding to date. Its investors include, among others, Alphabet Inc.’s GV fund, LG Electronics Co. Ltd. and CRV. 

After developers write a new machine learning algorithm, they hone its computing capabilities by having it crunch large volumes of training data. This process helps increase the accuracy and speed of AI software. As the number of artificial neurons in the average machine learning algorithm has increased over recent years, so has the amount of hardware necessary to carry out the training process.

Determined AI’s platform makes it easier to set up the large amounts of hardware needed to build complex AI algorithms. The platform spins up servers, creates network connections between those servers so they can coordinate their work and splits workloads across the machines. Determined AI sets up the environment with a fault-tolerant configuration that allows AI algorithms to continue their training even if one of the servers goes offline.

The platform speeds up machine learning projects because setting up an AI training environment is normally a manual task. As a result, engineers have to spend valuable time on chores such as ensuring the number of servers spun up doesn’t exceed the amount of infrastructure allocated to the project by their company. Moreover, in the absence of fault tolerance features such as those provided by Determined AI, a server outage can require engineers to restart a training run from scratch, which increases project durations even further.

The other way Determined AI promises to accelerate AI development is by optimizing the training process itself at the software level. To do so, the startup’s platform uses a customized version of Horovod, an open-source AI training framework originally released by Uber Technologies Inc. in 2017. The startup says its version of the framework can speed up training by dozens of times in some cases, reducing the duration of some projects from days to hours.

Determined AI’s Horovod implementation facilitates the speedup by increasing so-called scaling efficiency. As engineers scale an AI training environment by adding more servers, the amount of computing power provided by each individual server decreases in lockstep due to various technical factors. Horovod can provide unusually high scaling efficiency of up to 90%, meaning only 10% of servers’ processing power is lost as the AI training environment grows.

HPE will use Determined AI’s technology to boost its supercomputing capabilities. To that end, the company plans to fold the startup into the High-Performance Computing and Mission Critical Solutions division, which includes its supercomputer business.

The context for the acquisition is that supercomputers are increasingly being used to run machine learning workloads. This trend is reflected in the fact that a growing percentage of the new systems coming online feature graphics processing units. Thanks to the Determined AI acquisition, HPE can now better support high-performance computing customers running AI workloads by providing them not only with supercomputing hardware but also machine learning software.

Another notable aspect of the Determined AI acquisition is that the startup’s platform can run in the cloud. That’s important because supercomputing workloads, like other on-premises applications, are increasingly moving to the cloud. One recent project underscoring the trend is HPE’s April collaboration with Microsoft to set up a supercomputer in an Azure data center for the UK Meteorological Office. Because the Determined AI platform can run just as well off-premises as on-premises, HPE could offer it to the growing number of customers taking their high-performance computing projects outside the corporate network.

The Determined AI platform should also prove useful for buyers of HPE’s machine learning servers. HPE has multiple lines of GPU-equipped machines for machine learning projects that don’t necessitate a supercomputer but still require considerable on-premises computing capacity. HPE, citing research from International Data Corp., said that the accelerated AI server market is expected to grow 38% annually until 2024 to reach $18 billion.

“Over the last several years, building AI applications has become extremely compute, data, and communication intensive,” said Determined AI Chief Executive Officer Evan Sparks. “By combining with HPE’s industry-leading HPC and AI solutions, we can accelerate our mission to build cutting edge AI applications and significantly expand our customer reach.”

Determined AI offers its platform in an open-source edition and a paid version with additional features for enterprises. The startup said today on its blog that the open-source edition will continue to be available following the acquisition. Moreover, HPE plans to continue investing in its development going forward.

Acquisitions have played a central role in HPE’s strategy for the supercomputing market. To strengthen its presence in the segment, HPE two years ago paid $1.3 billion to buy Cray Inc., a major supercomputing maker responsible for building many of the world’s fastest high-performance computing systems.

Photo: HPE