Akamai Technologies Inc. is expanding its developer-focused cloud infrastructure platform with the launch of Akamai Cloud Inference, a highly distributed foundation for running large language models closer to their real-world users.

The new service is built atop Akamai Cloud, which Akamai claims is the world’s most distributed cloud infrastructure platform, allowing it to overcome the problems associated with centralized, cloud-based models.

According to Adam Karon, Akamai’s chief operating officer and general manager of cloud technology, one of the main problems with running powerful LLMs in centralized cloud platforms like Google Cloud and Microsoft Azure is low latency. Simply put, the data those models feed on is located far away from where the model is hosted, and the latency means their performance can take a substantial hit.

“Getting AI data closer to users and devices is hard and it’s where legacy clouds struggle,” Karon said. “While the heavy lifting of training LLMs will continue to happen in big hyperscale data centers, the actionable work of inference will take place at the edge.”

Karon said Akamai’s unique network, which was built over the last two-and-a-half decades, will become “vital” for the future of AI, setting the company apart from every other cloud provider.

The distributed advantage

Akamai’s big advantage is that it began life as a content delivery network, building up a system of geographically distributed servers that host web content close to users. By caching websites closer to their users, it improves website load times and lowers latency, improving the overall experience.

Having established itself as one of the world’s leading CDNs, Akamai expanded into the cloud infrastructure business following its $900 million acquisition of the developer cloud company Linode LLC. Since then, it has rapidly built out its offerings to include services such as storage, compute and networking, competing with traditional cloud infrastructure firms such as Amazon Web Services Inc. and Microsoft Corp.

Its cloud infrastructure is made up of more than 4,100 points of presence spread across ore than 1,200 networks in more than 130 countries, making it much more distributed than any of its traditional cloud computing rivals.

The Akamai Cloud Inference service is hosted on the company’s distributed CDN servers, providing tools for companies to build and run AI applications much closer to their users. Inference is the process where trained AI models generate outputs based on the user’s prompts. It’s often referred to as the operational phase of AI, where models apply what they have learned during their training.

Karon likened LLM training to the process of creating a map. It’s a slow and resource-intensive process that involves gathering data, analyzing the terrain and plotting routes. On the other hand, AI inference is more like using a GPS device, he said.

“It’s about instantly applying that knowledge, recalculating in real time and adapting to changes to get you where you need to go,” he said. “Inference is the next frontier for AI.”

Supercharged AI inference

The company claims it can provide triple the throughput for AI inference and reduce latency by up to two-and-a-half times over traditional cloud infrastructures. It also provides significant cost savings on AI inference workloads, the company says.

Akamai Cloud Inference consists of compute infrastructure resources ranging from traditional central processing units for low-cost, fine-tuned inference to powerful graphics processing units and tailored application-specific integrated circuits or ASICs, which provide additional horsepower for more demanding workloads. Its compute services are integrated with Nvidia Corp.’s AI Enterprise software suite to cater to developer’s needs.

The service also provides advanced data management capabilities thanks to Akamai’s partnership with Vast Data Inc., which provides access to a modern “data fabric” that’s customized for AI applications. Vast Data’s storage infrastructure helps to provide rapid access to data as it’s generated in real time, aided by Akamai’s close geographic proximity to end users.

Akamai’s inference platform supports containerized workloads too, enabling demand-based autoscaling, multicloud portability and performance optimization for AI applications. These capabilities are delivered through the Linode Kubernetes Engine, which is an enterprise-grade cloud orchestration platform built for managing large-scale cloud workloads.

Finally, Akamai Cloud Inference sets itself apart with its edge compute capabilities, enabling it to execute LLM inference on serverless infrastructure at the edge of the network, supporting latency-sensitive AI applications.

Akamai believes its cloud inference services are going to be in big demand, citing a report by Gartner Inc. that predicts 75% of the world’s data will be generated outside of centralized data centers and cloud regions by the end of the year. By processing this data closer to where it’s created, AI applications can make decisions faster and provide real-time insights in the most remote environments. The company says this will be of immense value for applications such as autonomous vehicles, AI-powered crop management, retail shopping experiences and more.

Image: SiliconANGLE/Dreamina