Cloud-native application networking startup Solo.io Inc. is taking on the challenge of network traffic management for artificial intelligence workloads, launching a new product called Gloo AI Gateway.

The company said today that generative AI is gaining traction in almost every industry and is expected to see an annual growth rate of 37% over the next 10 years. It believes this growth will cause serious headaches for application developers looking to integrate AI systems into their existing apps given the massive amounts of data they use. The Gloo AI Gateway is designed to solve those headaches by speeding up the communicative abilities of AI apps.

Solo.io is best known for its Gloo Enterprise platform, which is a “service mesh” layer that helps to connect, monitor and secure containers in Kubernetes clusters. Those containers are used to host microservices, which are basically the individual components of cloud-native software applications. The Gloo Enterprise service mesh can be thought of as a networking layer for those microservices to communicate with one another.

Offered as part of the Gloo Enterprise platform, Gloo Gateway is an Envoy-based application programming interface gateway and ingress controller that unlocks multicluster routing capabilities and enables developers to implement various traffic policies to secure, control, and monitor requests made to those clusters.

Gloo AI Gateway builds on the Gloo Gateway offering to facilitate the same high-speed, secure and scalable traffic flows for AI applications, the company said. It’s designed to simplify and accelerate the rate at which AI apps can access large language model APIs, helping to eliminate development friction, boilerplate code and avoidable errors, while also protecting those apps, models and data from unauthorized access. It also enables governance controls to be applied to AI apps more easily, while enhancing auditability and providing more visibility into how those apps are used.

Keith Babo, senior vice president of product at Solo.io, told SiliconANGLE that LLMs have some unique traffic management requirements around semantic caching, token-based rate limiting, prompt enrichment and retrieval-augmented generation, or RAG. “Legacy API gateways are not well equipped to handle these demands,” he said. “Gloo Gateway provides these new capabilities on top of the popular API gateway features used by Solo.io’s customers today.”

In addition, Gloo AI Gateway can help developers to leverage advanced AI integration patterns to support “high-volume, zero-downtime AI connectivity,” Babo said. It also aids in API key management by securely storing those keys as secrets.

According to Babo, Gloo AI Gateway enables developers to use advanced techniques to better handle traffic management for AI workloads, which weren’t possible before. For instance, it paves the way for LLM client application “rate limiting,” where users are restricted to how often they can use the app, using prompt tokens as a rate limit counter.

It also enables prompt auditing in real-time, as well as offline analysis of LLM prompt activity, Babo said. Developers can also initiate prompt guards, meaning they can reject malicious prompts or inappropriate content in real time, he added.

“It also supports AI consumption reporting, or usage reporting on LLM API calls, and it can improve security with its data exfiltration protection mechanisms that prevent personally identifiable information and other sensitive information from being leaked via LLM responses,” Babo said.

Gloo AI Gateway further aids LLM integrations using techniques called “prompt templating” and “prompt enrichment.” Moreover, it can help developers to ensure the accuracy and non-toxicity of their AI models, giving them an easy way to implement prompt guards and data exfiltration controls that can reject inappropriate user requests and sanitize LLMs’ responses.

Besides supporting a range of generative AI-powered applications, such as chatbots, code generators and image generation tools, Gloo AI Gateway also facilitates RAG, which allows LLMs to tap into a company’s proprietary data and knowledge bases to improve the accuracy of their responses.

Image: SiliconANGLE/Microsoft designer