Docker Inc., a company that’s best known for its containerized software development tools, is turning its attention to generative artificial intelligence with the launch of a new service called the Docker Model Runner.

In a blog post, Docker said Model Runner is all about helping developers to build and run AI models locally on their own hardware and within their existing workflows. It’s designed to enable the performance and control needed to create, experiment with and even deploy AI models directly into production.

The company explains that local AI development is becoming more popular because it provides advantages in terms of model performance, cost and data privacy. Despite this, it remains extremely challenging, with fragmented tooling, disconnected application development workflows and hardware compatibility issues just some of the headaches involved.

For instance, developers will often have to integrate multiple tools and configure their infrastructure environments manually, while managing their models separately from their container workflows. Moreover, storage is a serious hassle too, since there’s no standardized way to store, share or serve AI models. As a result, developers are held back by a disjointed experience and rising costs.

Local AI model execution

Docker Model Runner, available as part of the Docker Desktop 4.40 release, is being pitched as a solution to all of these challenges. It’s a new service that makes AI model execution “as simple as running a container,” the company said. The service is built on an inference engine within the Docker Desktop that’s accessible through the familiar OpenAI application programming interface. With this local inference engine, developers can test and iterate on their models in one place.

For developers using Apple Inc.’s Mac laptops, they’ll also be able to benefit from GPU acceleration, leveraging the built in graphics processing unit on their machine. The company explains that this host-based execution helps to get around the performance limitations of running models inside containers or virtual machines, meaning faster inference, smoother testing and better feedback loops.

With regards to model distribution, Docker Model Runner makes this easy too by packaging each model as an OCI Artifact. This is an open standard that allows models to be distributed through the same registries and workflows as traditional software containers, meaning they can be pushed and pulled from Docker Hub or an internal registry and integrated with existing continuous integration/continuous development pipelines. As a result, developers will be able to use familiar tools for automation and access control, the company said.

To make life even easier for local AI model developers, Docker is partnering with companies including Google LLC, Qualcomm Inc., HuggingFace Inc., Spring AI Inc., VMware Inc., and Dagger Inc. to provide users with access to an entire ecosystem of AI tools and services from within Docker Model Runner. So developers will be able to use the large language models, AI frameworks and development tools they’re already familiar with for local development.

Docker is best known for providing a suite of tools used by developers to build and manage software containers, which host the components of modern applications, on local machines. With Docker, they can create and test containers on a local machine before deploying them onto the cloud infrastructure of their choosing.

However, the company is no stranger to AI development. In 2023, it launched a service called GenAI Stack built in partnership with Neo4J Inc., LangChain Inc. and the open-source project Ollama, giving developers a preconfigured, secure and ready-to-code platform for developing AI models that consists of the LangChain development framework, a graph database, and a library of large language models to choose from.

The Docker Model Runner service builds on that platform, but it’s far from the finished article. Docker said it hopes to get more partners on board and expand the Docker Model Runner ecosystem, so developers will be able to run their models on more platforms. For instance, it wants to enable GPU acceleration on Windows laptops too, and expand its functionality to give developers more options in terms of model customization and distribution.

Image: SiliconANGLE/Meta AI