Container-centric startup CoreOS Inc. has taken the wraps off of a new open-source project called Torus. Just like the company’s other projects, Torus is about bringing the same Web-scale technologies used by giants like Google and Facebook to a much wider audience, and this time around the focus is on distributed storage.

CoreOS said it’s trying to deliver a reliable and scalable storage system for container-based apps that rely on Google’s Kubernetes container management tool. It called out persistent storage in container clusters as one of the “most interesting” problems in computing today, in a blog post written by software engineer Barak Michener.

“Where do we store the voluminous stream of data that microservices produce and consume, especially when immutable, discretely contained executable code is such a powerful pattern?” Michener asked.

Torus isn’t the first open-source distributed storage system. Its most famous rival is probably the Ceph Project, which is popular with OpenStack deployments. But Ceph and other storage solutions were designed for small clusters of large machines, which means containers were never really a focus, CoreOS argues. It says that what most developers want is a persistent storage system that can feed applications running in containers.

“Ensuring persistent storage for these container microservices as they are started, stopped, upgraded, and migrated between nodes in the cluster is not as simple as providing a backing store for a single server running a group of monolithic applications, or even a number of virtual machines,” Michener writes.

Pool storage

In his post, Michener explains how Torus is able to take a pool of storage resources and create multiple interfaces, including network-accessible block-level storage as a default option. He writes that Torus is deployed by Kubernetes alongside the apps it’s going to provide storage for, using the Flex volume plugin to dynamically mount storage volumes for nodes in the cluster. Doing so, it’s possible to have something like PostgreSQL run on top of Torus’ storage volumes.

Torus illustrates how CoreOS is working on what happens outside of containers as well as inside them. For example, Torus uses the etcd distributed key/value store to store and replicate metadata for all the files and objects stored in the pool.

The challenge of persistent storage

It’s not easy to provide persistent storage for containerized apps. The main problem with conventional attached storage is that it’s impossible to guarantee data locality or ensure that the data for a specific app can be stored near the containers that require it. Other vendors have tried to solve the problem using different methods. For example, Cisco Systems Inc. is trying to solve data locality issues at the hardware level, while MapR Technologies Inc. provides an option to use its Hadoop-powered Converged Data Platform as a persistent data store.

But Torus is a 100 percent software based solution that runs on software that’s most likely already in use. That’s not to say CoreOS suggests Torus is a full solution, but rather, it says it views Torus as the building blocks of a more robust systems that’s yet to be built. For example, Torus doesn’t yet provide object storage, though the company believes the feature can be added.

What with containers becoming more mature, vendors in the ecosystem are now turning their attention to the problems that arise as large, distributed apps are built around them. By pairing Torus with Kubernetes, CoreOS is providing a solution that reduces the number of new technologies needed to make it happen, doing justice to the theory that containers are meant to keep things simple.

Image credit: Skeeze via pixabay