UPDATED 08:00 EDT / FEBRUARY 24 2015

Paradigm shift: How Apache Mesos is reshaping the data center into a giant PC

hp-servers-in-apple-data-centerMost people couldn’t get much done on their computers without an operating system. Working the hardware directly would require the specialized knowledge and near-infinite patience to manually feed instructions to each processor on the motherboard in its specific language for every single action.

It’s an impossible proposition. Yet the infrastructure professionals who keep the world’s data centers humming along manage a comparable feat on an incomparably larger scale as a matter of daily routine. They determine workload requirements, allocate the necessary resources and perform all the other activates by hand that are usually executed automatically on a personal computer.

It doesn’t have to be that way, or at least not according to the backers of an emerging open-source project called Apache Mesos that promises to bridge the chasm between the consumer and enterprise worlds. The framework offers to provide an operating system for the data center.

The concept is a radical departure from the current reality of manual management that traces its roots back to the mainframe era. Mesos ofers a fresh approach that takes a page directly from world’s largest data center operators.

The platform was originally developed at UC Berkeley’s famed AMPLab, the birthplace of pioneering open-source technologies like Apache Spark and GraphX, with direct input from Google. The search giant was working on a new system to manage its data centers at the time and took the opportunity to exchange key lessons with the creators of Mesos.

After its release, the open-source framework went on to find a home at other web giants such as Twitter, Inc. and Airbag, which funneled the lessons they gleaned from running large-scale production implementations back into the upstream initiative. The end result is a platform that not only provides the conveniences of an operating system but acts like one, too.

Mesos elevates distributed workloads to the same level at which traditional single-tenant applications execute on a single computer. The complex details of the underlying hardware are hidden away under a programmatic interface that exposes – or more accurately, offers – resources for consumption on an as-needed basis.

Services, which are deployed in containers to provide the scale-out equivalent of process isolation, determine how much capacity to accept through a built-in scheduler that can be optimized for the specific requirements of the workload. That two-tier approach makes Mesos more flexible than other clustering technologies that make the choice independently of the application and puts it an entire world apart from how most organizations provision their infrastructure today.

Business processes are typically deployed in virtual machines that are complicated to manage and immensely difficult to re size, which leads to a situation in which there are almost always too few or too many resources allocated to an instance. Organizations didn’t have much choice but to accept that situation until now.

While Mesos addresses that trade-off, the operating system analogy doesn’t fully hold for a data center with thousands of computers. A container that ends up on a server with less bandwidth than the other nodes in a cluster, for instance, will inevitably create a bottleneck regardless of how many layers of separation there are between the hardware and the administrator. Some problems will always require moving down the stack, just as not every workload can be effectively accommodated in a fully abstracted environment.

Mesosphere, the startup that members of the original AMPLab team created to commercialize the project, has been working to fill the gaps with its commercial version of the platform. Unveiled in December, the distribution combines the framework with a number of other components from the open-source ecosystem to address key needs such as the ability to effectively scale services. But the question whether the enterprise is truly ready for a data center operating system still looms large.


A message from John Furrier, co-founder of SiliconANGLE:

Your vote of support is important to us and it helps us keep the content FREE.

One click below supports our mission to provide free, deep, and relevant content.  

Join our community on YouTube

Join the community that includes more than 15,000 #CubeAlumni experts, including Amazon.com CEO Andy Jassy, Dell Technologies founder and CEO Michael Dell, Intel CEO Pat Gelsinger, and many more luminaries and experts.

“TheCUBE is an important partner to the industry. You guys really are a part of our events and we really appreciate you coming and I know people appreciate the content you create as well” – Andy Jassy

THANK YOU