Netflix CEO Reed Hastings

Netflix Inc. is cementing its place in the open-source ecosystem with the release of yet another internal innovation promising to make the public cloud more viable for running production workloads. The release bumps up the number of projects on the video streaming giant’s GitHub page to a formidable 46.

Dubbed Dynomite, Netflix’s latest community contribution takes its name from the DynamoDB service on Amazon.com Inc.’s cloud platform, which hosts the company’s entire technology operations even as the retail giant promotes its own competing video-on-demand library. The project is similar to eBay Inc.’s newly released Kylin engine and many of the other technologies that make up the open-source ecosystem in that it was created to address a very specific internal mandate, namely the need to reduce the delay with which content is delivered to consumers.

The challenge Netflix faced was balancing that requirement with maintaining the availability of the single-server databases supporting its namesake service, part of a broader uptime problem that it has been struggling with for years. The difficulty lied in the fact that the streaming provider handles too much traffic to rely on conventional master-slave setups for mitigating downtime, while the alternative – sharding a dataset into smaller chunks distributed across different nodes – is immensely complex to execute at large-scale. That’s where Dynomite comes in.

The technology provides an abstraction layer that does away with the hassle of managing shards across an entire environment worth of single-server databases and makes it easier to scale implementations in the process. Dynomite accomplishes this by attaching a co-process to every instance that acts as as a proxy and a traffic router while also coordinating operations and exchanging information with its counterparts, which creates an interconnected cluster where none was possible before due to architectural limitations.

The resulting grid of databases provides a number of advantages over traditional setups. First, it’s highly reliable, with the traffic routing capabilities of the Dynomite allowing any node to receive a request even if it doesn’t possess the necessary information, thus providing redundancy. Moreover, Dynomite clients can failover to replicas in another rack or even a different data center thousands of miles away, which allows users to defend their organizations against both local failures and natural disasters affecting entire geographic areas.

The fact that the technology treats each database as just another node in the cluster has the added benefit of making it possible to add instances as needed without tinkering with the existing setup. Dynomite clients automatically work with a wide range of systems, including Memcached – which Facebook Inc. also employs in a similar clustering solution it open-sourced two months ago – Redis and, of course, DynamoDB. Netflix’s technology uses the native protocol of each system, which allows users to leverage existing tools to managed their environments. And rounding out the package is monitoring functionality for keeping track of key cluster metrics.

photo credit: chris.alcoran via photopin cc