UPDATED 13:15 EDT / MARCH 31 2014

Social network giants introduce WebScaleSQL, the scalable and customizable MySQL database

webscalesql-logoDatabases are used to manage information and to make programs available. These include meta-data such as user. They can also be searched by certain criteria and are useful in statistics and data analysis.

MySQL deployment and MySQL communities services continue to grow as the standard open source database for large companies. But to store the growing data of the massive number of user accounts and make it online at all times requires a massive database that is both scalable and customizable.

Companies like Facebook, Twitter, LinkedIn and Google have the task of operating and maintaining some of the biggest websites on the Internet, and due to not only the amount of content they host, but the amount of data generated by their users. These companies now have teamed up to create the perfect database designed to scale to massive proportions.

Google, Facebook, LinkedIn and Twitter launched the project WebScaleSQL, in which engineers of four companies will come together to solve the problems of information processing in massive databases. As the name implies, WebScaleSQL is a fork of MySQL (right now, that’s MySQL 5.6), optimized for large scale databases. WebScaleSQL will broaden existing activities in the MySQL community.

Highly scalable

So far, engineers have developed a collaboration system, a code review tool, and a bug tracker bug. An engineer may propose a change and one from another company will examine the code and provide feedback. If an agreement is reached, it will be pushed into the WebScaleSQL branch for everyone. Each company can then further customize WebScaleSQL for its own needs.

“Our goal in launching WebScaleSQL is to enable the scale-oriented members of the MySQL community to work more closely together in order to prioritize the aspects that are most important to us. We aim to create a more integrated system of knowledge-sharing to help companies leverage the great features already found in MySQL 5.6, while building and adding more features that are specific to deployments in large scale environments,” Facebook said in the company’s developers blog. “In the last few months, engineers from all four companies have contributed code and provided feedback to each other to develop a new, more unified, and more collaborative branch of MySQL.”

MySQL engineers have already produced a framework for automated tests, which once started will validate the whole process of MySQL testing validation; a series of stress tests to check system performance; and drive improvements in performance and features to facilitate the development of WebScaleSQL; and several changes to improve the performance of WebScaleSQL, including buffer pool flushing improvements; optimizations to certain types of queries; support for NUMA interleave policy; and more.

Deployments to unprecedented scale

MySQL is the most popular open source database management system in the world. Facebook along host one of the largest MySQL database deployments ever built. In choosing to build on MySQL, the WebScaleSQL community has opened up an avenue for hundreds of thousands of MySQL developers to grow their deployments to unprecedented scale.

The group is currently working on developing an asynchronous MySQL client that won’t have to wait to connect, send, or retrieve while querying; adding Facebook production-tested compression and versions of table, user, and compression statistics; and adding Facebook’s Logical Read-Ahead mechanism for “up to 10x” speed improvements in full table scans, such as nightly logical backups.

The engineers are also working on preparing to move Facebook’s production-tested versions of table, user, and compression statistics into WebScaleSQL, preparing to push the remaining components of Facebook’s current production-tested version of compression that were not already included in MySQL 5.6 into WebScaleSQL.

Open source bringing competitors together

The collaboration aims to keep all WebScaleSQL work open, to create a useful branch for others within the MySQL community who are focused on scale deployments.

The project is a great example of how open source can help competitors work together to solve problems. Facebook, LinkedIn, and Twitter already collaborate with many other companies on the Hadoop open source project, a means of analyzing large amounts of data.

The addition of Google is a huge win for this open source project. Google develops various programming language and open sources many of its project. The company has developed Go programming language in an attempt to combine the development speed of working in a dynamic language like Python with the performance and safety of a compiled language like C or C++. It has also the Dart language and the Android operating system to its kitty. Google’s experience in developing open source projects will certainly help WebScaleSQL to reach new heights in coming days.

The code for WebScaleSQL is available on Github. The zip file can be downloaded here. If others need help with the SSH keys see the Github reference page.


A message from John Furrier, co-founder of SiliconANGLE:

Your vote of support is important to us and it helps us keep the content FREE.

One click below supports our mission to provide free, deep, and relevant content.  

Join our community on YouTube

Join the community that includes more than 15,000 #CubeAlumni experts, including Amazon.com CEO Andy Jassy, Dell Technologies founder and CEO Michael Dell, Intel CEO Pat Gelsinger, and many more luminaries and experts.

“TheCUBE is an important partner to the industry. You guys really are a part of our events and we really appreciate you coming and I know people appreciate the content you create as well” – Andy Jassy

THANK YOU