UPDATED 07:00 EDT / AUGUST 28 2020

CLOUD

Beyond the fail whale: How former Twitter engineers designed a serverless database at Fauna

In the time it takes to read this sentence, more than 60,000 tweets will have already been sent. Twitter Inc. today has 330 million monthly active users, 1.3 billion accounts and 83% of the world’s leaders use the service.

Yet, as the company’s 15th employee hired in 2008 saw it, one of the most powerful communications platforms in the world struggled mightily to implement a modernized database platform. “When we started at Twitter, databases were bad,” Evan Weaver (pictured, left), said during an interview in 2018. “When we left, they were still bad.”

Weaver’s frustration with database technology was that first-generation NoSQL systems failed to evolve into a broadly capable platform as Twitter began to scale. Weaver and his team of software engineers had to create special-purpose, highly optimized datastores to power core workloads for Twitter.

Meanwhile, Twitter’s developer community was using APIs very aggressively, including as a general-purpose data platform. This led to a light bulb moment for Weaver, who left Twitter in 2011 and is now the co-founder and chief technical officer of Fauna Inc.

“We just never found a platform which was suitable for what we were trying to accomplish,” Weaver said. “A lot of what Twitter did was itself a platform, we had developers all over the world using the Twitter API to interact with tweets. We were frustrated that we had to become specialists in data systems because there wasn’t a data API we could just build the product on. Ultimately, that data API that we wished we had is now Fauna.”

Weaver spoke with Stu Miniman, host of theCUBE, SiliconANGLE Media’s livestreaming studio, in the latest episode of Cloud Native Insights, a series that focuses on transitions in the marketplace and how companies are making the journey to modernize and leverage cloud native technologies. He was joined by Eric Berg (pictured, right) chief executive officer at Fauna, and they discussed Fauna’s evolving role as a data layer for client serverless applications, an advanced technical protocol that powers the firm’s solution, making database interaction for developers easier, the startup’s growing market opportunity and the future of serverless technology.

Filling the database gaps

The problem Fauna set out to resolve stems from what got left behind in the transition from relational to NoSQL databases. Flexible schemas and horizontal scalability were significantly improved, but security, data consistency and relational querying were not. Database architectures needed to be retooled for the cloud.

In today’s reality, browser and mobile application clients are built on a globally connected ecosystem of serverless APIs and Fauna positions itself as a cloud native database, the new data layer for client serverless applications.

“The maturation of serverless and therefore using that as an underpinning for a new type of application is really just starting to take hold,” Berg said. “That creates a variety of different opportunities for people in that stack to build interesting businesses. Obviously, the database is an incredibly important part of that.”

FaunaDB is based on Calvin, a protocol for fast distributed transactions that was developed by Professor Daniel Abadi and a team of researchers from Yale University. Calvin relies on a deterministic ordering guarantee to reduce issues associated with the properties of database transactions.

These properties, known in the computer science world as atomicity, consistency, isolation and durability or ACID, are intended to guarantee data validity despite mishaps like errors or power failures.

“Databases are always the last to evolve because they’re the riskiest to change and the hardest to build,” Weaver explained. “Part of the reason the operational database, the database for mission critical business data, has remained a cost center is because the conventional wisdom was that something like Fauna was impossible to build. People said: ‘You literally cannot, in information science, create a global API for data which is transactional, consistent and suitable for mission critical banking, payments, user generated content, social graphs, internal IT data, anything that’s irreplaceable.’”

Addressing complexity

At a time when container native data storage is emerging as a potential growth market, Fauna is stepping forward with a value proposition that leapfrogs even that. The firm’s pitch is that by allowing developers to interact with databases via API, queries can be written more easily without having to worry about manual management of the underlying infrastructure.

“Databases in general traditionally have come with a lot of complexity,” Berg said. “Everybody would love to have a database that scales reliably and securely that they don’t have to manage. When you look at the technology and some of the problems that Fauna has solved in removing all of that operational burden when scaling a database, not only locally but globally, it’s a no-brainer.”

However, Fauna’s decision to base its ambitious cloud native solution on serverless technology presents challenges of its own. Although Weaver acknowledged that Google Ventures has invested in Fauna, he also noted that proprietary systems such as Firebase, Google’s mobile platform, could hinder the growth of serverless as an important enterprise tool.

The key will be collaboration among major players to help advance the serverless cause, according to Weaver, much as the web development platform of Linux, Apache, MySQL and PHP or LAMP benefited from enterprise support and adoption decades ago.

“Serverless is in the same place that LAMP was in the early ’90s,” Weaver noted. “’It’s not mature, it’s a toy, no one will ever use this for a real business.’ The movement and momentum in serverless is real and the challenge now is for all the vendors, in collaboration with the community of developers, to mature the tools as the products and applications being built on the new more productive stack also mature.”

Growing developer adoption

In July, Fauna secured $27 million in new funding and recruited Berg from Okta Inc. as its CEO, replacing Weaver, who moved to CTO. The company also tapped Bob Muglia, former CEO of Snowflake Inc., to be chairman of the Fauna board. Snowflake, which filed its S-1 disclosure for an initial public offering on Monday, parlayed its own database solution into a $12 billion valuation.

However, Berg is careful to note that data warehouse provider Snowflake is not a competitor.

“Fauna is very much on the operational database side,” Berg said. “We’re really focused on being the core operational store for your application.”

Fauna currently has “tens of thousands of customers,” according to Berg. Based on the current DB-Engines ranking, the startup has moved up over 90 spots in developer popularity compared with a year ago, an indication that its serverless solution is beginning to gain traction.

In the early days of Twitter, a service outage was generally noted with an image of the “fail whale,” a serene mammal being lifted out of deep waters by a flock of orange birds. The company formally retired the whale in 2013 as service improved, for which Weaver has been publicly given credit.

Now, Weaver is seeking to transform the enterprise database world by leveraging cloud-native and serverless technologies in a unique way.

“It’s sort of like we have no power company, so you call up Amazon and they drive a generator to your house and they hook you up,” Weaver said. “Right on, I didn’t have to install the generator myself, but it’s not a good experience. It’s still specific to the location you are at, it’s not getting utility computing from the cloud in a way that’s been a dream for many decades. It’s finally real with serverless.”

Here’s the complete video interview, the latest in the continuing Cloud Native Insights series and one of many CUBE Conversations from SiliconANGLE and theCUBE:

Photo: SiliconANGLE

A message from John Furrier, co-founder of SiliconANGLE:

Your vote of support is important to us and it helps us keep the content FREE.

One click below supports our mission to provide free, deep, and relevant content.  

Join our community on YouTube

Join the community that includes more than 15,000 #CubeAlumni experts, including Amazon.com CEO Andy Jassy, Dell Technologies founder and CEO Michael Dell, Intel CEO Pat Gelsinger, and many more luminaries and experts.

“TheCUBE is an important partner to the industry. You guys really are a part of our events and we really appreciate you coming and I know people appreciate the content you create as well” – Andy Jassy

THANK YOU