UPDATED 17:49 EDT / MARCH 01 2012

NEWS

Twitter’s Nathan Marz: Architect for Human Fault-Tolerance

Nathan Marz is a lead engineer at Twitter. He came over to Twitter from Backtype, a real-time social analytics company that Twitter acquired last year. Marz started working on Storm, an open source real-time computation system, at Backtype and continues to work on it for Twitter.

Today at Strata Marz appeared on theCube and explained Storm and its design philosophy. Although he describes Storm as doing for real-time what Hadoop does for batch processing, he sees the two technologies as complimentary. Twitter uses both Hadoop and Storm internally. As Marz explains it, Hadoop let’s you look at all your data at once, and Storm lets you look at it as it comes in.

Marz explains that Storm provides a set of primitives for real-time processing that developers can combine together to build sophisticated, fault-tolerant applications. Marz says the biggest challenge for data processing, and it gets even harder for real-time data processing, is fault-tolerance – the need to keep data accurate even when systems fail.

Most of the time when developers think about fault-tolerance they’re thinking about hardware or network failures. Marz points out there’s another type of fault-tolerance engineers need to think about: human fault-tolerance. Engineers need to play for human error and buggy code. Marz says he planned for human faults when architecting Storm and recommends other developer do as well.

You can find our previous coverage of Storm here and here.


A message from John Furrier, co-founder of SiliconANGLE:

Your vote of support is important to us and it helps us keep the content FREE.

One click below supports our mission to provide free, deep, and relevant content.  

Join our community on YouTube

Join the community that includes more than 15,000 #CubeAlumni experts, including Amazon.com CEO Andy Jassy, Dell Technologies founder and CEO Michael Dell, Intel CEO Pat Gelsinger, and many more luminaries and experts.

“TheCUBE is an important partner to the industry. You guys really are a part of our events and we really appreciate you coming and I know people appreciate the content you create as well” – Andy Jassy

THANK YOU