Twitter’s Nathan Marz: Architect for Human Fault-Tolerance
Nathan Marz is a lead engineer at Twitter. He came over to Twitter from Backtype, a real-time social analytics company that Twitter acquired last year. Marz started working on Storm, an open source real-time computation system, at Backtype and continues to work on it for Twitter.
Today at Strata Marz appeared on theCube and explained Storm and its design philosophy. Although he describes Storm as doing for real-time what Hadoop does for batch processing, he sees the two technologies as complimentary. Twitter uses both Hadoop and Storm internally. As Marz explains it, Hadoop let’s you look at all your data at once, and Storm lets you look at it as it comes in.
Marz explains that Storm provides a set of primitives for real-time processing that developers can combine together to build sophisticated, fault-tolerant applications. Marz says the biggest challenge for data processing, and it gets even harder for real-time data processing, is fault-tolerance – the need to keep data accurate even when systems fail.
Most of the time when developers think about fault-tolerance they’re thinking about hardware or network failures. Marz points out there’s another type of fault-tolerance engineers need to think about: human fault-tolerance. Engineers need to play for human error and buggy code. Marz says he planned for human faults when architecting Storm and recommends other developer do as well.
Since you’re here …
Show your support for our mission with our one-click subscription to our YouTube channel (below). The more subscribers we have, the more YouTube will suggest relevant enterprise and emerging technology content to you. Thanks!
Support our mission: >>>>>> SUBSCRIBE NOW >>>>>> to our YouTube channel.
… We’d also like to tell you about our mission and how you can help us fulfill it. SiliconANGLE Media Inc.’s business model is based on the intrinsic value of the content, not advertising. Unlike many online publications, we don’t have a paywall or run banner advertising, because we want to keep our journalism open, without influence or the need to chase traffic.The journalism, reporting and commentary on SiliconANGLE — along with live, unscripted video from our Silicon Valley studio and globe-trotting video teams at theCUBE — take a lot of hard work, time and money. Keeping the quality high requires the support of sponsors who are aligned with our vision of ad-free journalism content.