UPDATED 17:49 EDT / MARCH 01 2012

NEWS

Twitter’s Nathan Marz: Architect for Human Fault-Tolerance

Nathan Marz is a lead engineer at Twitter. He came over to Twitter from Backtype, a real-time social analytics company that Twitter acquired last year. Marz started working on Storm, an open source real-time computation system, at Backtype and continues to work on it for Twitter.

Today at Strata Marz appeared on theCube and explained Storm and its design philosophy. Although he describes Storm as doing for real-time what Hadoop does for batch processing, he sees the two technologies as complimentary. Twitter uses both Hadoop and Storm internally. As Marz explains it, Hadoop let’s you look at all your data at once, and Storm lets you look at it as it comes in.

Marz explains that Storm provides a set of primitives for real-time processing that developers can combine together to build sophisticated, fault-tolerant applications. Marz says the biggest challenge for data processing, and it gets even harder for real-time data processing, is fault-tolerance – the need to keep data accurate even when systems fail.

Most of the time when developers think about fault-tolerance they’re thinking about hardware or network failures. Marz points out there’s another type of fault-tolerance engineers need to think about: human fault-tolerance. Engineers need to play for human error and buggy code. Marz says he planned for human faults when architecting Storm and recommends other developer do as well.

You can find our previous coverage of Storm here and here.


A message from John Furrier, co-founder of SiliconANGLE:

Support our open free content by sharing and engaging with our content and community.

Join theCUBE Alumni Trust Network

Where Technology Leaders Connect, Share Intelligence & Create Opportunities

11.4k+  
CUBE Alumni Network
C-level and Technical
Domain Experts
15M+ 
theCUBE
Viewers
Connect with 11,413+ industry leaders from our network of tech and business leaders forming a unique trusted network effect.

SiliconANGLE Media is a recognized leader in digital media innovation serving innovative audiences and brands, bringing together cutting-edge technology, influential content, strategic insights and real-time audience engagement. As the parent company of SiliconANGLE, theCUBE Network, theCUBE Research, CUBE365, theCUBE AI and theCUBE SuperStudios — such as those established in Silicon Valley and the New York Stock Exchange (NYSE) — SiliconANGLE Media operates at the intersection of media, technology, and AI. .

Founded by tech visionaries John Furrier and Dave Vellante, SiliconANGLE Media has built a powerful ecosystem of industry-leading digital media brands, with a reach of 15+ million elite tech professionals. The company’s new, proprietary theCUBE AI Video cloud is breaking ground in audience interaction, leveraging theCUBEai.com neural network to help technology companies make data-driven decisions and stay at the forefront of industry conversations.