Nathan Marz is a lead engineer at Twitter. He came over to Twitter from Backtype, a real-time social analytics company that Twitter acquired last year. Marz started working on Storm, an open source real-time computation system, at Backtype and continues to work on it for Twitter.

Today at Strata Marz appeared on theCube and explained Storm and its design philosophy. Although he describes Storm as doing for real-time what Hadoop does for batch processing, he sees the two technologies as complimentary. Twitter uses both Hadoop and Storm internally. As Marz explains it, Hadoop let’s you look at all your data at once, and Storm lets you look at it as it comes in.

Marz explains that Storm provides a set of primitives for real-time processing that developers can combine together to build sophisticated, fault-tolerant applications. Marz says the biggest challenge for data processing, and it gets even harder for real-time data processing, is fault-tolerance – the need to keep data accurate even when systems fail.

Most of the time when developers think about fault-tolerance they’re thinking about hardware or network failures. Marz points out there’s another type of fault-tolerance engineers need to think about: human fault-tolerance. Engineers need to play for human error and buggy code. Marz says he planned for human faults when architecting Storm and recommends other developer do as well.

You can find our previous coverage of Storm here and here.