As promised,Twitter open sourced its big data streaming solution Storm (part of its acquisition of BackType) at the Strange Loop conference in St. Louis this week. BackType has been promising Storm for some time now, but the Twitter acquisition made some worry that it would never see the light of day.

Storm, like Apache Hadoop, is a distributed computing system. It distributes data to be processed over a cluster of commodity hardware. As explained on Twitter’s engineering blog, Storm runs never ending “topologies” on data sets instead of “jobs.” Here are the use-cases the company outlines last month:

1. Stream processing: Storm can be used to process a stream of new data and update databases in realtime. Unlike the standard approach of doing stream processing with a network of queues and workers, Storm is fault-tolerant and scalable.
2. Continuous computation: Storm can do a continuous query and stream the results to clients in realtime. An example is streaming trending topics on Twitter into browsers. The browsers will have a realtime view on what the trending topics are as they happen.
3. Distributed RPC: Storm can be used to parallelize an intense query on the fly. The idea is that your Storm topology is a distributed function that waits for invocation messages. When it receives an invocation, it computes the query and sends back the results. Examples of Distributed RPC are parallelizing search queries or doing set operations on large numbers of large sets.

Services Angle

We’ve been watching the rise of Apache Hadoop and now HPCC from LexisNexis. Some might find a new entrant to the big data market redundant, but it’s important to remember that things are just getting started. Don’t be surprised if a spin-off company offering enterprise Storm services launches soon.