Today “operational intelligence” vendor Splunk officially announced its IPO. In a video interview with Dell’s Barton George, the company’s CTO and co-founder Erik Swan explains the history of Splunk and how it went from being a complicated dashboard product to being the “Google of machine data.”

According to Swan, Splunk is the third company that he and co-founder Rob Das have worked on together. In 2002 Swan they sold CommerceFlow to Accenture and the two just needed something to do. To their wives’ chagrin, they spent a couple years not making money while they playing golf and tried to come up with an idea for a new company.

First they decided to solve a genomics problem: curing cancer. “It turned out about a week into it that we weren’t going to cure cancer,” Swan says. “I still have my bookmarks of genomics research in my browser and I never made it through the first paper.” So they decided to try to solve a problem that they understood instead.

Swan says that at CommerceFlow they had built a big transactional engine, but it was hard to troubleshoot these systems when something went wrong. They spent nine months talking to people who had similar problems, and found that many different problems all came down to the need to search through logs.

The Splunk team originally built a “really complicated” transactional engine dashboard, but when they showed it to potential customers everyone was most excited about the search feature. Swan says customers would say “Hey guys, that Google thing for IT machine data, that’s hot!” and then the Spunk team would say “No no no, we did all the work on the other thing, that thing is the hotness.” But eventually customers convinced the team that this simple concept of a search engine for machine data was a really powerful idea.  This feedback was a turning point in the history of Splunk. The company pivoted and created what the team now describes as “Google for machine data.”

Today Splunk consumes any sort of machine data – from logs to Twitter feeds to network traffic. Users can do sentiment analysis on Twitter, analyze logs and more all without the need for DBAs or Hadoop experts. However, a lot of Splunk’s customers do also use Hadoop. Swan says the product is different from Hadoop in that while Hadoop does batch processing. Many customers use both Hadoop and Splunk. They use Splunk to collect data for Hadoop to process, and thanks to Splunk’s Hadoop connector can add information from Hadoop into Splunk dashboards. For a more in depth look at what you can do with Splunk, take a look at Derek Harris’ profile of the company from 2010.

Since its launch Splunk has kicked off a mini-revolution in how companies handle machine data. In 2009 former Splunk employees launched Loggly to provide a log searching tool in the cloud. More recently Boundary launched a real-time monitoring software-as-a-service powered by machine data, and SumoLogic emerged from stealth to provide its own machine data in the cloud service. I have the feeling things are only just getting started.