The irony of Big Data at SXSW is the lack of overlap.  The common denominator amongst Big Data theories is the adaptability of Big Data itself.  SXSW has become a catch-all for geeks.  The result is an eclectic mix of startups, press, developers, musicians, advertisers, filmakers and gamers.  Big Data has its own story in any of these industries.  Why, then, is Big Data a relatively absent topic at our treasured SXSW?

Perhaps it’s because Big Data has already become an ubiquitous topic for the SXSW crowd.  Big Data is applicable to every community represented at SXSW, and the conference has become so large that it can support a Big Data hamlet all its own.  So with an event as beautifully diverse as SXSW, it’s not that Big Data is absent, but rather interwoven into most everything SXSW stands for.

That’s not to say Big Data isn’t a growing topic at SXSW.  Certainly, it’s gained traction since last year, when very few people understood the “Big Data” reference on my t-shirt.  The number of Big Data panels seems to have increased in recent years, though there’s been no shortage of submissions on the topic.  The types of panels accepted at SXSW also reflects the integrated nature of Big Data at this annual festival — one of the more interesting from this year was a discussion of Big Data and gamification.

Big Data panel @ SXSW: Build vs. Buy

.

I was invited to moderate a panel on Big Data this year, encompassing Big Data and the stack.  On stage with Infochimp’s Director of Product, Tim Gasper, MongoDB’s Senior Architect Sandeep Parikh, and Identified.com CEO Brendan Wallace, our panel addressed the question of building one’s own Big Data solution versus buying one, a topic I steered towards the startup audience for SXSW purposes.

Though this seems to be an enterprise-centric panel, the topic is befitting for SXSW given the transformative effect of Big Data solutions.  Impacting the entire stack, Big Data is spurring innovation in the data center from networking to storage architecture, and comes at a time when smarter software is redefining the requirements of time management and mobility.  Not only could my panel members offer recommendations to startups and developers looking to implement their own Big Data solution, but were able to speak on the democratizing benefits of Big Data for their own business strategies in an emerging market.

Some of the key topics we discussed included data integrity being “coded” into the algorithms underlying a Big Data solution, the challenges solution providers face in educating and servicing clients, and why expertise is so important for executing a successful Big Data deployment.  By the end of the discussion, it was clear that while companies are anxious to move past the Big Data exploration stage, there remains a great deal of complexity in this market.

Big Data will ooze to new departments, not industries

.

Serving the enterprise, all the startups represented on the panel have an insider track on industry adoption for Big Data solutions.  Citing a recent forecast report from analyst firm Wikibon, I mentioned the finance, retail and pharmaceutical industries as the most interested in Big Data for 2012.  Did the panel see interest from the same industries, and which industries would be most influential in 2013?

The panel generally found interest from the same industries, with an emphasis on finance and retail.  Identified even has a healthcare-centric product to appease the enterprise, culminating data for actionable insights by recruiters.

But rather than seeing a fresh batch of industries making a new mark on Big Data, Gasper sees the influence seeping horizontally within the existing markets of interest.  As Big Data solutions find their way beyond the IT department, actionable data is reaching more end users within an organization.  The marketing department is gaining new use cases for data insight, as is the HR department.  Steady improvements in user interface standardization, mobility and visualization will push along the type of end user access that can truly revolutionize the enterprise on every level.

Big Data still complex, just like SXSW party plans

.

Big Data sounds like a fun time, but it can become a daunting reality once you consider the challenges in scaling a solution that must remain adaptable, and must be designed with enough inherent intelligence to do so.  This adaptability is a key perk for Big Data, but also makes it increasingly difficult to pigeonhole for an event so encompassing as SXSW.

Rick Smolan, author of The Human Face of Big Data, anticipates Big Data becoming as ubiquitous as the Internet, eventually growing so large and penetrating so deep into our culture that its terminology will no longer describe the technology but the experience of Big Data.  The Internet isn’t a place, but a conduit.  And Big Data isn’t a thing, but a mechanism for cultivating and employing our knowledge.  And at SXSW, Big Data means a great many things, finding a home in every attendee, startup and campaign.

I discuss the SXSW panel in further detail in the below video, a taping of this morning’s live NewsDesk segment with Kristin Feledy.

Flickr Creative Commons: photo by StandUPP