Without a doubt, the Big Data Revenue and Market Forecast from Wikibon’s Jeff Kelly is a must-read. There are so many little nuances within that plain fact is that the report is the definitive knowledge point of where Big Data is today and clues to what’s ahead. It’s the non-proprietary ground zero for Big Data. One of the things that caught my eye was the spread of verticals where Big Data has taken off drastically.
Many would assume that finance would be a leader in this space and they are correct, but other industries that are ahead of the curve include pharmaceutical and retail, as the market sets to see growth across the board. Note that these are highly sensitive security-aware environments, with numerous regulations to fulfill at numerous technology points. The story of security always seems to follow the same arc – first the emergence of a new technology hits the scene, there’s debate about whether it is the next big thing, we have some early adopters, many do things right, many more tag along, and then comes security which unfortunately most people really think of after the fact for a lot of reasons, but partly because security standards, products and best practices can be slow to evolve and react. The report mentions these deficiencies:
The Big Data market is still within the confines of the early adopter phase and is poised for significant growth. For the Big Data market to reach its full potential, enterprises and vendors must overcome a number of obstacles. …
…A lack of best practices and related technologies for managing Big Data as a corporate asset, including data quality, data governance and security platforms and tools;
Big Data is clearly seeing a massive boost in interest whether groups are exploring it, or making their plans to go full bore and gain all the tactical business advantages. There are many that are or should be wrestling with the notions of security, data governance and the management of information in those endeavors. The questions are many, and they include gaining a posture on privacy requirements, they include evaluating the responsibilities of the handling of big data, what is the relationship and impact of change on the IT infrastructure and service model, and most importantly they need to look at the responsibilities they consume.
On the one hand- yes, Big Data is data, so keep those general principles the same. It all gets a little tricky from there because at the same time, Big Data is different, and you have to consider the multiple sources, complexities, and even risks. This means some new thoughts and challenges – From a security point of view, there are many questions CIOs and other decision makers have on their mind as they consider Big Data initiatives and it covers a lot.
Information governance is one such broad topic and major topic. These practices are designed for the corporation’s risk, regulations, environmental, operational and legal requirements today and going forward. Whether it be policies, processes, controls, standard procedures or business structures, the road ahead for the intersection of Big Data and the information it handles retains many undefined characteristics. As far as Big Data itself, part of the design must include rules and standardization of the information it receives and produces; the information further needs to be reported on and validated. The question itself is complex, in part because of the complex, multi-point disparate nature of the data sources in Big data, but also because that information is integrated, and analyzed – Big Data needs some constructs around these elements from a corporate goals and governance perspective. No one has gone out of the way to define this.
I have little doubt the industry will rise to this occasion and adopt structure and formal guidelines. Until such a time, the biggest thing to focus on is a collaborative and in-depth review of your organization’s existing requirements, encompassing those questions surrounding privacy, data retention, compliance, information lifecycles, and so on. This raises the tremendous opportunity for professional services and need for practitioners in this field that understand the landscape in various environments and have experience dealing with Big Data initiatives.