Microsoft’s Kate Crawford & Dark Data

Few trends in technology were hotter than Big Data over the last 12 months. In fact, so much has been said about it during 2013 that it’s hard to see what else we can say in 2014. But of course, there’s sure to be plenty of developments we don’t know about yet, as trends like the Internet of Things, mobile data and hybrid clouds continue to pick up pace, furthering Big Data’s adoption.

It’s still not easy to pin down the true potential of Big Data, but what’s easier to predict are the challenges businesses can expect to face over the year ahead.

Privacy and legal worries

The privacy challenges around Big Data are nothing new, but with ‘dark data’ now becoming a popular talking point, it’s obvious that the average consumer is becoming aware of these too.

The problem is compounded by the fact that most enterprises pursuing Big Data projects do so to benefit from the advanced analytics it can provide. According to a survey by Sand Hill, almost 62 percent of enterprises cited advanced analytics as their main reason for using Hadoop.

Legal concerns over Big Data privacy are likely to become a serious issue in 2014, as consumers become more aware of the impact of Big Data on their lives. What with the rise of the Internet of Things, more mobile devices and perhaps even drones buzzing over our heads, this is only going to generate even bigger data, sensory data and maybe even image data too, claims the IEEE Computer Society.

Brooke L. Daniels, of the Pilsbury law firm, advises that businesses ensure they’re ahead of the curve as far as compliance laws go, ascertaining who owns their data and who’s responsible if its inaccurate or falls into the wrong hands. Businesses should also keep themselves up to date with any changes to data protection laws.

Discrimination

One of the most touchy subjects that needs to be addressed is the way data can be used to discriminate. Last year, Microsoft’s Kate Crawford wrote that Big Data is leading to “more precises forms of discrimination,” with social media and health care data being the biggest worries.

Discrimination has been a problem for years of course, but the danger is that Big Data makes it more prevalent, a kind of ‘automated’ discrimination if you will. To defend against this, Crawford has proposed a system of due process that allows people to better understand how Big Data analytics is used by companies that make employment or health care judgments against them.

Businesses can sidestep issues of discrimination by creating transparent Big Data usage policies, whilst data governance committees can be set up to safeguard customer’s data.

Going against gut instincts

With most organizations pursuing Big Data to drive their decision-making, this could soon lead to conflict over the old way of doing things. Whilst analytics tells us to do one thing, our gut instincts often tell us to make different choices, because this is how most management employees are trained, notes Andrew McAfee, of the MIT Sloan School of Management.

Organizations may need to re-educate their employees, insists McAfee. Stubborn managers may not like it, but studies show that algorithms are far better at humans when it comes to making decisions, no matter if we’re trying to guess the outcomes of court cases or which supplier might perform better.