Enterprise is sometimes pretty quick on the uptake, as illustrated by the fact that the vast majority of organizations have given Big Data projects a very high priority in recent months. Businesses and their CIOs understand that gaining insights from Big Data will be crucial to their performance over the next few years, and that if they fail to do so they’ll almost certainly be left in the dust by their rivals.

But despite this realization, a huge proportion of Big Data projects remain incomplete or worse, fail altogether, highlighting the fact that while CIOs may understand its importance, they have a very limited grasp of the challenges involved.

A recent study by the Big Data analytics firm Infochimps interviewed more than 300 IT professionals, finding that some 58% of all Big Data projects undertaken by big business today remain incomplete. Even worse, a worryingly large number of projects that are completed fail to achieve their objectives – this in spite of the fact that more than four out of five companies have identified Big Data/advanced analytics as one of their highest IT priorities for 2013.

So who’s to blame for this Big Data failure? This might be a hard bullet to bite, but much of the blame seems to lie with CIOs and others sitting pretty in the upper echelons of management:

“It isn’t surprising to find that 55% of Big Data projects aren’t completed and many more fail to achieve their objectives — often those charged with implementation are the last consulted,” says Kaskade, alluding to a lack of communication between IT teams and CIOs.

“We created this report as a resource to give CIOs insight into the too-often overlooked views of those charged with the heavy lifting.”

The Biggest Challenges

.

Infochimps’ report lays out the most common obstacles that IT teams face with when dealing with Big Data, noting that CIOs rarely consider these from their IT team’s perspective. The biggest challenge highlighted in the report, cited by 76% of all respondents, is what IT professionals refer to as a ‘culture of siloed data’, which leads to difficulty in accessing the data stored in various applications across an organization. This in turn leads to a problem of ‘inaccurate scope’, which was cited by 58% of respondents as a reason for Big Data failures. Kaskade expands on this:

“Companies need to start with the business problem first to properly scope their projects. Too many organizations are building Big Data platforms intended to meet the entire organization’s needs. Unless they understand specific use-cases first, many will find such an approach falls short.”

Other big obstacles that need to be overcome include staff education and the ability to understand platforms, a lack of data context, and the resulting inability to connect the dots (something that suggests a lack of data analysis talent), and the processing, analyzing and ongoing management of data.

Most Critical Requirements

Surprisingly, IT professionals also claimed that CIOs failed to understand the most critical requirements for their Big Data projects, which include creating the ability to scale, ease of management, building flexible architecture, security and speed to deployment. Moreover, IT pros share the belief that these days Hadoop alone cannot solve all of their problems, with the ability to carry out batch analysis and real-time or near real-time analysis being just as important in gaining insights from Big Data and making decisions going on what they’ve learned.

What CIOs Need To Know, Above All Else

.

More than anything else, CIOs fail to grasp that when they go for Big Data they’re in it for the long haul. The one thing that respondents stressed over and over was that Big Data projects take time and commitment, and it’s precisely here where CIOs need to make their biggest investments.

For a full copy of the report, visit Infochimps website.