Big Data’s been overhyped and is destined to become a buzz kill of a buzz word, largely because the promise of Big Data has left a huge gap between potential and actual application.  Setting out to understand this gap, Kapow Software recently commissioned IDG Research Services to identify the top Big Data pain points for the enterprise (infographic below). Some 200 executives and IT professionals participated in the research firm’s study, and the majority indicated that the amount of time and resources required to extract value from their information is the main drawback of the technology.

The poll discovered that over 85 percent of business and IT leaders believe that Big Data offers “substantial value” from a decision-making standpoint. Competitive advantage, improved customer satisfaction and increased employee productivity were some of the benefits that participants mentioned.

Despite this consensus about the value of Big Data, 52 of the respondents rated the success of their organizations’ Big Data projects as “lukewarm”. Only 23 said that their projects are successful, and 60 percent noted that Big Data initiatives take at least 18 months to complete. The inability to automate in a timely fashion was listed as one of the biggest barriers to success.

Taking matters into their own hands

One reason Big Data needs to become more consumable is because more and more end users are taking matters into their own hands.  According to Kapow, 81 percent of organizations struggle with manual data aggregation and 91 percent of the workforce want IT to address this issue more effectively. For this reason, 85 percent of the respondents think Big Data should be more consumable and user-centric.

“Just like the BYOD revolution, business consumers today are far more technically savvy and require the same ease of use, availability and accessibility to data as they get in the consumer world,” says Karl Ederle, CPO of Kapow Software.  “Our survey reveals that the inability to automate structured and unstructured data quickly and effectively is among the biggest challenges, with 60 percent of respondents noting that Big Data projects typically take at least 18 months to complete. More than half say these projects typically require consultants and other third-party experts to complete.

“In an effort to get the data they need to support decision making in their role now and not 18 months, employees are taking matters into their own hands,” Ederle continues. “They can’t wait for large Hadoop projects to get deployed or for consultants to build customized solutions – they need access to critical and fresh data that provides timely insights about customers, competitors, market trends, partners, government regulations and financial risks. Manual data aggregation from various internal and external sources remains a fact of life for 81 percent of the workforce. While 91 percent of the workforce asks IT to enable information aggregation, there’s a significant resource and talent gap to meet the demand.”

IT must rethink its infrastructure to make Big Data more consumable

.

It’s all well and good to note the gap between Big Data hype and the real world of Big Data applications, but what can be done about it?  One place to start is the data center, where software-led solutions and re-architecture are introducing new concepts, and solutions, to the enterprise.

“IT architecture will need to change to better manage the issues around data access, data security and data distribution to make big data easier to consume.  Specifically, IT architecture will need to evolve to better handle the instant propagation of updates and changes, especially with changes in data sources and new types of analysis,” says Ederle. “The consumption of big data goes beyond internal consumers. IT architecture will need to address Secure sharing of specific and up-to-date information with partners and the supply chain.  Data governance will also need to be updated to accommodate new types of data and analysis, particularly cloud data, such as SaaS transactional data or unstructured content from social networks.”