Editor’s Note: the following is an excerpt from an interview conducted with Kapow CPO Karl Ederle regarding a recent IDG study commissioned by Kapow Software to better understand the current state of Big Data and whether or not it’s solutions are meeting demand.  Ederle speaks on the Big Data gap, the impact of consumers on IT, and the changes that need to happen in the datacenter to support Big Data solutions now and in the future.

Big Data:  hype vs. solutions

Over 85 percent of both business and IT respondents agree that Big Data offers substantial value in its ability to foster a data-driven culture and make intelligent business decisions. Despite the perceived business benefits, more than half of respondents rate Big Data project success so far as lukewarm (somewhat successful), and only 23 percent of them perceive Big Data projects to be a success.

So why is the gap so big?  Among the key perceived barriers are concerns about having the right skills, time it takes to derive real value from Big Data and the ability to effectively access and leverage a variety of disparate data sources.  The wish list of what Big Data solutions should provide include data integration from multiple sources, automation of data collection tasks, shortening the time to put data to work and making relevant data more consumable for workers without relying on data scientists. Unfortunately over 50 percent of respondents say their Big Data solutions today are not effective in addressing these requirements.

While manual data aggregation is still widely common among over 80 percent of the respondents, 85 percent believe Big Data strategies should be user-centric so that Big Data insights can be easily consumable by business consumers in sales, marketing, customer care and finance to make more agile decisions. These solutions will remove barriers related to talent, time and resources and will make Big Data readily accessible, affordable and actionable across the organization.

Workers are taking matters into their own hands

Just like the BYOD revolution, business consumers today are far more technically savvy and require the same ease of use, availability and accessibility to data as they get in the consumer world. Our survey reveals that the inability to automate structured and unstructured data quickly and effectively is among the biggest challenges, with 60 percent of respondents noting that Big Data projects typically take at least 18 months to complete. More than half say these projects typically require consultants and other third-party experts to complete.

In an effort to get the data they need to support decision making in their role now and not 18 months, employees are taking matters into their own hands. They can’t wait for large Hadoop projects to get deployed or for consultants to build customized solutions – they need access to critical and fresh data that provides timely insights about customers, competitors, market trends, partners, government regulations and financial risks. Manual data aggregation from various internal and external sources remains a fact of life for 81 percent of the workforce. While 91 percent of the workforce asks IT to enable information aggregation, there’s a significant resource and talent gap to meet the demand.

This trend will follow the same trajectory of the consumer cloud and smartphone markets but in a slower pace due to the continuous growth in data volumes and availability of new data sources and types. This is also impacted by limitations and restrictions imposed by enterprise security procedures and regulations.

Consumerization of IT influencing Big Data products & services

Eighty-five percent of IT and business respondents believe IT consumerization trends will have a significant or moderate impact on Big Data strategy at their organizations. A majority of all respondents say they’re very or extremely likely to adopt Big Data solutions that are user-centric. These trends will bring more flexible, user-centric solutions to the workplace, and ultimately, consumerize the Big Data experience. Business consumers will look for solutions that offer accessible, easy-to-consume format so that they can act upon this mission-critical data quickly.

The Internet of Things trend will dramatically change the volume and variety of data that could be analyzed to assess businesses, personal consumption, and the environment.  If today we have billions of people producing billions of data points daily, then the Internet of Things would be leveraging many trillions of objects producing several trillions of data points every day.  Observations have documented that human beings are surrounded by thousands of trackable objects in their everyday lives.  Big Data products and services would need to evolve to be able to support even more data than is utilized today and would need to have far greater capabilities to process streaming data.  Equally important, with an even greater variety of data sources that will need to be correlated, data integration capabilities that can associate even more disparate data sources will need to also grow dramatically.

IT architecture must change to address today’s data demands

IT architecture will need to change to better manage the issues around data access, data security and data distribution to make big data easier to consume.  Specifically, IT architecture will need to evolve to better handle the instant propagation of updates and changes, especially with changes in data sources and new types of analysis. The consumption of big data goes beyond internal consumers. IT architecture will need to address Secure sharing of specific and up-to-date information with partners and the supply chain.  Data governance will also need to be updated to accommodate new types of data and analysis, particularly cloud data, such as SaaS transactional data or unstructured content from social networks.

Software-led solutions pave the way

The software-led trends in the data center will make big data more consumable through greater throughput efficiency and storage capacity, resulting in information being processed faster and more cost effectively. Processing costs are dropping dramatically through software initiatives that make use of extremely low cost hardware, which was the premise behind the development of Hadoop.  The next wave of software-led initiatives in Big Data is visualization.  If businesses can process all of the data they need and make that available to business consumers through intuitive visualization tools, the need for data scientists versed in sampling techniques diminishes, thus making data easier to consume.