It’s safe to say that Big Data is at a turning point. 

Many leading companies are now moving beyond the daily challenges of how to make data available or accessible but they now strive to use it in ways that create true business value, by making it actionable to all stakeholders. 

Traditionally, the typical enterprise has looked at Big Data as a race to collect the largest database of information on their business/customer to gain a competitive advantage. The general perception in this climate was that the more data you had, the more insights you could ascertain from it. This thesis was flawed for many reasons, first being that storage and processing costs of such a large source of data were astronomical in past years, furthermore correlation in data does not always determine causation, meaning if you don’t collect the right data you’ll still lack meaning no matter how much you have.

However, today it’s more affordable than ever to collect, store and analyze data. In our increasingly connected society, we’re dealing with more and more data, as all types of devices are being connected to the internet, new data platforms are emerging and we continue to collect traditional data from existing channels.

Consequently, because these vast deposits of data are so easily accessible, organizations are finding the most important aspect of Big Data is not so much about the quantity, but more the real-time actionable insights derived from the data. Think about it: You can have mountains of incoming data, but if no understanding can be siphoned, it’s pretty much useless. 

Enterprises today also are recognizing that it’s not simply about the data they are collecting internally, but about tangential data related to their customers – that can help them better understand their customers in context. This approach has become possible as storage and processing costs have dramatically reduced over the past years, especially thanks to services like Amazon and Rackspace, Inc.

 Don’t let the skill gap stop business

As organizations continue to exploit Big Data analytics as a competitive tool, and the “Internet of Things” continues to explode at an astonishing rate, another major challenge has been the steady stream of big data scientists in high demand across all industry sectors available to help.

According to a recent McKinsey & Co. report, there will be a 50 percent gap in the supply of data scientists versus demand in 2018. Furthermore, the report projects “a need for 1.5 million additional managers and analysts in the U.S. who can ask the right questions and consume the results of the analysis of Big Data effectively.”

This shortage of data scientists has led some people to believe that the era of Big Data could take longer to be realized for organizations that don’t move swiftly. 

In my opinion, talent acquisition and the data scientist shortage should not be any enterprise’s major concern. The focus, rather, should be on solution, which means approaching data automatically, whether structured or unstructured.
With this said, Big Data is not without its own share of problems that could continue to halt adoption. Some of the challenges include:

• Integration with existing tools. There are a few companies out there already working on this approach, however, there are not any real standards being developed for tight integration. Eventually how you query data should be seamless to the person querying it, whether it is in a Big Data solution, RDBMS, JSON/XML, etc.
• Better security models. There is almost no security in place for virtually all Big Data tools: Once you get access, you get access to everything. Improvements are constantly being made, but they still aren’t enterprise grade.
• Security to protect privacy. As with all forms of customer data collection, companies need to be careful of how they collect the information, what details are given to the consumer, how the data is used, and other legal and ethical issues. Organizations need to start focusing more heavily on securing data to protect the privacy of their customers.
• More mature software. To date, Hadoop logs are riddled with errors, warnings and numerous other issues that are almost impossible to decipher. Yet, the platform still seems to magically work. Moving forward, there needs to be improvements in better predicting, as well as avoiding problems when running map-reduce jobs and giving human readable information to solve problems.
• Engineer dependent. Most tools still require an engineering degree to operate on a daily basis. Hadoop’s Yarn has made a vast improvement opening up accessibility to tech savvy business analysts, but the market is still far from a true GUI-based tool.

Many technological advances are emerging to empower companies to navigate vast data collections and discover new business models. Software frameworks, like Hadoop, allow data to be easily stored, retrieved and queried at scale by distributing it across a number of different computers. A few years ago, such tasks would have taken skilled computer scientists months to concoct; today, getting up and running with a distributed data platform is quick and easy with Hadoop because the software handles most of the complexity

To succeed in today’s ultra-competitive marketplace, businesses must leverage these technology tools that continuously learn and reveal actionable and unforeseen connections. This automated trend has the potential to drive a radical transformation in how enterprises research, innovate, market and ultimately grow moving forward.

Closing the loop between data outputs and action

With database-level technology in place and complementary data analysis tools maturing, I see the pivotal next step as closing the loop between data outputs and action. This is not action by humans, but action triggered by technology. Machine learning, predictive analytics and statistical analysis are just a few emergent trends focused on making automation a reality.

State-of-the-art software solutions are becoming more readily available to help the novice data scientist analyze issues and ask the right questions to address some of these issues. Enterprises must start to embrace these automated software solutions – many of them can enable typical business analysts to perform queries and analysis in Big Data environments without the need to know MapR, Pig or other highly technical languages previously needed to access the data. 

The benefits from properly leveraging Big Data are endless. Industries from warehousing to agriculture are already realizing these advances, according to one recent Accenture study. For example, with robotic systems for order fulfillment enabling big reductions in shipment times, many warehouses can process more orders in less time and with fewer employees. Another case can be seen in the area of taking robots to the skies. Agricultural applications can use infrared cameras to pinpoint crops that may be receiving too much or too little water. And in emergency and medical sectors, first responders are using drones to detect survivors of accidents and natural disasters, as well as to deliver supplies to emergency zones.

In other industries, companies relying on delivery systems are now utilizing GPS truck data to track delivery routes, speed, performance and scheduling. According to a recent Big Data Startup article, delivery company UPS uses Big Data optimize routes and save massive amounts of time and money. In 2011, the company reported saving 8.4 million gallons of fuel by trimming 85 million miles off daily routes.

Airlines are noticing big advantages, too. Big Data can be used to track bags, personalize offers, boost customer loyalty and optimize operations to deliver better service. One airline reported working with a provider of intelligent operations services to harvest and analyze data generated by hundreds of sensors working inside its planes. The tools allow the airline to monitor its planes in real time, reduce fuel costs, manage plane maintenance and even spot problems before they happen.

Storage processing is the holy grail for leveraging data

In the age of Big Data, it’s critical to capture and store it all. Data that might seem completely irrelevant to your business now might be a gold mine in the future. Hadoop has made it possible for enterprises to capture, store and analyze lots more data in a cost-effective way. And with the advent of Yarn in Hadoop, more creative solutions are gradually starting to take place that leverage the Hadoop platform, especially because Yarn makes it easier for non-technical users to leverage Hadoop.

The ultimate holy grail of data will be a storage and processing platform that can leverage multiple storage formats and processing architectures (i.e., Recommind). I also think there is a trend toward leveraging Hadoop as a platform that can accomplish this task.

It’s no secret that we are in the midst of a seismic shift in the analytics world. Industries around the globe are waking up to the reality that data is an asset, and not a simple storage necessity. Organizations wanting a competitive advantage need to start investing in new comprehensive data analysis software solutions. They are proven to help non-data scientists — especially marketers, product people and others on the business side of a company — extract actionable answers from their databases.

Those companies that can build a reputation for providing valuable services while using consumers’ personal data in trustworthy ways will have big advantages over their competitors.

 .

 About the Author

 .

Steven Sarracino is the founder of Activant Capital. Over his 10+ year career, Steve has invested more than $2 billion of equity across a broad range of technology companies. Recently he architected the investment in and merger of iCongo and hybris, which resulted in the acquisition of hybris AG by SAP.  He led the firm’s investments in Sunquest (acquired by Roper Industries), Mitratech, Upland and WellAware.

photos by: add1sun, Photo Extremist, braddougherty and Knee Deep Photography via photopin cc