What a $3 Million Prize Says About the State of Data Analytics Skills

IT pros with advanced data modeling and data analytics skills are in short supply, leading companies like Heritage Provider Network to take extraordinary measures.

The Northbridge, Calif.-based healthcare provider is offering $3 million to anyone who comes up with an algorithm that allows physicians to better identify at-risk patients, thus speeding up care and lowering the number of unnecessary hospitalizations. Kaggle, a sort of crowdsourcing platform for data analytics, is running the contest.

While admittedly a dramatic example, that an organization feels the need to offer a $3 million prize rather than rely on internal expertise to solve a data analytics conundrum says a lot about the number of qualified data scientists out there. There are only a handful of university-level programs training the next generation of data scientists, and current IT pros show a surprising lack of interest in acquiring data modeling and data analytics skills despite potential high salaries.

Which leads to the question: We hear a lot about the potential goldmine of insights waiting to be discovered inside Big Data. But with a dearth of data scientists on the job market, what if there aren’t enough (data)miners to sift through all that data and make the actual discoveries? The answer is one giant wasted opportunity.

To ensure that doesn’t happen, two need to happen.

First, schools and enterprises need to make data science a more appealing career choice. For schools, that means creating engaging, comprehensive programs that incorporate all the training data scientists need and better communicating the potential career paths for data scientists in the age of Big Data.

For enterprises, get out your checkbook. Data analytics pros are already well paid compared to non-analytics-focused IT pros. But the laws of supply-and-demand rule, meaning if enterprises truly cherish IT pros with advanced data analytics skills they’ll need to pay them like they do.

Second, while the ranks of data scientists are being filled, software vendors need to create user-friendly data analytics tools that non-data scientists and those without a PhD in statistics can operate. Just because an enterprise doesn’t have an in-house data scientist rock star, or the resources to offer a $3 million prize, doesn’t mean it must miss out on the benefits of Big Data and data analytics.

Luckily, the self-service movement, already well established when it comes to front-end business intelligence (BI) tools, is just now starting to take hold in the Big Data and data analytics spaces as well.

Revolution Analytics, for example, has created a front-end graphical user interface for writing algorithms and exploring large data sets using R, an open source, predictive analytics language. It includes point-and-click functionality to speed up code writing and a web-services framework to make integrating the code into applications easier.

Cloudera is another vendor looking to bring user-friendly tools to the world of Big Data. Its core platform, called Cloudera’s Distribution including Apache Hadoop v3, includes a new open database connectivity driver allowing easier integration with BI front-end tools and platforms.

I encourage enterprises to both ask their software vendors about user-friendly Big Data/data analytics tools and consider allocating the financial resources to hire highly skilled data analytics professionals, which should encourage more IT pros to enter the field.