Big data is flooding into companies faster than they can figure out what to do with it. Hungry as they are to monetize it, many are failing. Will new technologies like artificial intelligence and machine learning solve the problem with do-it-yourself data science for regular folks? Or, will universities have to turn out a whole new class of experts to handle complex data operations?

“Our organizations are increasingly awash in data — which is the lifeblood of our organizations — but we’re not using it,” said Dion Hinchcliffe (pictured, left), vice president and principal analyst at Constellation Research Inc. and ZDNet industry commentator.

Companies vary in how well they can capture, analyze and profit from data. Some have been quite successful — but most still struggle to blend the right technologies and skills together. Hinchcliffe, along with four others, debated the right recipe at a special panel during the IBM Data Science for All event in New York City. They spoke with Dave Vellante (@dvellante) and John Walls (@JohnWalls21), co-hosts of theCUBE, SiliconANGLE Media’s mobile livestreaming studio. (* Disclosure below.)

Joining Hinchcliffe in the discussion were Jennifer Shin (pictured, right), founder of 8 Path Solutions LLC and director of data science at Comcast Corp.; Joe Caserta (pictured, second from left), president of Caserta Concepts LLC; Bob Hayes, Ph.D. (pictured, second from right), president of Business Over Broadway; and Joe McKendrick (pictured, center), analyst and contributor at Forbes.com.

How much data is slipping through the fingers of information technology and business executives unanalyzed? About 90 percent of data that exists today is unstructured data, according to International Data Corp. Unstructured data is usually text-centric but may be numerical, and does not fit into a predefined data model. This is overwhelmingly dark data — meaning it is not analyzed by businesses, and any potential value it contains goes unrealized. The sheer volume of this data is one reason why human labor will never be sufficient to parse through it all, Hinchcliffe explained.

“There’s far more demand for data science than there ever could be produced by having an ivory tower filled with data scientists,” Hinchcliff said. Highly skilled data specialists are no doubt needed more than ever, but they can’t do it all. Data science must be productized with easy buttons added, “whether that’s machine learning or artificial intelligence or bots that go off and run the hypotheses and select the algorithms — maybe with some human help,” he added.

Data diploma

Extra human help may arrive soon, thanks to colleges stepping in with data science tracks. The University of California, Berkeley, has announced it will offer an undergraduate data science major. Fittingly, given that data science as we know it today is still gelling, the major is somewhat flexible depending on students’ interests.

Targeted programs such as this make sense since a data scientist’s work is so deep and challenging. A data scientist must understand data technically and conceptually and be able to model with it, according to Shin.

Still, there remain questions as to whether the return on investment will be worth it for the students of such programs and the companies who hire them. “Whether or not it’s an industry job in the way that we see it today, in like, five years or 10 years from now — I think that’s something that’s up for debate,” Shin said.

The more data becomes a science with rules that can be programmed into machines, the less necessary people with data science degrees may become. This is the benefit realized from the slide from data warehousing — which relied on people to manually gather insights — to data science as we know it now, according to Caserta.

“That’s why we need to build a science around it so that we can actually have machines actually doing the analytics for us,” he said.

Citizen analyst: data science fiction?

Within the next year, we will see even more innovation in data science technology tools, Caserta predicted. These tools will likely allow the average business person — and certainly the average IT person — to predict next year’s sales as easily as they can look up last year’s sales, he explained.

Enter the citizen data scientist. While many vendors are playing on this idea to sell software, some are not convinced that it’s feasible. “Data science is not easy,” Hayes said.

Hayes encourages end users of data analytics software to at least take a statistics course so they understand means, variability, regression analysis and the like. A grasp of basic analytics concepts is necessary to make use of the insights from software programs, he stated. “If you go to France and don’t know French, then people can speak really slowly to you in French … you’re not going to get it,” Hayes said. “You need to understand the language of data to get value from the technology that we have available to us.”

Is all the trouble of analyzing data worthwhile? Is data really telling companies anything ordinary business analysts aren’t? “I’m already seeing trends in all different industries, where conventional wisdom is starting to be trumped by analytics,” Caserta said.

“Predictive analytics’ aim isn’t only to assess human hunches by testing relationships that seem to make sense, but also to explore a boundless playing field of possible truths beyond the realms of intuition,” according to Eric Siegel, author of Predictive Analytics: The Power to Predict Who Will Click, Buy, Lie, or Die. “As strange, mystifying or unexpected as they may seem, these discoveries help predict,” he wrote in a Bigthink.com blog post. Seigel’s view of predictive analytics borrows from Freakonomics, the pop science of hidden connections. 

For example, Uber Technologies Inc. has discovered through data that high crime correlates with increased rides. The hypothesis is that high-crime areas have a greater non-residential population. Sometimes the correlation is less obvious, such as when human resources company Evolv Inc. found that the internet browser a candidate uses predicts job performance.

Predictions for predictive analytics’ future

Data science says the darnedest things; this has benefits not just for business, but also for keeping those in authority honest, according to Caserta. This fact is often missing in debates about artificial intelligence and human oversight.

“I personally would trust a machine that was programmed to do the right thing than to trust a politician or some leader that may have their own agenda,” Caserta said. An auditing system of checks and balances on data will appear in our lifetime, similar to the Generally Accepted Accounting Principles, or GAAP, he predicted.

Increased adoption of big data analytics at large corporations will expedite innovation and discovery that will trickle down, Shin pointed out. “They have funding that startups don’t have,” she said.

Even if using data becomes “iPhone easy” — Caserta’s prescription — it will not replace the original business intelligence inside people’s heads, according to Joe McKendrick (pictured, center), analyst and contributor at Forbes.com.

“Ultimately, data, IT, technology alone will not create new markets; it will not drive new businesses. It’s up to human beings to do that,” McKendrick said. “Technology is the tool to help them make those decisions, but creating businesses, growing businesses is very much a human activity.”

Watch the complete video interview below, and be sure to check out more of SiliconANGLE’s and theCUBE’s coverage of the IBM Data Science for All event. (* Disclosure: TheCUBE is a paid media partner for the IBM Data Science for All event. Neither IBM, the event sponsor nor other sponsors have editorial control over content on theCUBE or SiliconANGLE.)

Photo: SiliconANGLE