There’s a new position that’s emerged from the booming big data sector–that of the data scientist.  It’s a job that’s as fluid and pliable as the term big data itself, lingering between two worlds of technology information and information implementation, yet to become fully solidified as a career or education path.  As we continue to explore the role of the data scientist, we must also determine what this new breed of experts needs.  A recent study from Revolution Analytics indicates that big data technology solutions still have plenty of room to grow.

Polling attendees at the Joint Statistical Meeting earlier this summer, the study revealed that nearly 97% of data scientists believe big data solutions need improvement.  There’s three main obstacles these scientists foresee in the world of big data analytics, including the complexity of big data solutions themselves.  There’s also the difficulty in applying valid statistical models to the data, as well as having limited insight into the actual meaning of the data at hand.

These are all issues we’ve seen many in the field run into, and a number of companies have already set out to address the problems of big data analytics.  It’s still a very broad field, with different implications across different industries.  The result is a plethora of analysis solutions designed for niche markets, as well as other inclusive offerings that fail at providing comprehensive means for extracting useful conclusions from data sets.  And that’s just speaking to the software side.  When it comes to the technology itself, there’s a long history of machines to manage, languages with which to comply, and integration issues for incorporating new technology that’s always full of promises.

And Revolution Analytics is one of the many trying to develop the right solution for the right problem.  With a commercial product called R software, Revolution is looking to widen the offerings of its open source statistic language, moving it beyond the realm of academia.  Founded in 1995 by two professors in New Zealand specializing in bioiformatics, the program was used primarily by academics.  But as a statistical-based program, it was utilized rather broadly even within this demographic.  Statisticians, researchers, life sciences and all sorts of experts began using the software, in in the last decade R has become a useful tool for statistical research and teaching.  Its diversity has its own place amongst big data analytics, but Revolution has bigger plans yet.

“What we at Revolution have been doing is partnering with the R software community for the past couple of years, helping to drive R out of its home in those labs and experimental use, into mainstream use in enterprises,” says Jeff Erhardt, Revolution’s COO.

“The way we’re doing that is beyond just being the company to back it, like some other open source companies have done.  We’ve worked to enhance R to make it more powerful, taking advantage of modern compute architecture to make it faster.  We’re also making it more accessible to others.”

Revolution’s story isn’t entirely different from others in the industry, taking an existing open source project and commercializing it for enterprise use.  There are a few areas the company is focusing on in preparing R for the enterprise, taking a flexible statistics tool and linking it tightly to data warehouses and Hadoop clusters.  Revolution is also keeping an eye on the other end of the spectrum, embedding its advanced analytics into end user tools that businesses already have in place.  This would include custom web applications for Microsoft products, for example.

In addressing these overarching issues reiterated by data scientists participating in the aforementioned survey, Revolution is building access to advanced models from learning from big data itself.  R software has built-in support for social network analysis, data mining and clustering, to name a few.  This gives Revolution a flexible basis to enhance for applying R software at large scale and at speed.  Revolution also has a support system for training and deployment, extending web-based APIs and other tools to help both IT and decision makers implement the right solution.

With such a heavy focus on analytics, Revolution must have highly integrated tools and a wide partner program to make its software accessible and usable.  With this in mind, R enables you to take the analytics to the data — Revolution’s not interested in storage or architecture.  They want to offer the best analysis tools they can.  R supports SaaS to scale, in part inspired by the NoSQL movement.  In its base state, R is an in-memory product, but Revolution’s improved it for use in existing environments, like Attivo.

Beefing up the built-in flexibility of R means that Revolution is building a powerful decision-making tool for the enterprise, enabling a software solution that can be applied to a range of situations.  Run models that wouldn’t otherwise have been able to be studied in a virtual environment, granting a peek at the future for any industry, covering a very broad series of data sets.

With a goal of providing insight to big data as a whole, Revolution is very much focused on the market potential of wide adoption.  “We want you to gain insight to make decisions you couldn’t otherwise do,” Erhardt goes on.  “Understanding rare events, or long term capital.  People are making decisions based on normality and linear trends, but as we know, life isn’t linear.  We’re able to allow people to do analysis without forcing the conditions of normality, and we can provide tremendous insight.  You can microsegment populations.  We’re positioning R and Revolution as the de facto tool for doing just that.”