Sharmila Shahani-Mulligan, CEO and founder of ClearStory Data, discussed the company’s mission to turn data from disparate sources into insights available to all end users across corporations, with theCUBE co-hosts Dave Vellante and John Furrier, live at #BigDataNYC 2013 in New York.

“It has to get easier to consume data from many dissipate sources,” Shahani-Mulligan explained. “There’s going to be lot of data sitting in old repositories,” a lot of data circulating around the corporations, there’s always going to be data in the Hadoop clusters. Large companies are also consuming data from external data and the number of external data sources is increasing, with 8000 data APIs right now.

“Pulling all this dissipate data together to get to a broader insight is absolutely critical,” Shahani-Mulligan said. ClearStory’s data intelligence concept “is all about removing the barrier to accessing internal and external data and making it possible to converge data on the fly as you collect it from different external sources.” Being able to get from source data to an answer has to happen a lot faster.

“Data Intelligence is about speeding the whole process of going from dissipate data to a holistic view,” Shahani-Mulligan explained. It allows more people to access and see the insight than ever before.

How to democratize data

Asked how ClearStory gets the data in the hand of the users, Shahani-Mulligan said that currently there are 650 million information workers across corporations, a number expected to reach 800 million across business units, comprised of all “connected workers, that rely on data every day to make a decision.” A very small portion of them are data scientists and data architects. The company intends to make it easy and intuitive to understand what insights mean. “That is our target market,” and the company aims to achieve its goal through an application that is highly intuitive.

“Nobody is working in a silo. Everyone is working with other people in their business group, peers,” Shahani-Mulligan said. Allowing them to actively collaborate, contribute to insights is key. “We built into ClearStory the whole notion of data-aware collaboration.” “Data-aware collaboration is about letting lots and lots of people look at an insight” as the insight is created.

Scaling data integration

Explaining the integrated data processing engine and application of ClearStory, Shahani-Mulligan said it was “sitting on top of corporate repositories and in front of external sources of data.” The system determines or infers what’s in the data. “We understand the source.” The concept the company uses is intelligent data harmonization – “bringing dissipate sources together and determining what can be converged as a clear view.” In front of this platform there is an application layer “to make it very easy, simple, and intuitive to allow users to build a data story and interact with it through visual insights. Once in a data story you can collaborate with a lot of people. You allow people to reach a conclusion together and make a decision.”

“We build the data inference engine ourselves,” Shahani-Mulligan added. The data harmonization is also a technology the company has built and they currently have 7 patents pending. ClearStory uses Spark in connection with its in memory data unit that blends data from multiple sources. “That gets distributed in the Spark layer. Every time a user or information worker is clicking on the insight, it sends a signal to the back end and it pulls more data, generating a visualization.” It creates a constant cycle back to the source data. “This is the end user application with all the capabilities to iterate and drive through analysis,” Shahani-Mulligan added.

ClearStory’s smart visualization is based on an understanding of the shape, size, source of the data. “We present the best fit visualization for the nature of the data that you get. If you layer more data in, the visualization changes. As you drop data, the visualization changes. While you are doing this, you can collaborate with everyone across the company.”

Why ClearStory stands out

The company’s customers include a lot of companies in the CPG space, those in the food and beverage market which deal with lots of different locations and brands, organizations in the financial services areas, as well as national retailers.

Asked which differentiated the company, Shahani-Mulligan said, “we truly delivered on a product that has the ability for the business user to pick it up and get the insights.” Another plus is the external data which is readily available.