One of the major barriers to any successful Big Data project is mastering all of the technologies needed to make it happen. Big Data has evolved into a mish-mash of technologies, with Hadoop at the core and a clutch of associated platforms like the Apache Kafka messaging system and Apache Spark for in-memory computing, and ensuring they all run smoothly together is a challenge for even the best IT engineers.

With that in mind, a startup called Unravel Data emerged from stealth this week with $7 million in fresh funding, touting a new “performance intelligence” platform that’s designed to help enterprises easily monitor their Big Data environments.

The company’s pitch is that up until now, it simply hasn’t been possible to achieve visibility into the Big Data application stack – and that’s a problem for companies, because faults can arise anywhere in the stack and sorting it out can be a real pain point when they don’t know where to start looking. The second problem Unravel Data seeks to address is the ongoing lack of talent for running and maintaining Big Data systems – companies simply can’t find enough qualified personnel to keep things ticking over. But now they no longer need to worry about that.

Unravel Data is pioneering a new system that delivers “full-stack performance intelligence” by tracking all activity related to a businesses’ Big Data applications, datasets, infrastructure, and services, and automating their repair in case things go wrong. It’s capable of supporting all the major technologies in the Big Data ecosystem, such as Hadoop, Kafka, Spark and others, on-premises and in the cloud, and ensures that when problems crop up, IT can immediately get on top of them.

“Unlike monitoring tools that provide stack traces or logs and leave it to users to identify the root cause of issues, Unravel Data automatically discovers and fixes performance and reliability issues; proactively pin-points problems that could reside anywhere in the stack; provides end-to-end visibility and alerting for both data pipelines and ad-hoc applications; and provides deep drill-down visibility from the application level down to fine-grained resource utilization,” the company says in its pitch.

Speaking to IT BusinessEdge, Unravel Data CEO Kunal Agarwal explained how the company has spent several years analyzing more than 8 million Big Data jobs on a variety of platforms, creating a huge catalog of events which lead to Big Data application problems. Using that data, the platform is able to quickly determine the cause of of just about any issue that could crop up.

Unravel works by deploying sensors inside the various Big Data applications companies use. When the sensors come across a problem, those sensor quickly alert users to the problem and why it’s happening, and in many cases can automate whatever fix is required.

“Big Data systems are extremely complex, and getting them to work efficiently is considered a black art,” Agarwal said in a statement. “To truly derive business value, these systems and the applications running on them must be high-performing and easy enough for everyone to use. Unravel Data is comprehensive, solving the interoperability and performance challenges of multiple application engines in a complex Big Data environment. That’s the value we bring to the market.”

Photo Credit: amri_myriam via Compfight cc