Big data as a service company Qubole Inc. kicked off its 2017 Data Platforms Conference with the launch today of what it says is the world’s first “autonomous data platform.”

The new platform is designed to help enterprises scale their big data teams and projects by automating many tedious and time-consuming tasks that were previously done by human operators.

The new platform incorporates three separate products: Qubole Data Service Community Edition, QDS Enterprise Edition and QDS Cloud Agents, which combine to intelligently automate and analyze platform usage to drive greater efficiency, the company said.

Qubole burst onto the nascent big data scene back in 2011 with its original Qubole Data Service product. QDS, as it’s called, provided a simple user interface that masks the complexity of the Apache Hadoop big data framework, thereby helping organizations make use of its capabilities more easily.

It works like this: Data is retrieved using QDS’s pre-packaged connectors, allowing users to enter simple queries and let the algorithms under the hood take it from there. The platform allocates hardware resources needed for a particular request and immediately frees up the infrastructure after everything is done to minimize overheads. This kind of automation enables customers to take advantage of Hadoop and its surrounding ecosystem of components more more easily than they would be able to in a manually-managed deployment.

Qubole says the automation of data platforms is becoming more necessary in a world where organizations face immense difficulty in collecting, processing and analyzing data at scale. This is due to both the complexity of big data and also a lack of skilled data scientists, engineers and analysts – problems that make it necessary to automate manual effort and augment human tasks if companies are to scale their big data initiatives, Qubole said.

Qubole’s new platform is able to self-manage and self-optimize itself, learning by watching platform usage so it can make data teams more effective. Specifically, it analyzes metadata from queries, clusters, users and data being ingested by the platform, and uses machine learning to create alerts, insights, recommendations and also autonomous agents that can perform necessary actions without human input.

“Despite all the promise and technological advancements, operationalizing big data efforts is still impossible for most organizations,” said Ashish Thusoo (pictured), Qubole’s cofounder and chief executive officer. “Data teams simply can’t scale to meet the growing demand for data across organizations. To address this hurdle, we are on a mission to remove the manual effort that comes with maintaining a big data infrastructure and empower data teams to focus on high-value, strategic work.”

The most interesting component of the new platform is QDS Cloud Agents, which refers to a collection of “software agents” that can perform tedious and time-consuming tasks that were previously handled by humans.

Qubole has created three software agents for this initial release, including a Workload Aware Auto-Scaling Agent that’s able to optimize cluster sizes for workload requirements, thereby reducing over-provisioning of resources. There’s also a Data Caching Agent that allows users to optimize the locality of their data for faster access speeds.

Finally, there’s also a Spot Shopper Agent exclusively for Amazon Web Services, that shops around the AWS cloud to assemble compute instances in the most optimal cost/performance combination. The agent browses through various AWS Spot Instances, Spot Blocks, mixed instance types and different availability zones to find customers the best deal at that precise time. By doing so, Qubole reckons companies can save up to 80 percent on the cost of on-demand instances.

The new platform will be available in July.

Image: SiliconANGLE