UPDATED 09:00 EDT / MAY 05 2020

CLOUD

Dremio’s AWS-specific data lake query engine comes with a nice price: $0

Dremio Corp., a self-service analytics company whose technology is based on the Apache Arrow columnar processing engine, today is introducing a data lake query engine that’s purpose-built for Amazon Web Services Inc. infrastructure.

It also announced two new technologies that it said provide for on-demand service delivery and reduced cloud infrastructure costs. And in a nod to its open-source roots, the company is making the technology available at no charge.

Dremio, which raised $70 million in a funding round in March, was founded by Tomer Shiran (pictured, left) and Jacques Nadeau (right), who were both prominent developers of Arrow. The new Elastic Engines attacks the sticker shock that occurs when software that isn’t architected to take advantage of the cloud’s native elasticity causes customers to incur ongoing infrastructure costs for compute resources they aren’t using. Traditional scale-out query engines built on a single execution cluster can also cause those clusters to be either under-provisioned — which creates workload contention — or over-provisioned to cover peak demand, a situation that leads to under-utilization.

On-demand queries are often so slow that data engineers move the data into data warehouses, which incur additional overhead performing extracts, creating multidimensional cubes and aggregating tables. With Dremio AWS Edition, data teams can get high-performance query response to data in Amazon S3 object storage while reducing their cloud infrastructure costs by more than 90% compared with traditional SQL engines, the company said.

“People struggle to manage performance and cost in the cloud. We’re tackling both with Elastic Engines, which are independent and very resource-efficient,” said Jason Nadeau, vice president of marketing (pictured). “Every workload gets its own isolated environment that connects to a shared data set. You tailor the size of the engine to the workload it supports. Then we detect when workloads come in from a given engine and spin it up to its configured size until the queries stop.”

Dormant projects

Dremio uses the concept of projects, which live outside of the cluster and can be in a dormant state. “Think of projects as persistent-state data like definitions, virtual data sets, user data or anything you’d need to run an instance as a user,” said Kevin Dodson, the company’s senior director of product management.

Parallel projects are multitenant and isolated and have everything needed to manage the project, Nadeau said. “When you launch a project, it comes up with latest features, configured specifically for AWS.”

That includes automated configuration of a columnar cloud cache, Dremio’s physically optimized representations of source data known as Data Reflections optimized for S3 storage and automatic backups. “A project becomes a re-imagined version of a cluster,” Nadeau said. “It’s a high-level abstraction that allows things to go into a dormant state.”

Elasticity enables nodes to be added for a short period of time for performance purposes and then shut down, delivering what Dremio estimates is 60% cost savings by itself. “For a similarly sized (AWS) EC2 instance you’ll get better results out of Dremio,” Nadeau said. “You’d need a lot more cloud hardware with other query engines.”

The free pricing doesn’t have any usage caps. “You can go full production, all the high-scale analytics you want for free,” Nadeau said. The company is betting that as users scale up their workloads, they will opt in to the enterprise version of the product, which carries a license fee.

“We’re trying to make an impact,” he said. “We’re going for share.”

Executives declined to say whether they plan to make similar features available on other cloud platforms.

Photo: Dremio

A message from John Furrier, co-founder of SiliconANGLE:

Your vote of support is important to us and it helps us keep the content FREE.

One click below supports our mission to provide free, deep, and relevant content.  

Join our community on YouTube

Join the community that includes more than 15,000 #CubeAlumni experts, including Amazon.com CEO Andy Jassy, Dell Technologies founder and CEO Michael Dell, Intel CEO Pat Gelsinger, and many more luminaries and experts.

“TheCUBE is an important partner to the industry. You guys really are a part of our events and we really appreciate you coming and I know people appreciate the content you create as well” – Andy Jassy

THANK YOU