As companies begin to move data to cloud data warehouses and data lakes, traditional data warehouse solutions are becoming less desirable. But these new data solutions for the cloud come with a few nagging bottlenecks as well.

“There’s been a large amount of adoption of cloud data warehouses, but fundamentally they still come with a lot of the same challenges that have always existed with the data warehouse,” said Mark Lyons (pictured), vice president of product management at Dremio Corp.

Founded in 2014, Dremio is a data lake engine that has intrinsic support for interactive queries. It is the first of its kind to be purpose-built to support Apache Arrow and was created by its founders to overcome the perceived barriers enterprises have with accessing big data, Lyons explained.

Lyons spoke with theCUBE industry analyst Lisa Martin in advance of the AWS Startup Showcase: “Data as Code — The Future of Enterprise Data and Analytics” event, an exclusive broadcast on theCUBE, SiliconANGLE Media’s livestreaming studio, airing April 5. They discussed new Dremio announcements and how they’ll fit into the company’s enterprise data lake offerings. (* Disclosure below.)

The market’s current state

Proprietary data warehouses are great at analyzing structured data that seep in from the data center but struggle with things like the volume and velocity of modern data streams. While many stop-gap solutions are being used by today’s organizations to combat those drawbacks, it’s all resulting in changes to data pipelines that are fickle and can often take unprofitable amounts of time.

Dremio helps companies better operationalize their data, according to Lyons. Since data has a half-life and decays, it’s imperative that companies analyze data sooner rather than later, ahead of the impending decay, he added

“For Dremio and our customers, our story is simple. It’s to leverage the data where it is: accessing data in all sorts of sources — whether it’s a Postgres database or an S3 bucket, don’t move or copy the data; analyze it in place,” he said. “And don’t limit the scope of the data you’re trying to analyze.“

In addition, where there are new use cases and datasets to be added, users can just import them into S3 and proceed as usual. Dremio’s solution makes the jobs of data engineers and analysts easier and much more productive, according to Lyons, who added that the company’s solution reduces the time spent “worrying and waiting” and ensures that crisp, concise insights reach the end consumers.

The lakehouse concept

Dremio’s open data lakehouse is made up of two constituent parts: Arctic and Sonar. Arctic is built atop Apache Iceberg, giving it considerable flexibility and scalability in automating the data management process. On the other hand, Sonar is the SQL query engine that works directly within the data lake, mitigating the need for things like complex, failure-prone data pipelines and other costly workarounds, Lyons explained.

In stark contrast to the traditional data lake, which was often a “parking ground for just lots of files,” the lakehouse is a more modern iteration with increased accessibility and improved governance, auditing and access authentication.

“Everything is great from the front of the house, with access policies and data masking, everything that you would expect through commits, tables, transactions, inserts, updates deletes and auditing of that data,” Lyons stated. “You’re able to see who made the changes to the data, which engine, which user, when were they made and the whole history of a table and not just a mess of files in a file store.”

The “forever free” promise from Dremio on its standard software tier is also a prominent topic. While customers still have to account for costs like cloud infrastructure bills like S3 with AWS, accessing the actual software with the standard configuration tier from Dremio is indefinitely free. Dremio also undertakes all of the query planning, engine management, and general administration.

“Because of the Dremio cloud architecture, where the execution and data live in the customer’s cloud account, we’re able to basically say … the Dremio service side of this platform is forever free for users,” Lyons concluded.

Here’s the complete video interview, part of SiliconANGLE’s and theCUBE’s pre-event coverage of the AWS Startup Showcase: “Data as Code — The Future of Enterprise Data and Analytics” event.

(* Disclosure: TheCUBE is a paid media partner for the AWS Startup Showcase: “Data as Code — The Future of Enterprise Data and Analytics” event. Neither Dremio Corp., the sponsor for theCUBE’s event coverage, nor other sponsors have editorial control over content on theCUBE or SiliconANGLE.)

Photo: SiliconANGLE