Dremio Corp., developer of an SQL query engine that works on the semistructured data repositories called data lakes, is kicking off its Subsurface conference today with what it says is the most significant set of new and updated products in its history – and all are perpetually free.

They include the general availability of its cloud service and two new services: Dremio Sonar, a lakehouse engine built for SQL, and Dremio Arctic, a metadata and data management service for the Apache Iceberg high-performance format for large analytic tables.

The company said Dremio Cloud is the first lakehouse platform to be built from scratch for SQL workloads. “In the past, companies had to weigh the pros and cons of data lakes versus data warehouses,” Chief Executive Billy Bosworth said in a prepared statement. “We’ve eliminated this tradeoff by providing a frictionless and infinitely scalable service that delivers the combined benefits of both.”

‘Easy to get started’

Dremio Cloud is being made available as a perpetually free service on the assumption that “companies will become familiar with Dremio, run their analytics on our platform and some percentage of them will later realize it’s valuable and want enterprise security and support, service-level agreements and integration with their cloud identity management system,” said co-founder and Chief Product Officer Tomer Shiran (pictured). “We’re focusing on making it easy for people to get started.”

Dremio Sonar is a free service built for SQL that delivers all the performance and features of a data warehouse on Amazon S3 object storage. Based on the Apache Arrow columnar in-memory analytics engine, Dremio Sonar is 68% faster than its predecessor and supports inserts, updates, deletes and transactions, or “all the operations you can do in a data warehouse query,” Shiran said. This enables users to run business intelligence workloads directly on the lakehouse.

Dremio enables subsecond query response times by automatically and transparently rewriting query plans to utilize different aggregations or layouts of tables and views, the company said. Its built-in semantic layer allows data operators to provide a consistent and secure view of data to consumers along with self-service curation of new data sets.

Dremio SQL Runner and SQL Profiler are new visual tools within Sonar that enable SQL-proficient users to run and troubleshoot queries and scripts directly from the Dremio interface. A set of database connectors allows for joins across multiple data sources, including relational stores. Connections are built in to popular client tools, including Microsoft Corp.’s Power BI, Tableau Software Inc.’s Tableau, the dbt data transformation tool, Hex Technologies Inc.’s collaborative data workspace and Preset Inc.’s data exploration and visualization platform.

Like Git for data

Dremio Arctic is another free service based on the open-source Apache Iceberg analytics table format and Nessie data catalog that delivers a Git-like experience for the data lakehouse. Git is an open-source distributed version control system favored for its snapshotting capabilities.

The Dremio Arctic metastore simplifies data engineering and analysis by making the experience of working with data workflows similar to working with source code. That includes branching for data ingest, transformation and experimentation in isolation, the ability to reproduce historic dashboards or machine learning models without maintaining data copies and a GitHub-like interface.

The cloud service persists data in Apache Iceberg tables backed by Apache Parquet files on Amazon Web Services Inc. S3 storage. Users are free to use any query engine they want, even if it doesn’t come from Dremio, Shiran said.

“The Git experience hasn’t existed for data,” he said. “This fits with how engineers work with data. They want to work in isolation and then join it and to have tables that span multiple users.”

Dremio Sonar and Dremio Cloud are now generally available. Dremio Arctic is in public preview.

Photo: SiliconANGLE