Core capabilities

Formats and optimization

Onehouse organizes your tables into lakes and databases and keeps them fast with table services that run in the background, on Apache Hudi and Apache Iceberg tables. Any table can also be read as Delta Lake.

Lakes, databases, tables

Every table lives in a lake and a database, and each level maps to a directory in object storage. Access is granted per lake, and separate lakes can sit in separate buckets.

A managed lake holds the tables Onehouse writes.
An observed lake registers Hudi and Iceberg tables that other engines write. Point it at a path prefix, and Onehouse finds the tables there, so you can monitor and optimize them without moving or copying them.

Table overview: record count, storage size, file sizing, partitions, and data written per day.

Table monitoring

Browse every table in the console under Data, as lake, database, and table, with format-aware statistics that show whether a table is healthy, for Hudi and Iceberg alike:

Overview: total records, storage size, average file size, average partition size, and file group count, with data written and other table metrics charted over a time range you choose.
Partition insights: the distribution of partition sizes, and per-partition file-size percentiles from p10 to p99, with each partition’s skew against the table median and its commit activity over the past day, so oversized, undersized, and hot partitions stand out.
Compaction backlog: un-compacted log files by partition and file group, their change over the past day, and when each was last compacted.
Operations and history: every commit and table service run, with its status and duration.

Alerts fire on thresholds you set, such as commit lag, compaction backlog, or file sizes, and arrive in the console, by email, or in Slack.

The partition insights view, showing partition size distribution and per-partition file-size percentiles, with skewed partitions flagged.

Partition insights: size distribution and per-partition file-size percentiles, with skew flagged.

Table services

Table services keep tables fast as data lands. They run on managed tables and on external tables you register. Tables created by Flows get cleaning and compaction automatically. You turn on the other services per table, to run automatically or on demand.

Service	Purpose	Hudi	Iceberg
Clustering	File sizing and sort layout: linear, Z-order, or Hilbert	Supported	Supported
Compaction	Merges log files or deletion vectors into base files	Supported	Supported, including Iceberg v3 deletion vectors
Cleaning	Reclaims storage past the time-travel retention window, including orphan files	Supported	Supported
Metadata maintenance	Expires snapshots and rewrites manifests so query planning stays fast	Supported	Supported
Metadata sync	Registers the table to catalogs and translates between formats	Supported	Supported

Automatic runs trigger on a commit-count frequency (snapshot count, for Iceberg) or on an hourly, daily, weekly, or monthly schedule. Each service has a status page, configuration, and run history.

By default, table services share a cluster with the Flows writing the same tables, and the two can compete for resources. You can also move table services to a separate Managed cluster, so the Flows writing those tables are unaffected. Clustering can use a lot of resources, depending on its configuration, so it is a common candidate for a cluster of its own.

At a $700B+ pension fund, Table Optimizer sped up Athena 1.7x, Spark 9x, and Trino 12x on the same tables, with no query changes. Table Optimizer runs these services asynchronously, with 2–4x less compute. The services run on Quanton, whose native columnar merging compacts Hudi and Iceberg tables about 4x faster than open-source Spark and clusters them in about 75% less time.

Clustering history on an Iceberg table. Each run is a replace snapshot, and the ID shown is the Iceberg snapshot ID.

Incremental clustering

Clustering keeps up with the table as data lands. Each run plans only the partitions and files written since the last completed clustering, using commit metadata, write patterns, and table statistics, so new data is organized without rewriting what is already clustered. Writers keep committing while clustering runs in the background, with snapshot isolation between them, so ingestion latency is unaffected.

You choose the clustering columns and the layout strategy, from linear sorting to multi-dimensional Z-order and Hilbert curves, and the layout evolves as new data is written instead of being fixed at table creation. Clustering works the same way on Hudi and Iceberg tables. See What is clustering in an open data lakehouse.

Iceberg tables

Table Optimizer supports Iceberg v3, including deletion vectors and variant columns with shredding. It compacts and clusters Iceberg tables written by any engine, through Iceberg’s own rewrite_data_files procedure, committing each run as a native snapshot, and compacts deletion vectors in the background so update-heavy tables do not slow readers down.

In a comparison with Amazon S3 Tables on the same 953.7 GB of small files, Table Optimizer on a single m8g.xlarge compacted every 30 minutes instead of every 3 hours, averaged 11.3 minutes per compaction against 56.2, and cost 2–3x less even after AWS’s price cut. It keeps file sizes in check within about a minute of writes, and you control the compaction frequency, target file size, the budget of data compacted per run, sort keys, and layout strategy. See AWS S3 Tables after the 10x price cut.

Iceberg metadata is maintained at write speed through Apache XTable (Incubating): snapshots and manifests expire without scanning data files, and data files and orphans are reclaimed incrementally and asynchronously, so near-real-time writers never wait on maintenance. See Managing Apache Iceberg metadata for near-real-time workloads.

Record-level and secondary indexes on Iceberg tables are maintained on write and used by Quanton’s index-aware MERGE; see Indexed joins and merges.

Table formats

OneFlow and Table Optimizer work natively on Apache Hudi and Apache Iceberg, and Quanton reads and writes Delta Lake as well. Iceberg is supported natively across all Onehouse services:

Latest Iceberg 1.x ships on every Spark cluster.
SQL and Spark clusters run directly against Iceberg REST catalogs.
Clustering on Iceberg tables uses Iceberg’s own rewrite_data_files procedure and commits as a native snapshot.
External Iceberg tables register for optimization.
Quanton’s published Iceberg benchmark is described in Apache Iceberg on Quanton: 3x faster Spark workloads.

Through sync built on Apache XTable (Incubating), one physical copy of a table is also exposed in the other formats, such as Delta Lake, so engines that prefer different formats read the same data with no copy. Optimizations applied to the underlying files benefit every format the table is exposed as. Catalog registration is covered in Catalog interoperability.