Core capabilities

The Quanton engine: storage-aware Apache Spark

Quanton is the execution engine behind every Managed, Spark, SQL, Notebook, and Lakegres cluster on Onehouse. It runs Spark jobs, Spark SQL, PySpark notebooks, and Lakegres queries. There is nothing to install or turn on: create a cluster, and your Scala, PySpark, and Spark SQL code runs on Quanton unchanged. On the TPC-DS suite at 10 TB, Quanton runs 6x faster than open-source Spark and 25% faster than Databricks Photon on the same hardware, and on ETL it delivers 3-5x better cost-performance than Databricks with Photon, AWS EMR, and Snowflake.

Using Quanton

Existing jobs run as they are. Code written for open-source Spark, Amazon EMR, Databricks, or Dataproc moves over without a rewrite.
It reads and writes Hudi, Iceberg, and Delta Lake tables. Iceberg is a first-class target; see Apache Iceberg on Quanton: 3x faster Spark workloads.
Indexes speed up merges and joins on high-cardinality columns. Name the columns as accelerated columns, and Quanton builds and maintains the indexes itself; see Indexed joins and merges.
You can estimate the savings before you move. The Spark Cost Analyzer reads your Spark History Server and reports estimated savings per pipeline.
New releases reach you as platform upgrades. You don’t upgrade the engine yourself. Benchmarks, release notes, and engineering posts are on the engine’s own site, quanton.dev.

How fast

Quanton releases often and these numbers change with each release; quanton.dev/benchmarks has the latest results. At the time of writing, Quanton is faster than open-source Spark at every stage of an ETL pipeline:

Stage	Benchmark	Speedup vs open-source Spark
Read and transform	TPC-DS, 10 TB	6x
Load and update	LakeLoader, 1 TB	4.1x
Maintain tables	Iceberg and Hudi compaction	~4x

These are separate workloads and configurations, not a combined ETL score.

TPC-DS at 10 TB

TPC-DS at 10 TB: total runtime for all 99 queries on 11 × m8gd.4xlarge. Lower is better.

The same results, with Quanton’s speedup over each engine:

Engine	Runtime	Quanton is
Open-source Apache Spark	12,200 s	6x faster
Apache DataFusion Comet (tuned)	9,122 s	4.5x faster
Apache Gluten (tuned)	8,563 s	4.2x faster
Databricks Photon 18.2	2,550 s	25% faster
Quanton	2,034 s

Comet and Gluten use tuned configurations; their out-of-the-box runs failed with out-of-memory errors on queries 67 and 93.

Other workloads

Workload	Scale	Speedup vs open-source Spark
TPCx-BB	1 TB	2.0x
TPC-DI	1 TB	2.05x
LakeLoader (load and update)	1 TB	4.1x
Iceberg and Hudi compaction		~4x

These are engineering comparisons using TPC workloads, not audited TPC results; speedup is baseline runtime divided by Quanton runtime. Configurations, hardware, and versions for each run are on quanton.dev/benchmarks. The methodology is described in Measuring ETL price-performance on cloud data platforms, the load generator is open source as LakeLoader, and you can run the TPC-DS comparison yourself.

In production

On a public fintech’s production ETL, Quanton runs 25% faster than Databricks Photon. Photon is billed at about 2.9x the DBUs of standard Spark on the same instance, and Quanton carries no such premium, so the speed advantage turns into a larger cost advantage. The same fintech moved 532 pipelines from Amazon EMR onto Quanton and saves $2.6M a year.

Faster every release

Quanton ships on a regular release train, and the TPC-DS 10 TB runtime on the same hardware has fallen release over release: from 12,200 seconds in January 2025, matching open-source Spark, to 3,000 seconds in January 2026 and 2,034 seconds with v0.43.0 in September 2026. The full history is on the releases page.

Why it’s fast: storage-aware execution

Most of the time in a lakehouse job goes to reading files, shuffling data, and writing it back, and native vectorized execution alone does not reduce it. Quanton combines native execution with an understanding of how your tables are stored. It uses open table format metadata, table layouts, and available indexes to cut the data a job must read, process, and write, then runs the remaining work through fast native operators and optimized SQL plans. It optimizes planning, storage access, and execution together, at every stage of the pipeline.

Extract

Native reads. Native readers decode data in columnar batches and apply table-format operations directly. I/O pipelining overlaps storage requests with processing, and scan scheduling absorbs uneven file sizes. On TPC-DS query 23, the two scan stages drop from about 3.5 minutes each to about 35 seconds. See Speeding up lakehouse reads.

Transform

Rewritten native operators. Quanton builds on Velox with relational operators rewritten for execution efficiency, memory management, and cloud CPU architectures. See Introducing Quanton.
SQL plan reshaping. Rule-based rewrites driven by table layouts, column statistics, and indexes remove duplicate scans, collapse self-joins, and cut unnecessary shuffles, while your SQL stays the same. Pre-aggregating before joins alone takes 221 seconds off the TPC-DS 10 TB run. See Partial aggregation before joins.
Native rollup. A native ROLLUP operator replaces Spark’s row-expanding plan; TPC-DS query 67 at 10 TB drops from 4.4 minutes to 55 seconds. See GROUP BY ROLLUP without the row explosion.
Index-aware joins. A join operator that standard Spark does not have looks keys up in an index to find the matching files and row positions, then reads only the matching row groups. Indexed joins and merges below covers it.
Memory resilience. Quanton tracks native allocations against the executor’s memory budget and spills to disk instead of failing, so it completes queries where other native accelerators run out of memory. See Why native Spark accelerators get OOMKilled.

Load

Low-shuffle MERGE. Index lookups and a columnar MERGE path cut the shuffle in updates. A 1 TB MERGE INTO updating about 5% of rows drops from 21 minutes to 3.6 minutes, with about 1 TB less data shuffled. The indexes are covered in Indexed joins and merges. See Low-shuffle MERGE INTO for Apache Iceberg.
Native writers and compaction. Native columnar writers maintain the data files and metadata Iceberg and Hudi require, and native clustering of a 2 TB table takes about 75% less time. See Native clustering for Hudi and Iceberg tables.

The full design is on the engine architecture page, and Quanton compared sets it against Databricks, open-source Spark, Comet, and Gluten feature by feature.

Indexed joins and merges

Merges and joins on high-cardinality columns, such as a user ID or an order ID, end up scanning the table. Partition pruning and data skipping cut little when the matching keys are spread across every file, as late-arriving CDC usually is, so Spark reads and shuffles most of the target to find a small number of rows. The cost grows with the size of the table, not the size of the change.

Quanton adds a join operator that standard Spark does not have. It looks the keys up in an index to find the matching files and row positions, reads only those, and merges into the matched files in place, without shuffling the target table. It works on open tables, Hudi and Iceberg included, rather than being a feature of one format.

To use it, set spark.quanton.sql.optimizer.dynamicPruning.enabled=true and name the join or merge columns in spark.quanton.iceberg.accelerated_columns, or let Quanton pick them from the ON clause of your MERGE INTO statements. Your SQL does not change. Quanton checks the cardinality of each accelerated column and decides whether an index would pay off. Where it would, it builds the index in the background and keeps it consistent with concurrent writers. The first runs can be slower while the index builds; as the pipeline’s patterns settle and the indexes fill in, repeated runs get faster. This is Quanton’s overall design: the engine continuously builds and maintains indexes on your data, so repeat and incremental workloads speed up without extra tuning from you.

The index is kept current on every write. At plan time, Quanton uses pruning and data skipping when they are enough and the index when keys are scattered, then merges without a target shuffle.

With the indexes in place, MERGE INTO on Iceberg runs 16x faster than open-source Iceberg and 3.5x faster than with a simple index. See Index-aware MERGE for the design.

Quanton outside Onehouse

Quanton is also available unbundled from the platform, on its own, for teams that want to just run the fast engine on their own Kubernetes. It installs as a Kubernetes operator with a single helm upgrade --install, replaces the Kubeflow Spark Operator, and runs existing jobs unchanged on EKS, GKE, AKS, or on-premises. Switching a job back to open-source Spark is one configuration line. The local quickstart runs in under 15 minutes, and the Quanton Operator page covers the commercial offering.

The standalone Quanton Operator on your own Kubernetes. Purple-bordered components are supplied by Onehouse; gray ones are yours.

Quanton also runs inside Snowflake, on Snowpark Container Services, as real Apache Spark: the same DataFrame API, SQL, and spark-submit. On 1 TB TPC-DS it uses 63% fewer Snowflake credits than a Gen2 Large warehouse. See Real Apache Spark. Inside Snowflake.

Estimate savings on your own jobs

Benchmarks tell you how Quanton does on TPC-DS. The Spark Cost Analyzer tells you how it would do on your pipelines. It is a free, self-service tool: pip install spark-analyzer, point it at your Spark History Server, and it reads the runs that are already there.

Production Spark jobs are mixed. Some stages Quanton accelerates, some are dominated by custom code and UDFs, and the relational stages differ in whether they are scan-heavy, I/O-heavy, or transformation-heavy. The analyzer measures that split across the stages of each pipeline run, applies Quanton’s measured speedup to each kind of stage, and reports a weighted estimate of cost savings across your actual runs. The result is an estimate for your workload, not a benchmark number applied to it.

The usual path is to run the analyzer, pick the pipelines with the largest estimated savings, and move those from your current engine to Quanton first.