The best Spark accelerators in 2026, compared

On Databricks, Photon is the only Apache Spark™ accelerator, and on Google Cloud or Microsoft Fabric, the bundled Apache Gluten™ engine is the shortest path. On self-managed Spark, benchmark Gluten and Apache DataFusion™ Comet first, use the NVIDIA cuDF plugin for Apache Spark (formerly the RAPIDS Accelerator) on GPU-heavy joins and sorts, and Quanton matches Photon on transformation speed, with option to be used outside its platform (Onehouse). It uniquely addresses cost reduction and speed up of I/O intensive workloads like scans, merges and table maintenance.

Each engine named above is a Spark accelerator: it makes existing Spark jobs run faster without a rewrite. Most do it by replacing Spark’s row-at-a-time JVM execution with a native, vectorized engine, and others move the work onto GPUs.

The options in 2026 include open-source plugins, a GPU library, engines bundled into managed platforms, and commercial drop-in engines. They differ on what they accelerate, where they run, and how you pay for them. Some speed up only the compute stage of a query and hand reads, writes, or unsupported operators back to Spark.

Quanton, the Spark engine from Onehouse available on its own at quanton.dev, gets the same treatment as every other entry below: how it works, where it helps, and where it falls short.

What are Spark accelerators?

Spark plans a query with its Catalyst optimizer, then executes the physical plan on the JVM through generated Java code, largely one row at a time. A Spark accelerator keeps the same APIs, the same SQL, and the same logical plan, and changes how the physical plan executes. It takes the physical plan, runs the operators it supports on a faster engine, and falls back to Spark for the rest.

Most accelerators are CPU-native engines written in C++ or Rust that process columnar batches with SIMD instructions. Velox (open source, originated at Meta) and Apache DataFusion™ are the two most common foundations. GPU acceleration does the same job on NVIDIA hardware.

Quanton and Photon go beyond physical execution and bring storage awareness into the engine. Quanton’s storage-aware planning reads Apache Iceberg™ and Apache Hudi™ table metadata and reshapes the query plan around it. On Databricks, predictive I/O, which is exclusive to Photon, reads only the data a selective query needs and, on Delta Lake tables with deletion vectors enabled, rewrites less data during MERGE, UPDATE, and DELETE.

The APIs, SQL, and logical plan stay the same. Supported operators run on a native engine and the rest fall back to Spark’s JVM engine, while Quanton’s storage-aware planning also reads table metadata to reshape the plan.

This list covers execution accelerators: engines that run the same Spark job, unchanged, on faster machinery. Caching layers and managed tuning services work differently, so they are not ranked here.

Spark accelerators at a glance

The practical differences come down to where each accelerator can run and how it is billed. Hardware matters too, since only one entry needs GPUs.

Accelerator	Engine underneath	Runs on	Cost model	Best for
Quanton	Storage-aware, end-to-end optimized engine on Velox with reimplemented operators (C++), CPU	Your Kubernetes, VMs, or existing Spark platform	Per GiB processed, first 100 GiB/month free	General Spark speedups and lower bills, from SQL queries to ETL and lakehouse writes, on any Kubernetes cluster (on-prem, cloud)
Databricks Photon	Proprietary C++ engine, CPU	Databricks only	Databricks DBUs, at a Photon-specific rate on classic compute	Teams committed to Databricks
Apache Gluten™	Velox or ClickHouse backend (C++), CPU	Self-managed Spark 3.x and 4.x	Free, Apache 2.0	Technical teams that can self-manage and tune off-heap memory
Apache DataFusion Comet	DataFusion (Rust), CPU	Spark 3.4 to 4.1 (4.2 experimental)	Free, Apache 2.0	Technical teams that can self-manage and tune off-heap memory
NVIDIA cuDF plugin for Apache Spark	cuDF on NVIDIA GPUs	Spark, EMR, Databricks, Google Cloud	Free plugin, paid GPU instances	Large joins, sorts, and aggregations
Apache Auron™	DataFusion (Rust), CPU	Self-managed Spark	Free, Apache 2.0	A second DataFusion option to benchmark
Google Lightning Engine, Fabric native execution engine	Gluten and Velox with vendor additions	Google Cloud or Microsoft Fabric only	Platform pricing (premium tier on Google serverless)	Teams already on those platforms
Flarion, DataPelago	Proprietary engines	Existing Spark platforms	Commercial licence	Teams that want a vendor-supported plugin

Status as of October 2026. These projects ship often, so confirm details against each project’s documentation, and see the feature-by-feature comparison for Quanton against Photon, Gluten, and Comet.

1. Quanton: a Velox-based engine that also optimizes SQL plans, reads, writes and table maintenance

The Quanton engine is a drop-in replacement for Spark’s execution engine, built by Onehouse on Velox with optimized and reimplemented operators. Quanton runs in your own cloud account, on your Kubernetes clusters, on plain VMs, or inside an existing Spark platform. Existing jobs run with no code changes.

The difference from the other CPU accelerators is scope: Quanton is storage-aware and optimized end to end, across the read, the compute, and the write. Its storage-aware planning reads Apache Iceberg™ and Apache Hudi™ metadata to reshape query plans, and its index-aware joins locate rows instead of scanning for them. Quanton also writes natively, with a low-shuffle columnar MERGE and native compaction.

In Onehouse’s own TPC-DS 10 TB run (all 99 queries on 11 × m8gd.4xlarge instances), Quanton finished in 2,034 seconds against 12,200 seconds for open-source Spark, about 6× faster. Tuned Gluten took 8,563 seconds and tuned Comet took 9,122, which puts Quanton about 4.2× ahead of Gluten and 4.5× ahead of Comet. In out-of-the-box runs, Gluten and Comet hit out-of-memory failures on q67 and q93.

Against Databricks Photon the gap is much smaller. Photon 18.2 finished the same suite in 2,550 seconds, so Quanton was 25% faster on these TPC-DS reads. That margin moves with every release of both engines, and the full setup is in the published benchmark results.

With performance that close, the bigger difference for TPC-DS style workloads from Photon is the billing model. Quanton bills per GiB processed instead of per compute hour, adds no DBU markup, and leaves your reserved and spot discounts with you.

Quanton has trade-offs. The TPC-DS results above are Onehouse’s own, and no independent run has reproduced them yet. The engine replaces only Spark’s execution layer, so catalogs, orchestration, and job definitions stay where they are.

In spite of deep lakehouse expertise in open-source, Onehouse is also a younger vendor than Databricks or NVIDIA, and Quanton is open core, so parts of the engine are under a commercial licence. Billing per GiB tracks data volume, so scan-heavy jobs with light compute see less benefit from a faster run.

Choose Quanton if the goal is a lower Spark bill than EMR**/Databricks**, or improving performance on Kubernetes, or on-prem clusters, or if MERGE, compaction, and other writes take a large share of your runtime.

2. Databricks Photon: the native engine inside the Databricks runtime

Photon is Databricks’ vectorized query engine, written in C++. Spark’s Catalyst optimizer still plans each query, and Photon takes over execution for the operators it supports, falling back to the Spark runtime for the rest. Photon runs only on Databricks, where Photon is on for SQL warehouses and serverless compute and enabled by default for classic compute created in the UI (clusters created through the API must set runtime_engine to PHOTON).

Photon covers scans, hash joins and aggregations, sorts, window functions, and a native Apache Parquet™ writer for Delta Lake, Iceberg, and Parquet tables, including MERGE INTO. The largest gains come on long-running queries over large tables.

The limits are documented. Photon does not support UDFs, RDD or Dataset APIs, or stateful streaming, and queries that finish in under two seconds see no meaningful gain. On classic compute, Photon instance types consume DBUs at a different rate than the standard runtime, so a faster job is not automatically a cheaper one.

Our guide to Databricks Photon works through that DBU math in detail. Choose Photon if you are committed to Databricks and your workloads are SQL and DataFrame heavy rather than UDF heavy.

3. Apache Gluten: Velox under open-source Spark

Apache Gluten is a Spark plugin that converts Spark’s physical plan into a Substrait plan and hands the supported operators to a native backend. Velox is the default backend and ClickHouse is the alternative. Gluten is an Apache Top-Level Project under the Apache 2.0 licence, and the project also underpins the Google and Microsoft engines further down this list.

On the Gluten project’s benchmarks, Gluten with Velox ran TPC-DS 3.0× and TPC-H 3.3× faster than Spark 3.3.1 at 3 TB. In an independent AWS Labs benchmark on TPC-DS 3 TB, Gluten 1.6.0 finished about 9% faster overall than Comet 0.16.0.

AWS Labs’ Gluten benchmark setup used 23 executors with about 52 GB each (20 GB heap plus 32 GB off-heap) on 12 r8gd.12xlarge nodes, roughly 1.2 TB of memory for 3 TB of data. Many production clusters run with less memory per terabyte, so the same result may not hold.

Unsupported operators, expressions, and file formats fall back to vanilla Spark, with row-to-columnar conversions at every boundary. Support for ANSI mode, which Spark 4 turns on by default, is still in progress.

Native work also moves into off-heap memory, which the container limit enforces even when Gluten’s own accounting shows headroom. Onehouse has documented why native accelerators get OOMKilled under that kind of memory pressure.

Choose Gluten if you run self-managed Spark, want a free native engine, and have time to size off-heap memory and check fallbacks job by job.

4. Apache DataFusion Comet: a Rust engine that reached 1.0

Apache DataFusion Comet translates Spark physical plans into DataFusion plans and runs them in Rust on Apache Arrow™ columnar batches. Comet reached version 1.0.0 in August 2026, and 1.1.0 followed in October. The 1.0 release supports Spark 3.4 through 4.1, with experimental support for 4.2, and fully supports ANSI mode.

Comet’s distinctive feature is codegen dispatch. For an expression without a native implementation, Comet runs Spark’s own generated code inside the native pipeline instead of sending the whole subtree back to Spark. That keeps more of each plan native and lets existing Scala and Java UDFs run without a rewrite.

Comet’s native scans cover Parquet files and Iceberg tables, so Delta Lake and Hudi tables still read through Spark. Comet 1.1.0 (October 2026) added native handling of Iceberg V3 deletion vectors, and other V3 features still fall back. In the AWS Labs run, which used Comet 0.16.0 before the 1.0 release, Gluten finished about 9% ahead overall, though Comet was faster on more individual queries.

Our Comet deep dive maps what Comet accelerates and what it leaves on Spark. Choose Comet if you run open-source Spark on Parquet or Iceberg, value a stable 1.x release line, and want UDF-heavy jobs to stay mostly native.

5. NVIDIA cuDF plugin for Apache Spark: GPU acceleration for heavy SQL

The NVIDIA cuDF plugin for Apache Spark is NVIDIA’s GPU plugin for Spark. The plugin replaces supported parts of Spark’s physical plan with GPU operators built on the cuDF library, and it includes a shuffle that can move data directly between GPUs. The plugin is open source under Apache 2.0, and 26.08 is the current release.

NVIDIA’s plugin FAQ is specific about fit. High-cardinality joins, aggregations, and sorts benefit most, along with large windows, Parquet and Apache ORC™ writes, and format transcoding. Complicated queries that run entirely on the GPU typically see 3× to 7× speedups, with 4× the typical case.

The plugin runs on open-source Spark, Amazon EMR with GPU instances, Databricks, and Google Cloud. The trade-off is hardware. Executors need NVIDIA GPU instances, so the savings depend on GPU pricing and availability in your region. Gains on actual production Spark workloads are very subjective, since it is highly sensitive to whether most operators can run on GPU or not. Else, GPUs can sit idle and rack up bills.

Fit matters as much as hardware. Short jobs pay two to three seconds of GPU initialization each time an executor starts, and rarely recover it. Dataset code built on opaque lambdas and RDD-based work stay on the CPU, and structured streaming has not been a focus for the plugin.

Choose the cuDF plugin if your heaviest jobs are large joins, sorts, and aggregations and you already run, or can justify, GPU capacity.

6. Apache Auron: a DataFusion engine that started at Kuaishou

Apache Auron began at Kuaishou as Blaze and graduated to an Apache Top-Level Project in September 2026. Like Comet, Auron translates supported Spark operators into a native plan that DataFusion executes in Rust, and Auron installs as a Spark extension.

Auron covers Parquet scans, shuffle and broadcast exchanges, the main join types, hash and sort aggregation, and window functions. Auron can also fall back a single unsupported expression, including a UDF, to Spark instead of the whole operator. Its operator support list flags the gaps: the native Parquet writer is disabled by default, the ORC scan is incomplete, and window frames are limited to the default range.

Auron’s listed adopters are mostly large internet companies in China, including Kuaishou, DiDi, and Bilibili, and English-language benchmark data is thinner than for Gluten or Comet. Choose Auron if you want a second DataFusion-based engine to benchmark and are comfortable working closely with the project’s community.

7. Google Lightning Engine and Microsoft Fabric: Gluten bundled into the platform

Two cloud platforms ship Gluten and Velox as a managed feature. Google’s Lightning Engine has been generally available on Managed Service for Apache Spark (formerly Dataproc) since June 2026. Google adds cached broadcast hash tables, aggregation pushdown, and faster Cloud Storage and BigQuery connectors, and reports up to 4.9× faster performance than standard open-source Spark.

Microsoft Fabric’s native execution engine is also built on Velox and Gluten. The Fabric engine runs on Fabric Runtime 1.3 (Spark 3.5) and Runtime 2.0 (Spark 4.1), supports ANSI mode on Runtime 2.0, and does not accelerate JSON or XML reads or structured streaming.

The trade-off is portability. Both engines run only on their vendor’s platform, serverless Lightning Engine jobs require Google’s premium tier, and both differ in behavior from open-source Gluten. Choose them if you already run Spark on Google Cloud or Fabric and want acceleration as a configuration setting.

8. Flarion and DataPelago: commercial drop-in plugins

A few startups sell proprietary Spark accelerators that install as a JAR on an existing platform. Flarion’s execution engine is written in Rust on Arrow and DataFusion and deploys on Databricks, EMR, Google Cloud, Azure HDInsight, Kubernetes, or on-prem clusters. Flarion adds plan rewrites and workload caching on top of native execution and cites 3× faster jobs at 60% lower cost.

The DataPelago Accelerator for Spark, launched in August 2025, runs on DataPelago’s Nucleus engine and decides at runtime whether each operation runs on CPU or GPU. DataPelago claims up to 10× speedups and 80% lower cost, with customer results between 2× and 4× in production.

Neither engine is open source, and both vendors publish headline figures without a full benchmark setup on those pages. Choose one if you want a vendor-supported plugin on your current platform and will validate the claims against your own jobs.

How to choose a Spark accelerator

Headline benchmarks matter less than a few facts about your own environment. The platform you run narrows the list, native coverage on your jobs decides the real speedup, and the billing model decides whether a faster job is also a cheaper one.

Narrow the list by where Spark runs

Photon, Lightning Engine, and the Fabric engine come with their platforms, so for teams already there the decision is mostly whether to switch them on. On EMR, Kubernetes, or on-prem clusters, the candidates are the portable ones: Gluten, Comet, Auron, the cuDF plugin, Quanton, and the commercial plugins.

Hardware is the other filter. The cuDF plugin needs NVIDIA GPU instances for its executors, while every other entry runs on the CPU instances you already have.

Measure native coverage, including writes

Every accelerator on this list falls back to Spark for something. Read the plan and stages on your largest jobs and measure the native share by runtime, not by count, because one fallen-back stage can hold most of a pipeline’s cost.

Check the write path too. Gluten and Comet speed up the join execution with columnar shuffle inside a MERGE, and both are adding native writers, but neither has a low-shuffle MERGE or native compaction, so upsert-heavy pipelines don’t shed much of their cost on Spark. Without such optimizations, the gains on native read/write paths will remain only marginal 10-20%.

Understand how a faster job changes the bill

With the free plugins, a faster job means fewer instance hours, and the GPU plugin trades fewer hours for more expensive instances. Photon, Lightning Engine, and the Fabric engine bill through their platforms, where the accelerated runtime can carry its own rate.

Quanton bills per GiB processed, so a faster run lowers compute instance hours while the Quanton fee stays flat based on data volume. The Quanton TCO calculator models a run against Databricks Classic, Databricks Serverless, and open-source Spark on your own inputs.

Which Spark accelerator should you run?

For Databricks teams, Photon is the default, and the open question is cost per job. On Google Cloud or Fabric, the bundled Gluten engines are the shortest path. For self-managed Spark, Gluten and Comet are free and worth benchmarking, and the cuDF plugin fits GPU-heavy pipelines.

Quanton fits teams whose cost sits in large ETL transformations and lakehouse writes and who want each speedup to show up also as a lower bill. Quanton runs in your own account, bills per GiB, and reverts to open-source Spark with one config line. The first 100 GiB each month are free: create an account and run the side-by-side on your own jobs, in your own VPC.

FAQs

Are caching layers like Alluxio Spark accelerators?

Not in the sense used here. Alluxio is a distributed caching layer for data in remote storage. Deployed alongside Spark, Alluxio keeps that data close to the executors, which cuts I/O latency and storage API costs without application rewrites.

A cache changes how data reaches Spark and leaves execution alone, so the operators still run on Spark’s JVM engine. An execution accelerator changes how the query runs once the data is read.

Do Spark accelerators require code changes?

Not for SQL and DataFrame code. Every entry on this list installs as a plugin, a config setting, or a platform toggle, and the same jobs run unchanged. Coverage is what varies: UDFs, RDD code, and Dataset lambdas often run on Spark, and the native engines need memory settings such as off-heap sizing.

Is a GPU or a CPU Spark accelerator cheaper?

The answer depends on how compute-bound and how long-running your jobs are. A GPU plugin can deliver a larger speedup on heavy joins and aggregations, but GPU instances cost more per hour, and short or I/O-bound jobs do not recover the difference.

CPU accelerators run on the instances you already pay for, so a speedup turns into fewer instance hours. NVIDIA’s qualification tool and Gluten’s equivalent read your Spark event logs and estimate the fit job by job.

Can you run more than one Spark accelerator on the same cluster?

Not on the same job, because each accelerator replaces the same execution layer. You can choose per job, though. The Quanton Operator, for example, lets you set spark.quanton.accelerator=gluten or spark.quanton.accelerator=comet on an individual Spark job, so Quanton, Gluten, and Comet can run side by side on one Kubernetes cluster.