Apache DataFusion Comet: what it accelerates, and what it doesn't

Apache DataFusion™ Comet is a plugin that speeds up Apache Spark™ by running supported parts of a query on a native, vectorized engine instead of the JVM. It is open source, drop-in, and genuinely fast on the workloads it covers. It also hands a large share of real pipelines back to Spark, often with no obvious signal that it did.

The useful question is not whether Comet is fast. It is which parts of your workload actually run on Comet, and which quietly stay on Spark. That line, between what Comet accelerates and what it leaves behind, decides whether it cuts your bill or barely moves it.

What Apache DataFusion Comet is

Comet is an Apache Software Foundation project that accelerates Spark SQL and DataFrame workloads. It intercepts Spark’s physical plan, translates the operators it supports into a serialized format, and hands them to the DataFusion engine, a vectorized query engine written in Rust. Data moves between the two in Apache Arrow™’s columnar format.

The idea is that you get native execution speed without rewriting anything. You add a JAR and a set of Spark configs, and existing queries run on the native engine wherever Comet can handle them. There is no new API and no migration.

Comet runs on commodity hardware, with no GPUs or FPGAs required, and it is licensed under Apache 2.0. It is closest in concept to Databricks Photon: a native execution layer bolted under Databricks’s Spark distribution. The difference is that Comet is open source and community-governed, while Photon is proprietary to the Databricks runtime.

Comet is a partial accelerator by design. It runs the parts of a plan it has native implementations for, and Spark handles the rest. How much of your workload lands in each bucket is the whole story.

How Comet accelerates your existing Spark jobs

Comet’s speedup comes from two design choices: a native columnar engine for the operators it supports (DataFusion operators), and a fallback path for everything else. Understanding both is how you predict what Comet will do for a given job.

Vectorized native execution on the DataFusion engine

Spark’s default execution runs on the JVM and processes data a row at a time through generated Java code. Comet replaces that path for supported operators with DataFusion’s vectorized engine, which processes data in columnar batches and uses the CPU more efficiently on scans, filters, and aggregations.

On the workloads it covers, the gains are real. Comet’s own documentation reports a meaningful speedup over stock Spark on TPC-DS at 1 TB, and an independent AWS Labs benchmark on a 3 TB TPC-DS workload put Comet in the same performance class as Apache Gluten™. These are specific benchmarks on standard datasets, so your own jobs may land lower under different conditions. For example, DataFusion’s blog describes the 3 TB setup: 23 executors with 58 GB of RAM each, about 1.3 TB of RAM in total, processing roughly 600 GB–1 TB of Apache Parquet™ data after compression. A memory-to-data ratio close to 1:1 lets Comet keep most of the work in memory, and production Spark clusters usually run with less memory per terabyte of data, so the same speedup may not hold on your cluster.

The speedup is largest on CPU-bound, scan-heavy, aggregation-heavy work. That is the shape of a lot of ETL, which is why Comet is worth evaluating in the first place.

When an operator isn’t supported, the plan falls back to Spark

Comet does not implement every Spark operator and expression. When it hits something it cannot run natively, it falls back to Spark for that part of the plan. The extension detects unsupported features automatically, so nothing breaks, but the unsupported work runs at ordinary Spark speed.

Comet softens this with a codegen dispatcher, a feature Gluten does not have at this time. For expressions that are only partially compatible with Spark, such as some regex, JSON, and datetime functions, Comet runs Spark’s own generated code inside the native pipeline on Arrow data. That keeps results byte-exact with Spark and stops a single awkward expression from forcing an entire stage back onto the JVM.

The dispatcher also lets Comet run existing Scala and Java UDFs inside the native pipeline without a rewrite, evaluating the UDF on the JVM while the surrounding operators stay native. It is a genuine strength, and it narrows the fallback surface. It does not eliminate it.

Where Comet’s native acceleration starts and stops

The practical map of Comet is a split: a set of operators that run natively and a set that stays on Spark. The table below is the short version, and the two subsections underneath explain each side.

Runs natively on Comet	Stays on Spark
Parquet scans and native Apache Iceberg™ scans	Iceberg, Delta, and Apache Hudi™ writes
Filters, projections, and sorts	Delta Lake, Hudi, and Apache Paimon™ table reads
Hash and sort-merge aggregations	Expressions with no native implementation
Joins, including sort-merge joins	Features disabled by default for compatibility
Comet shuffle via its shuffle manager	Anything the plan marks unsupported

Coverage reflects Comet 1.x on Spark 3.4 through 4.1; the exact list moves every release. Check the Comet compatibility guide for the current matrix before you rely on any row.

Runs natively: scans, joins, and aggregations

The read path is Comet’s strongest ground. It accelerates Parquet scans and ships a native Iceberg scan built on iceberg-rust that supports spec v1 and v2, schema evolution, time travel, and positional and equality deletes on merge-on-read tables. For Iceberg-on-Parquet lakehouses, a large fraction of a scan-heavy plan can run natively.

Compute operators follow. Filters, projections, sorts, hash and sort-merge aggregations, and joins all have native implementations, and Comet provides its own columnar shuffle manager so shuffles do not automatically drop back to the JVM.

Spark 4.0 turns on ANSI SQL semantics by default, and Comet implements ANSI behavior for the expressions it supports natively. That matters because on a competing accelerator, ANSI mode can push large parts of a query back to Spark.

Stays on Spark: writes, unsupported expressions, and non-Iceberg formats

The write path is the clearest gap. Comet accelerates reads, but Iceberg writes still go through Spark, and Comet does not currently provide native integrations for Delta Lake, Hudi, or Paimon. For upsert-heavy or MERGE-heavy pipelines, the expensive half of the job is the write, and that half does not get the native speedup.

Table format is the second gap. If your lakehouse is on Delta or Hudi rather than Iceberg, Comet has no native scan path for those formats today, so more of the plan runs on Spark. Apache Gluten, the other Velox-based accelerator, ships dedicated modules for Delta, Hudi, and Paimon, so format coverage is one place the two projects genuinely differ.

Expressions are the long tail. Anything without a native implementation falls back, and features with known compatibility differences are disabled by default and only run natively if you opt in. The Comet configuration reference runs to hundreds of settings, and an independent walkthrough describes the tuning and compatibility surface as a real, ongoing time cost.

How to tell what actually ran natively

Because fallback is silent, a job can be “on Comet” while most of its runtime is still Spark. The number that matters is the share of your plan that executed natively, and you have to read it rather than assume it.

Turn on Comet’s fallback explanation setting and inspect the query plan. Comet annotates which operators ran natively and which fell back, and reading that output on your real jobs is the only reliable way to know your native coverage. A plan that looks accelerated at the top can still hand its most expensive stage to Spark.

Do this before you draw any cost conclusion. A workload that runs 80% natively and one that runs 20% natively both count as “using Comet,” and they produce very different bills.

Comet vs Gluten: two takes on the same idea

Comet and Gluten solve the same problem with a similar architecture: both are Spark plugins that serialize the physical plan and run it on a native engine. The main split is the engine underneath. Comet uses DataFusion in Rust, while Gluten most commonly uses Velox in C++.

The language choice has real consequences. Rust gives Comet memory safety without a garbage collector, which reduces a class of crashes for code that runs inside every executor. Gluten, on Velox, currently covers more table formats natively and, per the AWS Labs benchmark, finished roughly 9% faster than Comet across the full TPC-DS query set, with wide per-query variation in both directions.

Neither result settles the choice for your workload. The Comet and Gluten comparison, maintained by the Comet community, is a fair starting point, and the honest recommendation on both sides is to benchmark both on your own jobs.

The architectural ceiling every compute-stage accelerator shares

Comet and Gluten are both compute-stage accelerators. They vectorize execution, and they do it well, but they optimize the part of the job that runs after data is read and before it is written. The plan they receive is still Spark’s plan, and the storage layer is still Spark’s job.

That ceiling explains the gaps you have already seen. Writes fall back because native accelerators optimize the compute stage, not the write path. Non-Iceberg formats fall back for the same reason. There is no storage-aware planning that reads table metadata to reshape the query, and no index-aware join that locates rows instead of scanning for them, because those optimizations live below the layer these accelerators operate on.

For scan-and-aggregate ETL on Apache Iceberg, that ceiling may be high enough. For upsert-heavy pipelines, compaction, and MERGE, the work that a compute-stage accelerator cannot touch is often the work that costs the most.

Where Quanton fits for teams outgrowing Comet

Free open-source accelerators exist and they work, and the free tier of Quanton ships Comet and Gluten pre-installed so a side-by-side run is easy to set up. The question is not whether Comet is worth trying. It is what to do when the fallback share stays high and the write path stays slow.

Full-stack optimization past the compute stage

The Quanton engine is also a drop-in replacement for Spark’s execution engine, built on an optimized fork of Velox. Where compute-stage accelerators speed up execution alone, Quanton also optimizes the read and the write. It also does storage-aware planning that reads Iceberg and Hudi metadata to reshape the compute-stage plans, index-aware joins, and native low-shuffle MERGE and compaction that run columnar instead of falling back to Spark.

That is where the extra speed comes from on the workloads Comet leaves on the JVM. On TPC-DS, Quanton runs about 6x faster than open-source Spark and 25% faster than Databricks Photon on identical hardware, with roughly 4x faster compaction than Spark. These figures move, so treat them as a checkpoint and confirm the current numbers in the Quanton benchmark results.

The maturity point is the fair counter to a free accelerator, made on evidence rather than insult. Comet and Gluten still carry known compatibility limits disabled by default and documented gaps in their own trackers, and both can OOM on complex queries once workloads push past simple ETL. The free tier is there to run the comparison and settle it on your own numbers.

Capability	Comet	Gluten	Quanton
Native engine	DataFusion (Rust)	Velox (C++)	Optimized Velox fork (C++)
Compute-stage vectorization	Yes	Yes	Yes
Storage-aware planning on table metadata	No	No	Yes
Native low-shuffle MERGE and compaction	No	No	Yes
Index-aware joins	No	No	Yes
Formats accelerated	Iceberg scans	Iceberg, Delta, Hudi, Paimon	Iceberg, Hudi, Delta
Licensing	Apache 2.0	Apache 2.0	Open core

Simplified, and capabilities move each release. Confirm Comet and Gluten against their own docs and Quanton against its published benchmarks before relying on any row.

Open core, a free tier, and a one-line path back

The reason to look past a free accelerator is usually not the license, so Quanton answers the license question up front. It is open core, ships a free tier for the first 100 GiB processed each month, and reverses to open-source Spark with a single config line. There is no lock-in to weigh against the speedup.

The cost model is the other half. Quanton bills per GiB of data processed rather than per compute hour, and it runs inside your own cloud account, so your reserved and spot discounts stay yours. Because billing is per GiB, every speedup lowers your cloud bill rather than a vendor’s. You can model the whole thing as one line item on the Quanton pricing page.

Should you run Comet?

Try it. Comet is free, open source, and legitimately fast on scan-and-aggregate ETL over Iceberg, and the codegen dispatcher keeps more of a plan native than a strict fallback model would. Turn on the fallback explanation, read your query plans, and measure the native share on your own jobs before you count on any savings.

If the fallback share stays high, if your lakehouse is on Delta or Hudi, or if the write path is where your cost lives, a compute-stage accelerator has reached its ceiling. Quanton optimizes the read and the write as well, bills per GiB so speedups cut your bill, and reverses to open-source Spark with one config line. You can run it against Comet and Gluten on the same jobs on the free tier: create an account and compare the numbers in your own VPC.

Frequently asked questions

Is Apache DataFusion Comet free and open source?

Yes. Comet is an Apache Software Foundation project licensed under Apache 2.0, and it runs on commodity hardware with no proprietary runtime required. You add it to an existing Spark deployment as a plugin, so there is no vendor license to buy.

Does Comet require code changes to my Spark jobs?

No. Comet is a drop-in plugin: you add a JAR and a set of Spark configs, and supported operators run on the native engine while everything else falls back to Spark. You should budget time for configuration and compatibility tuning, which is more involved than the one-line install suggests.

What does Comet not accelerate?

Comet accelerates reads and compute but leaves several things on Spark: Iceberg writes, tables in Delta, Hudi, or Paimon (which have no native scan path today), expressions without a native implementation, and features disabled by default for compatibility. Enable the fallback explanation and inspect the query plan to see the exact split on your workloads.

How does Comet compare to Gluten?

Both are Spark plugins that run the plan on a native engine, with Comet on DataFusion in Rust and Gluten on Velox in C++. Gluten covers more table formats natively and held a small lead in the AWS Labs TPC-DS benchmark, while Comet offers memory safety and a codegen dispatcher that reduces full fallbacks. Benchmark both on your own jobs.

Is Comet the same as Databricks Photon?

They share the concept of a native execution layer on top of Spark, but Photon is proprietary to the Databricks runtime, while Comet is open source and runs on any supported Spark deployment. Both accelerate the compute stage; neither optimizes the write path or reads table metadata to reshape the plan. Databricks Photon (like Quanton) is significantly faster than Comet on TPC-DS at 10 TB scale, at this time.