Comet Falls Back to Spark on Every Delta Table. I Wrote the PR That Fixes It

My last post ended with a scoreboard: EMR and Glue page-blind, stock Iceberg page-blind, and two working exits, stock Spark 4.2 for parquet and Delta, Comet 1.0.0 for parquet and Iceberg. There’s a hole in that scoreboard. Comet, the fastest reader in the entire investigation, covers parquet and Iceberg. The format I actually run, Delta, isn’t on its list.

The reason is one line of code. Comet’s isFileFormatSupported check requires exactly ParquetFileFormat, and Delta’s DeltaParquetFileFormat is a subclass. So Comet declines every Delta table and hands the query back to Spark. No native execution, no row-group pruning, no page skipping, on what is probably the most widely deployed table format in the Spark ecosystem. And a Delta table is just parquet files with a transaction log on top. The 3.4x reader was sitting right there.

So I wrote the read path: apache/datafusion-comet#5365. Most of it turned out to be smaller than it looks, for one reason worth explaining, and the part that wasn’t small is deletion vectors.

The Observation That Deletes Most of the Work

The obvious architecture is the heavy one: embed a Delta implementation (delta-kernel-rs) on the native side, let it replay the transaction log, resolve the snapshot, plan the files. An earlier contrib PR (#4366) built exactly that, and working through it taught me why I didn’t want it. The kernel’s parquet reader has no page-index support, which is the very thing I came for, and its planning repeats work that has already happened by the time the query gets anywhere near Comet.

Because by the time Comet’s scan rule sees a Delta query, delta-spark has already done all the Delta work. Log replay, snapshot resolution, time travel, partition pruning, all of it is finished before the FileSourceScanExec lands in transformV1Scan. What arrives is a list of concrete parquet files with pushed-down filters. That isn’t a Delta scan anymore. It’s a parquet scan wearing a Delta badge.

Which means the native side needs zero Delta planning. The scan routes through the same DataFusion ParquetSource path Comet already uses for plain parquet, and inherits the full pruning stack for free: row-group statistics, page indexes (#5142), and native filter pushdown (#4722, the same change whose flipped default cured Comet’s page-blindness in 1.0.0). A second, minimal PR (#4669) had spotted this earlier for plain tables; #5365 takes that insight and extends it across the full feature matrix. Both prior authors are credited as co-authors, since the contrib module shape and test catalog come from one effort and the core idea from the other.

The Part That Had to Be Written From Scratch: Deletion Vectors

The one thing delta-spark hasn’t already resolved when the plan reaches Comet is deletion vectors. DV-enabled tables mark deleted rows in roaring bitmaps instead of rewriting files, and stock Spark applies them as a filter after the scan. The native path can do better than that.

delta_dv.rs decodes each file’s DV blob (CRC-verified unframing, then roaring decode in both wire layouts delta-spark writes) and turns it into a per-file ParquetAccessPlan, DataFusion’s row-level plan for a parquet file. The core is a single sweep over the sorted deleted-row indexes:

1
2
3
4
5
6
7
8
9

/// Translate deleted row indexes into a [`ParquetAccessPlan`]: fully-deleted
/// row groups become `Skip`, untouched groups stay `Scan`, and partially
/// deleted groups get a `RowSelection` selecting the complement of the deleted
/// rows. Page-index pruning later INTERSECTS with these selections, so DV
/// skips and page skips compose.
pub fn build_access_plan(
    row_group_row_counts: &[i64],
    deleted: &RoaringTreemap,
) -> Result<ParquetAccessPlan, ExecutionError> {

That INTERSECTS is doing the heavy lifting. Deleted rows get dropped inside the scan itself, and the two skip mechanisms cooperate rather than fight: a page consisting entirely of deleted rows is never read at all, and a surviving page still gets its predicate pruning. One landmine for anyone following this trail: DataFusion looks the plan up by concrete type in the file’s extensions map, so storing it as Arc<ParquetAccessPlan> instead of ParquetAccessPlan silently disables DV application. The comment in the source exists because I lost an afternoon to it.

What It Covers, What It Declines

The support matrix, because a reader that quietly gets deletion vectors wrong is worse than no reader:

The right column matters as much as the left. Anything the native path can’t prove it handles falls back to Spark’s reader, visibly, with a tag in EXPLAIN, rather than returning wrong rows. That philosophy, along with the decline-gate scenario catalog, is inherited straight from the kernel-based attempt.

Spark coverage: 3.5 / 4.0 / 4.1 against Delta 3.3.2 / 4.0.1 / 4.3.1. Spark 3.4 declines outright, and the spark-4.2 profile sits dormant until Delta ships a compatible release. Delta 4.1.0 and 4.2.0 are binary-incompatible with Spark 4.1.3, which is its own small saga.

The Numbers

Local benchmark first, 20M rows, selective predicate: 1.44x faster than stock Spark at 9.4% of the bytes read, with DV tables running at time parity with plain ones thanks to the in-scan bitmap application.

Then the terabyte. Same methodology as the last post, and one design choice that makes the three-way comparison unusually clean: all three arms read byte-identical parquet files. The 1 TB Hilbert-clustered store_sales table (2.75B rows, on S3) is a Delta table; its Iceberg twin was registered over the very same data files with add_files, no rewrite; and the parquet arm globs those files directly, transaction log ignored. Whatever differs between columns below is the table-format scan path, nothing else. Spark 3.5.6 standalone, 252 executor cores of Graviton, the branch built with -Pspark-3.5,delta. Four selective query families, twenty queries each; fractions come from executed-plan scan metrics, so they are hardware-independent; the control is the same session with Comet off and the vectorized reader off, the row reader that honors page indexes. Fraction of the table’s rows decoded (native · control), then the native arm’s bytes per query:

Warm stopwatches, native vs the row-reader control on the same fleet:

The headline is the Delta column: it is the parquet column. Same fraction, same bytes, on every family. The design claim of the PR, that a claimed Delta scan routes through the identical ParquetSource path and inherits the identical pruning, is not approximately true at 1 TB, it is true to every measured digit. The table format costs nothing on the read path. Two supporting observations: the Iceberg arm reproduces my Comet 1.0.0 rerun of the same suite to the fourth decimal (0.00091 on boxes, across different builds and a different cluster shape), and its row fractions are not comparable to the other arms because CometIcebergNativeScanExec counts rows after filter pushdown, which is why the bytes column exists (the Iceberg arm actually reads 2 to 3x the bytes of the parquet and Delta arms here).

Now the estimates, labeled as such. Against the strongest stock opponent, Spark 4.2 with AWS’s open-sourced analytics stream, the reader the last post crowned: the campaign measured its vectorized Delta reads at 0.81s to 2.1s on 128 cores for these same families, and scaled to this fleet that lands at rough parity with the native path, the stock reader probably a shade faster on the tiny boxes query where Comet’s JNI overhead shows, the native path perhaps 1.2 to 1.3x ahead where decode volume grows. These families are the least favorable case for Comet: highly selective literal predicates that an honest vectorized reader already prunes to the floor. The differentiated wins live elsewhere. Scalar-subquery bounds, which stock Spark 3.x cannot push into the reader at all (it scans 1.000 of the table; the native path pushes them at execution time and scanned 5% in the local bench, a 3.35x win that grows with table size). Deletion-vector tables, where the bitmap is applied inside the scan instead of as a post-scan filter. And compute-heavy plans, where the scan feeds native execution instead of handing rows back to the JVM. Against EMR’s still-page-blind reader the estimate is less charitable: on this layout, page-blind pruning decodes 3 to 16x the rows (measured on these exact files via the equally blind stock Iceberg reader: 0.0975 vs 0.00593 on boxes), which works out to roughly 2x the time and, with EMR’s 20% price premium, 2.5 to 3x the cost per query.

The claim that pruning actually fires isn’t a benchmark footnote, it’s a test assertion. The differential suite hard-checks page_index_rows_pruned > 0 and row_groups_pruned_statistics > 0 on the plans it runs. If a refactor ever silently breaks a tier, the suite goes red. Readers silently dropping a pruning tier is the entire subject of my last post, so this one ships with a tripwire.

Testing a Reader You Intend to Trust

Three layers, since differential testing is the only honest way to validate a reader replacement:

• A 36-test differential suite. Every scenario runs comet-on and comet-off and compares results: the DV matrix, column mapping, DPP, schema evolution, and each decline gate, green on all Spark 3.5/4.0/4.1 cells.
• Delta’s own test suites, with Comet injected. DeletionVectorsSuite 29/29. TimeTravel, ColumnMapping, DeleteSQL, UpdateSQL 197/197. MergeIntoSQLSuite 664/665, where the single failure is a scan-telemetry count assertion, a plan-shape artifact; its data assertions pass.
• A DML repro suite proving DELETE still writes deletion vectors rather than rewriting files under the new scan, in both useMetadataRowIndex modes. In other words, swapping the reader doesn’t quietly change writer behavior.

Plus Rust unit tests on the bitmap decode edge cases, because an off-by-one in a DV decoder is the kind of correctness bug you otherwise meet in production a year later.

Try It Before It Merges

The PR is open and under review, but nothing stops you from running it today: the branch builds like any Comet.

1
2
3
4

git clone -b feature/delta-native-scan https://github.com/dwsmith1983/datafusion-comet.git
cd datafusion-comet
# needs: JDK 17, Rust stable (1.88+), protoc 3.15+
make release PROFILES="-Pspark-3.5,delta"   # or -Pspark-4.0 / -Pspark-4.1

Two jars come out: spark/target/comet-spark-spark3.5_2.12-*.jar, which embeds the native library for whatever platform you built on, and contrib/delta/target/comet-contrib-delta-spark3.5_2.12-*.jar. Add both to your Comet confs from the last post, plus spark.comet.scan.delta.enabled=true.

Two deployment lessons, paid for so you don’t have to. First, build on an OS no newer than the machines that will run it: my Ubuntu 24.04 build refused to load on an older fleet image (GLIBC_2.38 not found) and Comet fell back to Spark silently, which is exactly the failure mode the last post taught me to gate against. Second, keep one classloader: on a standalone cluster where the comet jar rides spark.executor.extraClassPath (it must, CometShuffleManager loads at executor startup), the contrib jar and the delta-spark jars have to sit on that same classpath as real files. Ship delta-spark via --packages and it lands in Spark’s child classloader, where neither core’s ServiceLoader discovery nor the contrib’s Delta linking can see it. The result is not an error message you can search for, it is a NoClassDefFoundError in the middle of query planning.

Where This Lands

This closes out the scoreboard from the last post. Comet fronts parquet, Iceberg, and Delta with a page-index-aware native reader, and picking a fast reader no longer means giving up skipping on one format. The layout argument carries over too: Hilbert clustering pays out through the page tier on Delta under Comet, and running DELETEs against the table no longer costs you that payoff.

The change is apache/datafusion-comet#5365, open and under review as I write this, part of the broader Delta integration tracked in #174. The 1 TB numbers above were measured against the branch as submitted; if review reshapes the scan path I will re-run them, and when it merges this post gets the release number. It covers reads. Writes and CDF are someone’s future PR, possibly mine.

• The previous post: Your Parquet Column Indexes Are Being Ignored on EMR and Glue
• DataFusion Comet
• Delta Lake deletion vectors