Databricks Never Skips a Page. Its Writer Does Not Even Write Them

Two posts ago I found that EMR and Glue never read parquet column indexes. Last post I gave DataFusion Comet a native Delta read path so the fastest reader in that investigation could prune the format I actually run. One vendor was missing from the scoreboard: Databricks. So I created a workspace, pointed the same instrumentation at it, and reran the series benchmark with the platform’s own clustering in the ring: classic OPTIMIZE ZORDER BY and liquid clustering, against the learned Hilbert curve from the earlier posts.

The outcome is cleaner than the EMR story, and stranger. EMR’s defect was a reader ignoring page statistics the writer dutifully produced. Databricks falls short on both sides of the contract at once, then ships the workaround as a flagship feature.

Names, Precisely

Every party here ships a Hilbert-family curve, so the labels collide. “learned-hilbert” is MY table, produced by my Spark extension (research code, not yet released): the ordering logic is mine, the files themselves are written by stock Spark’s parquet writer, column indexes included. “dbr-zorder” is the table Databricks’ classic OPTIMIZE ZORDER BY wrote: branded Z-order, but implemented with a Hilbert curve in their runtime per Databricks’ own engineering material (true Z-order interleaving lives in the OSS Delta lineage). “dbr-liquid” is the CLUSTER BY table; public material calls it Hilbert-based while the patent behind it describes kd-epsilon trees. I measure the files the closed implementations produced, not the branding.

So the core matchup is Hilbert against Hilbert: theirs written without page indexes for a platform that skips at file boundaries, mine written with them for readers that descend inside. Four platform arms carry the headline: dbr-zorder and dbr-liquid on Databricks’ engines, learned-hilbert on Comet, learned-hilbert on stock Spark 4.2. Every other combination in this post (Databricks reading my table, open readers reading theirs) exists as diagnostic evidence for why those four differ.

How Everything Was Measured

These are adversarial claims about a commercial product, so the instrumentation deserves more than a footnote. Nothing below relies on timing heuristics, sampling, or estimates. Every number comes from the engine’s own executed-plan metrics, read through py4j after each query completes, and streamed to S3 one JSON record at a time.

Rows scanned. For stock Spark scan nodes, the scan’s numOutputRows counts rows the reader decoded, before the filter above it runs. Photon fuses the filter into its scan node, so its numOutputRows is post-predicate; the decoded count is numOutputRows + numRowsFilteredByDataFilters. I verified the composite two ways: on the same table, Photon and the standard engine agree to the fourth decimal, and Photon’s filesSize (bytes read) tracks the composite, not the raw output count. Comet reports DataFusion’s output_rows. Scan fraction is rows decoded divided by table rows.

Bytes and files. Photon exposes filesSize and numFiles; Comet exposes bytes_scanned; stock JVM scans publish file counts but no byte metric. Where a table lacks sub-file structure, Comet’s measured bytes on the identical read stand in as the transfer proxy for OSS, and the tables mark derived values explicitly.

Page skipping, three independent instruments:

1. Files that provably carry indexes. My table’s footers show column_index=True, offset_index=True on every dimension (pyarrow, 10 of 10 sampled files). Stock Spark 4.2 and Comet prune it to 0.0059 at 1 TB. Any reader scoring far above that on identical bytes is not evaluating the indexes. All four Databricks configurations score 0.0971.
2. A discriminating workload. The verdict test from the EMR post: ten staggered 5%-wide windows on the sort column of a time-sorted table. An index-evaluating reader lands near 0.05 on this table; a reader limited to row-group statistics lands near 0.28. Every reader in the study ran it on the same physical table.
3. The engine’s own telemetry. Photon publishes page-statistics counters (numRowsSkippedWithStats, pageCacheIndexRowsSkipped, a family of PageStats byte counters). Across every run they read zero, while its file-pruning counters work normally.

Cross-validation. Six reader configurations ran the same twenty queries per family on byte-identical files: DBR 18 Photon, DBR 18 standard, DBR 19 Photon, DBR 19 standard, stock Spark 4.2 (Delta 4.2), and my Comet fork. Wherever a layout offers only file-level structure, all six agree to the fourth decimal. They diverge exactly and only where sub-file structure exists to exploit. That convergence is the strongest defense against “your measurement code is wrong”: independent implementations, two of them not mine, reproduce each other perfectly, and the split follows the footers.

The statistics objections, answered in advance. Delta’s file-skipping statistics are collected automatically at write time (first 32 columns; these tables have 23). I verified them in the transaction logs directly: the verdict table carries min/max on 304 of 304 file entries including the sort column, and both Databricks-written tables carry them on every clustering column. They demonstrably functioned, driving all the file-level pruning recorded in the same runs. Catalog statistics from ANALYZE TABLE are a different layer: none of these external tables have them, Databricks’ auto-ANALYZE (predictive optimization) covers managed tables only, and per Databricks’ own guidance ANALYZE belongs in the nightly maintenance job beside OPTIMIZE and VACUUM. It feeds the query optimizer’s cardinality estimates, not the scan’s page reads. And the verdict experiment closes the question by construction: one physical table, one statistics state, and the readers still diverge 0.049 against 0.282. The managed-table escape dies on the writer: background OPTIMIZE runs through the same writer that emits no page indexes.

Everything ran on default configurations, dedicated-mode job clusters, S3-direct reads, no disk cache warmup. The probe scripts, per-query JSONL evidence, and footer dumps are committed in the campaign repo.

The Hardware

All Graviton (m7g), all ap-southeast-1, all reading the same S3 bucket. Every latency in the main tables comes from the 224-core shape; scan fractions are hardware-independent.

Building the Layouts: Cost to Optimize

Identical 1 TB source (2.75 B rows, 111.9 GB zstd baseline), identical three clustering columns (ss_store_sk, ss_item_sk, ss_sales_price). Month partitioning kept where the method allows it (liquid forbids partitioning).

Liquid is the cheapest build in compute by roughly 2.3×, because the clustering rides the initial write and the follow-up OPTIMIZE FULL finds nearly nothing to do. It pays in storage: +63% over baseline against the Hilbert table’s +32% (codec differs, snappy vs zstd, noted throughout).

What the Files Contain: the Writer Claim

At 100 GB the first Databricks result set the tone: Photon answered every query on its own freshly z-ordered table by reading 100% of it, all families, every time. Cross-reads killed the blind-reader theory immediately: stock Spark 4.2 and Comet, readers that prune my table two orders of magnitude deeper, scanned that same table end to end as well. Nobody can prune inside these files, because there is nothing to prune with. The footers say why, and the sample is now proper:

dbr-zorder  (1 TB): 10/10 sampled files  column_index=False  offset_index=False
dbr-liquid  (1 TB): 10/10 sampled files  column_index=False  offset_index=False
learned-hilbert:    10/10 sampled files  column_index=True   offset_index=True

Databricks’ parquet writer omits column and offset indexes entirely, the page-level min/max structures the format has carried since 2018, and it writes multi-million-row row groups. Both of its write paths behave the same (the OPTIMIZE rewrite and the CLUSTER BY ingest). I found no documented configuration that changes this, and Databricks’ Photon writer documentation makes no mention of page-index emission.

The constructive half: my table gets its indexes for free, and not from anything proprietary. My extension only decides row order; the bytes are written by stock Spark’s parquet-mr writer, which has emitted ColumnIndex and OffsetIndex by default since parquet-mr 1.11, in 2018. The OPTIMIZE ... USING MIXER 'hilbert' rewrite is an ordinary Spark write, so every file carries per-page min/max at roughly megabyte granularity. OSS Spark evaluates those structures in its vectorized reader; Comet does through DataFusion’s page-index pruning. Databricks removed a seven-year-old open-format default and sells the workaround. I kept the default and taught the fastest readers to use it on Delta.

The Four Platform Arms at 1 TB

Fractions and bytes first. DBR 18 and DBR 19 measure identical to the fourth decimal in every cell, on both engines, so they share rows here; the latency table below keeps all four separate.

*Stock JVM scans expose no byte metric; parenthesized values are Comet’s measured bytes on the identical reads.

The open arms win every family on both fractions and bytes. Liquid, the platform’s best case, reads 11× the bytes of the Comet arm on boxes and roughly 16× on the single-dimension families. Zorder trails everything.

Warm latency (median of 3), every row on the 224-core shape:

Honest reading, both directions. On matched hardware at 1 TB, liquid plus Photon is the latency winner on every family: its four fat kd-leaf files stream contiguously at fleet bandwidth, while my table’s page-precise reads scatter small slices across ~500 files and pay S3 request overhead. Photon itself contributes a consistent 3–17% over the standard engine at identical fractions: it is not a different reader, it is a faster consumer of the same reads. Meanwhile the byte columns do not move: the platform’s fastest configuration transfers an order of magnitude more data to be half a second faster, and even reading Databricks’ own liquid table, Comet moves 14× fewer bytes than Photon at near-parity latency (1.9 vs 27 GB on price_only, 793 vs 602 ms).

One sidebar, deliberately outside the main comparison, because it taught me something about cluster sizing. I reran two of these arms on a single 32-core node. Reading the liquid table’s boxes queries there took 377 ms, faster than the entire 224-core Photon fleet took on the same bytes (516 ms): the query touches only four files, read start to finish, and a big cluster adds scheduling overhead to a job that small without adding any useful parallelism. My table showed the opposite pattern. Its single-dimension queries fetch thousands of small page ranges scattered across ~500 files, and on one node those requests queue up: 2.3 to 2.6 seconds, against 1.7 to 1.9 seconds when the fleet spreads them across 224 cores. The rule of thumb that falls out: a few large contiguous reads do not benefit from a fleet, while many small scattered reads are exactly what a fleet is for. Pick cluster size by read pattern, not by data size.

Cross-Read Diagnostics: Every Reader, Every Layout

Fractions on byte-identical files, all six configurations:

This table is the whole argument in one grid. On Databricks-written files, six independent readers agree to the fourth decimal: the layouts carry file-level structure only, and everyone can use exactly that much. On my files, the readers split cleanly along one line: the two that evaluate page indexes descend 16× deeper than the four Databricks configurations, which sit at the file-pruning bound. Layout capability and reader capability, separated by construction.

The Verdict Test: Page Skipping, Same Table, Six Readers

Ten staggered 5%-wide windows on the sort column of the time-sorted baseline, the single most page-skippable workload in the series, all readers against one physical table:

*Stock JVM scans expose no byte metric; the parenthesized value is Comet’s measured bytes on the identical read.

Every reader touches the same 71.9 files, so file-level pruning is identical across the board. Inside those files the split is 63×: the index-evaluating readers fetch 0.05 GB of matching pages while all four Databricks configurations decode 3.15 GB of row groups.

Four Databricks configurations, one number, and it is the row-group-bound signature from the EMR conviction. The index-evaluating readers agree with each other to the fourth decimal at 5.7× less data. DBR 19 bundles Spark 4.2, the release whose reader ended the honesty fork in the first post; Databricks replaced that reader with its own long ago, so the upgrade never arrives. Waiting for the next runtime is not a remedy.

Liquid Clustering Is the Workaround, and It Is Good

Once you know the platform skips only at file boundaries, liquid clustering stops resembling a clustering algorithm and starts resembling what it is: a layout-side patch for a reader that cannot see below the file. Its kd-tree writes each leaf as a file, an axis-aligned box whose min/max statistics discriminate almost surgically. Boxes queries touch 4 of 512 files. Within its world it is well engineered: cheapest build, contiguous reads, the platform’s best numbers of the whole campaign.

The ceiling is structural. Those four files get read whole, because nothing inside them is indexed. Liquid trails Hilbert-plus-indexes on every family at every scale measured, transfers 11–16× the bytes, and carries double the storage premium. The open stack prunes at file granularity and then keeps pruning inside.

What Each Run Cost

Per-run economics at 1 TB, from measured durations and exact ap-southeast-1 on-demand rates (fleet EC2 plus EBS: $12.49/hr for the 224-core shape, $7.27/hr for the 128-core build fleet). DBU dollars use the $0.15/DBU premium jobs list rate with a bracketed emission estimate [fleet ≈ 37 DBU/hr standard, ≈ 74 DBU/hr Photon]; these cells will be replaced with billed actuals from the account usage export.

The Hilbert build is the cheapest layout in dollars despite the slowest wall clock: fewer nodes and no per-DBU charge. On the query side the DBR arms pay roughly double the open fleet’s hourly rate for the same hardware, before counting the redundant bytes.

Will the Fast One Be Overtaken?

Liquid buys latency with bandwidth. Whether that trade survives scale is computable from the measured growth between 100 GB and 1 TB (bytes per decade of data) and the measured fleet throughput (Photon sustains 50–70 GB/s against S3 on 224 cores in the full-scan runs):

The broad families exhaust the fleet’s transfer budget around the 10 TB mark on this hardware class; ultra-selective boxes stays cheap to stream at any projected scale. On bytes there is no crossover to wait for: the 10–16× gap is scale-stable because the growth exponents nearly match, so every cost that scales with data moved (transfer, concurrency, cores, DBUs) favors the index-reading stack today and by the same ratio tomorrow.

The Scoreboard, Updated

Where This Goes Next

The kd-tree-at-file-boundaries idea and the curve-with-page-indexes idea are not rivals. One optimizes request geometry, the other minimizes bytes. A writer that emitted curve-ordered, stats-tight files while keeping its column indexes would take both columns of that scoreboard at once. That is roughly the direction the learned-curves project’s sfc-tree connector is exploring, and it now owns a measured target: 0.0083 and four files, set by the platform that cannot read a page.