Parquet

Aug 19, 2026

Databricks Never Skips a Page. Its Writer Does Not Even Write Them

Databricks' parquet writer emits no column indexes, and none of its four reader configurations evaluates them on files that carry them. Measured across three layouts and six readers at 1 TB, on identical 224-core fleets, down to the parquet footers. Liquid clustering turns out to be the workaround: a kd-tree whose leaves are the file boundaries, because file min/max is the only statistic the platform can act on. The open reader stack beats its best case on every query family at a tenth of the bytes.

databricks photon liquid clustering apache spark delta lake parquet datafusion comet data engineering performance

Aug 17, 2026

Comet Falls Back to Spark on Every Delta Table. I Wrote the PR That Fixes It

DataFusion Comet declines every Delta table over a one-line format check, giving up native execution and all parquet pruning. PR #5365 adds a native Delta read path that inherits row-group and page-index pruning for free, and applies deletion vectors inside the scan itself. Measured at 1 TB: the Delta arm is indistinguishable from raw parquet.

apache spark datafusion comet delta lake parquet rust data engineering performance open source

Aug 10, 2026

Your Parquet Column Indexes Are Being Ignored on EMR and Glue

A benchmark that made no sense led to a four-environment A/B test proving the AWS Spark runtime's vectorized reader skips row groups but never pages, plus a 120-line MWE anyone can run to verify it.

apache spark parquet emr aws glue data engineering performance delta lake benchmarking

Jul 28, 2021

Data Optimization for Compacted Partitions: Achieving 77% Storage Reduction

How intelligent data optimization with linear ordering and Z-ordering achieved 77% storage reduction and 90% runtime improvements on petabyte-scale data lakes.

apache spark data engineering big data optimization parquet orc