Comet 1.2.0 is the first release that can read Delta tables natively. The reader is already inside the comet-spark jar on Maven Central, but the small contrib jar that switches it on isn't published yet, so this is a step-by-step guide to building it in Docker, running it on Spark 4.1, and reading the plan to confirm the scan really went native.
Read more
The native Delta Lake read path for DataFusion Comet took 58 days to merge. Here is how the PR grew from 3.9K to 21.5K lines, shrank once along the way, what review and testing uncovered, and the side fixes it produced.
Read more
Databricks' parquet writer emits no column indexes, and none of its four reader configurations evaluates them on files that carry them. Measured across three layouts and six readers at 1 TB, on identical 224-core fleets, down to the parquet footers. Liquid clustering turns out to be the workaround: a kd-tree whose leaves are the file boundaries, because file min/max is the only statistic the platform can act on. The open reader stack beats its best case on every query family at a tenth of the bytes.
Read more
DataFusion Comet declines every Delta table over a one-line format check, giving up native execution and all parquet pruning. PR #5365 adds a native Delta read path that inherits row-group and page-index pruning for free, and applies deletion vectors inside the scan itself. Measured at 1 TB: the Delta arm is indistinguishable from raw parquet.
Read more
A benchmark that made no sense led to a four-environment A/B test proving the AWS Spark runtime's vectorized reader skips row groups but never pages, plus a 120-line MWE anyone can run to verify it.
Read more