Databricks Never Skips a Page. Its Writer Does Not Even Write Them
Databricks' parquet writer emits no column indexes, and none of its four reader configurations evaluates them on files that carry them. Measured across three layouts and six readers at 1 TB, on identical 224-core fleets, down to the parquet footers. Liquid clustering turns out to be the workaround: a kd-tree whose leaves are the file boundaries, because file min/max is the only statistic the platform can act on. The open reader stack beats its best case on every query family at a tenth of the bytes.
Read more