Data Engineering

Aug 19, 2026

Databricks Never Skips a Page. Its Writer Does Not Even Write Them

Databricks' parquet writer emits no column indexes, and none of its four reader configurations evaluates them on files that carry them. Measured across three layouts and six readers at 1 TB, on identical 224-core fleets, down to the parquet footers. Liquid clustering turns out to be the workaround: a kd-tree whose leaves are the file boundaries, because file min/max is the only statistic the platform can act on. The open reader stack beats its best case on every query family at a tenth of the bytes.

databricks photon liquid clustering apache spark delta lake parquet datafusion comet data engineering performance

Aug 17, 2026

Comet Falls Back to Spark on Every Delta Table. I Wrote the PR That Fixes It

DataFusion Comet declines every Delta table over a one-line format check, giving up native execution and all parquet pruning. PR #5365 adds a native Delta read path that inherits row-group and page-index pruning for free, and applies deletion vectors inside the scan itself. Measured at 1 TB: the Delta arm is indistinguishable from raw parquet.

apache spark datafusion comet delta lake parquet rust data engineering performance open source

Aug 10, 2026

Your Parquet Column Indexes Are Being Ignored on EMR and Glue

A benchmark that made no sense led to a four-environment A/B test proving the AWS Spark runtime's vectorized reader skips row groups but never pages, plus a 120-line MWE anyone can run to verify it.

apache spark parquet emr aws glue data engineering performance delta lake benchmarking

Feb 24, 2026

Interlock: A STAMP-Based Safety Framework for Data Pipelines

How I built a STAMP-based safety framework in Go with declarative sensors, failure classification, and centralized observability for data pipeline reliability on AWS

data engineering go aws safety dynamodb step functions terraform eventbridge observability

Feb 7, 2026

PySpark Pipeline Framework: Configuration-Driven Pipelines for the Python Ecosystem

How pyspark-pipeline-framework brings configuration-driven architecture, lifecycle hooks, and resilience patterns to PySpark

python pyspark data engineering open source configuration streaming

Dec 23, 2025

Fitting 100 Statistical Distributions at Scale: 1000x Memory Reduction with PySpark

How spark-bestfit 3.0 fits distributions across Spark, Ray, and local backends with survival analysis, mixture models, and multivariate support

spark python data engineering data science statistics optimization ray distributed computing survival analysis

Dec 23, 2025

Building Production-Ready Spark Pipelines with Configuration-Driven Architecture

How spark-pipeline-framework reached 1.0 with Spark Connect support, streaming, and enterprise features

spark scala data engineering open source observability spark connect streaming

Jul 28, 2021

Data Optimization for Compacted Partitions: Achieving 77% Storage Reduction

How intelligent data optimization with linear ordering and Z-ordering achieved 77% storage reduction and 90% runtime improvements on petabyte-scale data lakes.

apache spark data engineering big data optimization parquet orc