Open Source

Preview for 58 Days to Merge: Getting Native Delta Lake Into DataFusion Comet

58 Days to Merge: Getting Native Delta Lake Into DataFusion Comet

The native Delta Lake read path for DataFusion Comet took 58 days to merge. Here is how the PR grew from 3.9K to 21.5K lines, shrank once along the way, what review and testing uncovered, and the side fixes it produced.

apache spark datafusion comet delta lake open source
Read more
Preview for Pay Once, Not Per Task: Four Repeated Costs Comet 1.1.0 Stopped Paying

Pay Once, Not Per Task: Four Repeated Costs Comet 1.1.0 Stopped Paying

Apache DataFusion Comet 1.1.0 is out, with 14 of my PRs in it. Four of them fix the same kind of waste: work that gives the same answer every time but was being redone for every task, batch or shuffle block. Here is what each one changed, what the benchmarks really show, and the correctness fix I'd point to first.

apache spark datafusion comet rust scala data engineering performance open source
Read more
Preview for Comet Falls Back to Spark on Every Delta Table. I Wrote the PR That Fixes It

Comet Falls Back to Spark on Every Delta Table. I Wrote the PR That Fixes It

DataFusion Comet declines every Delta table over a one-line format check, giving up native execution and all parquet pruning. PR #5365 adds a native Delta read path that inherits row-group and page-index pruning for free, and applies deletion vectors inside the scan itself. Measured at 1 TB: the Delta arm is indistinguishable from raw parquet.

apache spark datafusion comet delta lake parquet rust data engineering performance open source
Read more
Preview for PySpark Pipeline Framework: Configuration-Driven Pipelines for the Python Ecosystem

PySpark Pipeline Framework: Configuration-Driven Pipelines for the Python Ecosystem

How pyspark-pipeline-framework brings configuration-driven architecture, lifecycle hooks, and resilience patterns to PySpark

python pyspark data engineering open source configuration streaming
Read more
Preview for Contributing to Gastown: Multi-Agent Orchestration for Claude Code

Contributing to Gastown: Multi-Agent Orchestration for Claude Code

5 merged PRs and 8 open contributions to Gastown, covering daemon resilience, fresh installation fixes, and autonomous patrol improvements

claude code multi-agent open source gastown llm ai agents
Read more
Preview for Building Production-Ready Spark Pipelines with Configuration-Driven Architecture

Building Production-Ready Spark Pipelines with Configuration-Driven Architecture

How spark-pipeline-framework reached 1.0 with Spark Connect support, streaming, and enterprise features

spark scala data engineering open source observability spark connect streaming
Read more