PySpark Pipeline Framework: Configuration-Driven Pipelines for the Python Ecosystem
How pyspark-pipeline-framework brings configuration-driven architecture, lifecycle hooks, and resilience patterns to PySpark
Read moreBlog
Technical articles on data engineering, Apache Spark, Python, MLOps, and site reliability engineering by Dustin Smith.
Hardening Gastown: Role-Based Access Control for Multi-Agent Workflows
Configuring Gastown for production use with custom role contexts, Claude Code hooks, git guards, and file guards to enforce principle of least privilege across AI agents
Read more
Contributing to Gastown: Multi-Agent Orchestration for Claude Code
5 merged PRs and 8 open contributions to Gastown, covering daemon resilience, fresh installation fixes, and autonomous patrol improvements
Read more
Fitting 100 Statistical Distributions at Scale: 1000x Memory Reduction with PySpark
How spark-bestfit 3.0 fits distributions across Spark, Ray, and local backends with survival analysis, mixture models, and multivariate support
Read more
Building Production-Ready Spark Pipelines with Configuration-Driven Architecture
How spark-pipeline-framework reached 1.0 with Spark Connect support, streaming, and enterprise features
Read more
Delivery Hero 2023 January Layoffs
My experience from Delivery Hero's 2023 layoffs.
Read more
Capital Budgeting with Monte Carlo Simulations in Python
How to use Monte Carlo simulations in Python to make better capital investment decisions, with a practical example of evaluating cloud migration costs.
Read more
Configuration Files in Python Using Dataclasses
How to use the dataconf library to parse HOCON, JSON, YAML, and properties files directly into Python dataclasses with full type safety.
Read more
Data Optimization for Compacted Partitions: Achieving 77% Storage Reduction
How intelligent data optimization with linear ordering and Z-ordering achieved 77% storage reduction and 90% runtime improvements on petabyte-scale data lakes.
Read more