Data Engineering

Feb 24, 2026

Interlock: A STAMP-Based Safety Framework for Data Pipelines

How I built a STAMP-based safety framework in Go with declarative sensors, failure classification, and centralized observability for data pipeline reliability on AWS

data engineering go aws safety dynamodb step functions terraform eventbridge observability

Feb 7, 2026

PySpark Pipeline Framework: Configuration-Driven Pipelines for the Python Ecosystem

How pyspark-pipeline-framework brings configuration-driven architecture, lifecycle hooks, and resilience patterns to PySpark

python pyspark data engineering open source configuration streaming

Dec 23, 2025

Fitting 100 Statistical Distributions at Scale: 1000x Memory Reduction with PySpark

How spark-bestfit 3.0 fits distributions across Spark, Ray, and local backends with survival analysis, mixture models, and multivariate support

spark python data engineering data science statistics optimization ray distributed computing survival analysis

Dec 23, 2025

Building Production-Ready Spark Pipelines with Configuration-Driven Architecture

How spark-pipeline-framework reached 1.0 with Spark Connect support, streaming, and enterprise features

spark scala data engineering open source observability spark connect streaming

Jul 28, 2021

Data Optimization for Compacted Partitions: Achieving 77% Storage Reduction

How intelligent data optimization with linear ordering and Z-ordering achieved 77% storage reduction and 90% runtime improvements on petabyte-scale data lakes.

apache spark data engineering big data optimization parquet orc