Fitting 100 Statistical Distributions at Scale: 1000x Memory Reduction with PySpark
How spark-bestfit 2.0 fits distributions across Spark, Ray, and local backends with a class-based API and 1000x memory reduction
Read moreData Engineering
Building Production-Ready Spark Pipelines with Configuration-Driven Architecture
How I built a configuration-driven Spark pipeline framework with structured observability and reflection-based component instantiation
Read more
Data Optimization for Compacted Partitions: Achieving 77% Storage Reduction
How intelligent data optimization with linear ordering and Z-ordering achieved 77% storage reduction and 90% runtime improvements on petabyte-scale data lakes.
Read more