Open Source

label Delta pull requests with area:Delta

read each native scan partition through the object store its files live in

give each nondeterministic dispatched expression its own kernel

honor subexpression elimination off in the native null short-circuit

read fs.comet.libhdfs.schemes for the scan rule from the Hadoop conf

fall back to Spark for NOT IN over an ordered streamed side

keep unconverted DPP filters in a scan's canonical form

add native Delta Lake scan contrib module (page/row-group pruning)

do not resolve DPP subqueries when AQE reads a native scan's partitioning

expect Spark 3.5's empty LocalTableScan in the single-split DPP join test

run the S3 suites against an available MinIO image

cover container field ids next to INT96 timestamps in the native Parquet scan

resolve Azure ABFS authentication through Hadoop and keep ambient credentials out

retry a native acquire that Spark failed after dropping the task's entry

retry a shuffle page allocation that Spark failed after dropping the task's entry

keep the sort-merge join when the forced hash join's build side is too large

load the Iceberg storage scheme lists from the native factory

cover sequence's per-batch element limit and the batch size remedy

convert native ReadAncientDatetime errors to SparkUpgradeException

add native metrics from every plan a task runs instead of keeping the last

read the shuffle natively in operators AQE reuses from the initial plan

key each scan's planning data by its plan node in a native block

check Iceberg DPP pruning by planned file tasks

reject a file without field ids at any depth whether or not id matching is on

report task input metrics after the native iterator closes and add to Spark's counters

match Spark's duplicate field and field id semantics in parquet field lookup

let decimal SUM recover from an intermediate overflow like Spark

gate the regr_r2 degenerate-case swap on the Spark patch release

guard page skipping in the native Iceberg scan

run length on binary input natively

drop the redundant width_bucket shim registrations and guard serde uniqueness

cache parsed plan data across a stage's tasks

reuse zstd compression contexts across shuffle blocks

decline structs with duplicate field names before they reach Java Arrow

compile user regex patterns once per planned expression

label pull requests by changed paths and title prefix

expand object store option references, uniquify constant metadata names, drop dead parquet JNI

reuse per-partition scratch in the shuffle write path

charge semi/anti sort-merge join key groups by their sliced size

normalize float hash join build keys once

skip functional dependency work when inputs carry none

stop redoing per-chunk setup on the hash join probe path

preserve field metadata when coercing INT96 timestamps (branch-55 backport)

skip wasted per-batch work in TopK insert_batch

SPARK-54593: enable DPP for materialized filtering sides

revisions for spark sql performance tuning to improve readability and grammar

updated some grammar and a missed period in the tuning doc

expose parse on the package for config trees

release 0.2.1

Python 3.10 support

Scala-style sealed trait support via dataclasses

type hints use Text instead of str