fix: adapt input batches with stricter nested nullability to planned schema (#24394)
Sole Author: Patrick Ribbsaeter (@patrickswedish) · Immutable Commit SHA: 2326917aa8d691aa5123518ca602ce63ab18ffc6
Distributed Rust query engine fix adapting stricter nested Struct, List, and Dense/Sparse Union nullability at the MemoryStream producer boundary without copying underlying Arrow buffer memory.
PR #24394
Merged Mainline
2326917
Immutable Git SHA
Rust
Systems
72/72 CI Green
CI & Test Matrix
01 · Engineering problem
The failure mode
In-memory RecordBatches frequently carry stricter nested struct, list, or dictionary nullability than the planned aggregate schema, causing physical schema mismatch errors at aggregation boundaries.
02 · Technical response
What changed
- 01Implemented recursive nested schema nullability adaptation at the MemoryStream and aggregate execution boundary.
- 02Preserved zero-copy array buffers while aligning nested Field nullability metadata to the planned schema.
- 03Added +1,093 lines of physical-plan and SQL aggregation regression tests.
03 · Verified outcome
Evidence-backed result
- ✓Eliminated nested nullability mismatch errors across DataFusion aggregation pipelines without plan disruption.
- ✓Passed 35/35 CI checks and merged upstream from commit 2326917aa8d691aa5123518ca602ce63ab18ffc6 (shipped in DataFusion v55.1.0).
04 · Cryptographic Git record
Immutable source evidence
Independent open-source engineering authored solely by Patrick Ribbsaeter (@patrickswedish). Company and project names identify the public repository context only under nominative fair use. No employment, client, vendor, or partnership relationship is implied unless explicitly stated.