fix(parquet): support int96 coercion in legacy list layouts (#26395)
Sole Author: Patrick Ribbsaeter (@patrickswedish) · Immutable Commit SHA: a0ebcfa01bde9ea115a9b52bea738948b86c7915
Parquet schema-coercion fix recursively coercing legacy INT96 physical timestamps nested inside 2-level Spark/Hive list layouts to target Arrow Timestamp units without reader schema mismatch.
PR #26395
100% CI GREEN · 39/39
a0ebcfa
Immutable Git SHA
Rust
Systems
39/39 CI Green
CI & Test Matrix
01 · Engineering problem
The failure mode
Parquet INT96 timestamps coerced to target Arrow timestamp units risked truncation or mismatch across Parquet row-group pruning and physical coercion boundaries.
02 · Technical response
What changed
- 01Aligned Parquet INT96 timestamp coercion and statistics pruning semantics in DataFusion's Parquet datasource.
- 02Added end-to-end Parquet INT96 nanosecond/microsecond/millisecond regression tests.
03 · Verified outcome
Evidence-backed result
- ✓Passed 100% of Apache DataFusion CI checks with zero failures on commit a0ebcfa01bde9ea115a9b52bea738948b86c7915.
04 · Cryptographic Git record
Immutable source evidence
Independent open-source engineering authored solely by Patrick Ribbsaeter (@patrickswedish). Company and project names identify the public repository context only under nominative fair use. No employment, client, vendor, or partnership relationship is implied unless explicitly stated.