fix(physical-plan): compact view buffers in PartitionedTopK (#26402)
Sole Author: Patrick Ribbsaeter (@patrickswedish) · Immutable Commit SHA: 264141bcfa0416be385f0a4cadf8f054ed3a618e
Memory-footprint fix compacting pinned StringView and BinaryView variadic data buffers inside PartitionedTopK operators to prevent multi-gigabyte batch buffer retention during streaming Top-K queries.
PR #26402
100% CI GREEN · 39/39
264141b
Immutable Git SHA
Rust
Systems
39/39 CI Green
CI & Test Matrix
01 · Engineering problem
The failure mode
Partitioned TopK operators holding Utf8View / BinaryView batches retained uncompacted variable-length backing buffers, inflating peak query memory.
02 · Technical response
What changed
- 01Triggered string/binary view buffer compaction when retaining rows inside partitioned TopK heaps.
- 02Added memory-accounting and correctness tests for Utf8View partitioned TopK queries.
03 · Verified outcome
Evidence-backed result
- ✓Passed 100% of Apache DataFusion CI checks with zero failures on commit 264141bcfa0416be385f0a4cadf8f054ed3a618e.
04 · Cryptographic Git record
Immutable source evidence
Independent open-source engineering authored solely by Patrick Ribbsaeter (@patrickswedish). Company and project names identify the public repository context only under nominative fair use. No employment, client, vendor, or partnership relationship is implied unless explicitly stated.