Nested Struct Nullability Adaptation in Apache DataFusion Aggregation

Engineering objective

System challenge

MemTable::try_new accepts RecordBatches with stricter schemas than the declared table schema via Schema::contains. However, MemoryStream advertised the declared schema while emitting the underlying stricter RecordBatches without adaptation. Downstream operators such as AggregateExec with array_agg or distinct aggregation received batches with stricter nested schemas, causing runtime type mismatch errors.

Constraints

Working within real limits

• ✕Preserve the existing Schema::contains acceptance semantics in MemTable::try_new
• ✕Handle recursive nested Struct, List, Dense Union, and Sparse Union type adaptation
• ✕Preserve Union type IDs and dense offsets without copying buffer data
• ✕Ensure all emitted RecordBatches conform exactly to MemoryStream::schema()
• ✕Maintain existing SQL CAST semantics (requires_nested_struct_cast untouched)

Approach

How it was built

1. 01

   Implemented adapt_batch_to_schema in datafusion_common::nested_struct for recursive schema adaptation.

2. 02

   Reconstructed compatible nested Struct/List types, explicitly handling Dense and Sparse Union conformance.

3. 03

   Normalized batches at the MemoryStream producer boundary in poll_next when runtime batch schema differs from self.schema.

4. 04

   Added regression coverage: unit tests for adapt_batch_to_schema, MemoryStream emission tests, and end-to-end SQL aggregation integration tests.

Technical validation

Implementation evidence

Observable engineering evidence. Discipline: UPSTREAM SHIPPED

Public evidence

Review the upstream contribution

Outcome

What was delivered

• Queries aggregating in-memory tables with stricter nested nullability now succeed as expected

• Recursive Struct/List/Union schema adaptation handles all nested type combinations

• MemoryStream producer boundary guarantees schema conformance for all downstream operators

• Merged upstream and shipped in Apache DataFusion v55.1.0 release with 8 review comments and 4 labels (core, common, physical-plan)

Technologies

Stack and tools

Next engineering system