fix: support Spark Connect expressions via object-level runtime dispatch (#548)
Sole Author: Patrick Ribbsaeter (@patrickswedish) · Immutable Commit SHA: 8a3c0e632d820dc2b80a82b3cea0fe980f8bef10
Distributed data reconciliation fix replacing broken process-global SPARK_CONNECT_MODE_ENABLED state assumptions with dynamic object-level dispatch across DataFrame and Column comparators.
PR #548
ADOPTED & SHIPPED · 2/2 CI GREEN
8a3c0e6
Immutable Git SHA
Python
Systems
2/2 CI Green
CI & Test Matrix
01 · Engineering problem
The failure mode
SparkSQLCompare selected PySpark classic vs. Spark Connect expression modules using process-global SPARK_CONNECT_MODE_ENABLED state, failing when Connect sessions were instantiated externally.
02 · Technical response
What changed
- 01Replaced global-flag assumptions with runtime inspection of the active DataFrame and Column objects.
- 02Unified array, boolean, numeric, and string comparators behind object-level Connect/Classic dispatch.
03 · Verified outcome
Evidence-backed result
- ✓Passed 11/11 CI checks on commit 8a3c0e632d820dc2b80a82b3cea0fe980f8bef10 and adopted upstream by Capital One maintainers with credit to @patrickswedish / PR #548.
04 · Cryptographic Git record
Immutable source evidence
Independent open-source engineering authored solely by Patrick Ribbsaeter (@patrickswedish). Company and project names identify the public repository context only under nominative fair use. No employment, client, vendor, or partnership relationship is implied unless explicitly stated.