Spark Connect Compatibility Architecture & Upstream Adoption for Capital One’s DataComPy

Engineering objective

System challenge

When SparkSQLCompare received an already-created Connect session, expression selection relied on process-global Spark state (SPARK_CONNECT_MODE_ENABLED) rather than the actual DataFrame or Column object. As a result, classic Spark expressions could be constructed against Spark Connect DataFrames, causing runtime failures across remote cluster and notebook environments.

Constraints

Working within real limits

• ✕Preserve existing Spark Classic behavior while adding complete Spark Connect compatibility
• ✕Avoid duplicating comparator logic across two runtime-specific implementations
• ✕Dispatch dynamically from the actual DataFrame or Column object type rather than trusting global session state
• ✕Cover array, boolean, numeric, and string comparison paths consistently
• ✕Validate upstream maintainer adoption with full regression coverage and green CI

Architecture

System design

Approach

How it was built

1. 01

   Diagnosed root cause: SPARK_CONNECT_MODE_ENABLED assumption fails when Connect sessions originate outside the library.

2. 02

   Engineered runtime-aware dispatch (PR #548) that resolves Spark functions and Window implementations directly from the target DataFrame or Column.

3. 03

   Replaced direct classic-function imports throughout SparkSQLCompare with object-level runtime dispatch.

4. 04

   Unified compatibility boundaries across array, boolean, numeric, and string comparator implementations.

5. 05

   Capital One maintainers integrated the object-dispatch approach into upstream PR #552, explicitly crediting @patrickswedish / #548 for the design and root cause diagnosis.

6. 06

   Capital One extended the PR with comprehensive regression coverage, Connect validation, and green CI across distributed test matrices.

Workflow

Implementation flow

Technical validation

Implementation evidence

Observable engineering evidence. Discipline: UPSTREAM ADOPTED · PR #552

Public evidence

Review the upstream contribution

Outcome

What was delivered

• Capital One upstream PR #552 adopted the runtime object dispatch architecture, explicitly crediting @patrickswedish / #548

• Root cause of process-global Spark Connect defect diagnosed and resolved

• Spark Classic and Spark Connect compatibility unified through one shared runtime-dispatch boundary

• Array, boolean, numeric, and string comparator paths updated consistently

• 100% passing CI on Capital One upstream repository with full regression coverage

• Enterprise-grade reliability established for banking and FinTech distributed data reconciliation

Technologies

Stack and tools

Next engineering system