Spark Connect Compatibility Architecture for Capital One’s DataComPy

Engineering objective

System challenge

DataComPy comparison paths imported classic PySpark functions and Window classes directly. Spark Connect uses different DataFrame, Column, functions, and window implementations, so otherwise valid comparisons could fail when expressions were built for a remote Connect session without an active local SparkContext.

Constraints

Working within real limits

• ✕Preserve existing Spark Classic behavior while adding Spark Connect compatibility
• ✕Avoid duplicating comparator logic across two runtime-specific implementations
• ✕Dispatch from the actual DataFrame or Column type rather than global session state
• ✕Cover array, boolean, numeric, and string comparison paths consistently
• ✕Represent the public outcome accurately: the submitted PR was closed and not merged

Architecture

System design

Approach

How it was built

1. 01

   Introduced helpers that select the Spark functions module and Window class from the actual classic or Connect object in use.

2. 02

   Replaced direct classic-function imports throughout SparkSQLCompare with runtime-aware dispatch.

3. 03

   Applied the same compatibility boundary to array, boolean, numeric, and string comparator implementations.

4. 04

   Covered merge, duplicate handling, mismatch reporting, masking, caching, aggregation, and window-expression paths.

5. 05

   Added regression coverage proving that Spark Connect expressions can be constructed without an active local SparkContext.

6. 06

   Published the patch, validation record, and discussion through pull request #548 for upstream review.

Workflow

Implementation flow

Technical validation

Implementation evidence

Observable engineering evidence. Discipline: PUBLIC DESIGN INFLUENCE

Public evidence

Review the upstream contribution

Outcome

What was delivered

• A DataComPy maintainer publicly confirmed that some of the contribution’s design was incorporated into a separate implementation branch

• Spark Classic and Spark Connect compatibility was expressed through one shared runtime-dispatch boundary

• Array, boolean, numeric, and string comparator paths were updated consistently

• 34 focused comparator/helper tests and 8 end-to-end SparkSQLCompare tests were reported passing

• Spark Connect regression validation demonstrated expression construction without an active local SparkContext

• The public record clearly shows the PR was closed without merge, preserving an accurate evidence boundary

Technologies

Stack and tools

Next engineering system