Spark 4 support #452

ShimonSte · 2025-11-16T12:25:10Z

Add Spark 4.0 Support

Overview

This PR adds support for Apache Spark 4.0 with Scala 2.13 to the ClickHouse Spark Connector.

Changes

Core Implementation

Spark 4.0 Module: Added spark-4.0/ directory with connector implementation and integration tests
Scala 2.13: Full compatibility with Scala 2.13 (Spark 4.0 requirement)
Java 17: Updated build configuration to use Java 17 for Spark 4.0

Compatibility

Spark 4.0.x with Scala 2.13
Java 17 (minimum requirement for Spark 4.0)
Maintains backward compatibility with Spark 3.3, 3.4, and 3.5

windsurf-bot · 2025-11-16T12:25:13Z

This PR has too many files to review (>50 files).

windsurf-bot · 2025-11-16T14:33:36Z

This PR has too many files to review (>50 files).

mzitnik

Since the PR is huge, let's have a 1:1 to go over it, but a few comments for now

mzitnik · 2025-11-16T16:26:27Z

.github/workflows/build-and-test.yml

        with:
          distribution: zulu
-          java-version: 8
+          java-version: 17


I think we can merge run-tests and run-tests-with-specific-clickhouse into one test. Do you know what version we are running on run-tests

The run-tests run on 23.8 (the default version in the ClickHouseSingleMixIn file).
We could merge them into one test but then we'll need to update the matrix that would run all tests on all CH versions (might actually be a good idea but it'll create a lot of new tests. maybe we should change this only when we fix the tests so that they will be able to run concurrently)

In general, this test is against local ClickHouse, not the cloud, so it does not have the concurrency issue

mzitnik · 2025-11-16T16:27:32Z

.github/workflows/cloud.yml

-        scala: [ 2.12, 2.13 ]
+        spark: [ '3.3', '3.4', '3.5', '4.0' ]
+        scala: [ '2.12', '2.13' ]
+        java: [ 8 ]


why not java: [ 8, 17 ]

Again this would cause many more versions of the tests to run (today it is only running 8).

I'am ok with that since we might have a so feature that is not supported in 8 that can break so it better to test

mzitnik · 2025-11-16T16:28:08Z

.github/workflows/tpcds.yml

-        scala: [ 2.12, 2.13 ]
+        spark: [ '3.3', '3.4', '3.5', '4.0' ]
+        scala: [ '2.12', '2.13' ]
+        java: [ 8 ]


same here java: [ 8, 17 ]

Same as previous comment

mzitnik · 2025-11-16T16:29:17Z

build.gradle

    apply plugin: "java-library"
-    apply plugin: "org.scoverage"
+    // Disable scoverage when running Metals' bloopInstall to avoid plugin resolution issues
+    def isBloopInstall = gradle.startParameter.taskNames.any { it.contains('bloopInstall') }


Did you check if it does not affect IntelliJ?

It shouldn't affect as IntelliJ uses its own build server and it doesn't use Bloop at all

clickhouse-core/src/testFixtures/scala/com/clickhouse/spark/base/ClickHouseSingleMixIn.scala

- Copied spark-3.5/clickhouse-spark to spark-4.0/clickhouse-spark - Copied spark-3.5/clickhouse-spark-it to spark-4.0/clickhouse-spark-it - Copied build.gradle with module name references updated to 4.0 - This is an exact copy of Spark 3.5 code before API adaptations - No functional changes, only directory structure and module naming

- Updated AnalysisException constructor to require errorClass and messageParameters - Updated ArrowUtils.toArrowSchema to include largeVarTypes parameter (false for compatibility) - These are breaking API changes in Spark 4.0 that require code updates

- Added Spark 4.0.1 version configuration to gradle.properties - Updated ANTLR version to 4.13.1 to match Spark 4.0 dependencies - Added Spark 4.0 examples module to settings.gradle - Updated build.gradle with Spark 4.0 compatibility settings

- Added BeforeAndAfterAll trait to ClickHouseSingleMixIn - Fixed trait linearization order: BeforeAndAfterAll before ForAllTestContainer - Added logging for container lifecycle debugging - Resolves Spark 4.0 test compilation issues with trait conflicts

- Added Spark 4.0 with Scala 2.13 to build-and-test workflow - Added Spark 4.0 to check-license and style workflows with Java 17 - Added Spark 4.0 to cloud and tpcds workflows - Excluded Spark 4.0 + Scala 2.12 combinations (not supported) - Configured Java 17 requirement for Spark 4.0 builds

- StreamingRateExample: Streaming app using rate source for continuous debugging - SimpleBatchExample: Simple batch app for basic debugging scenarios - Comprehensive README with setup and usage instructions - Gradle and SBT build configurations for IDE support - Examples allow setting breakpoints in connector code during execution

- Added Logging trait to ClickHouseSingleMixIn - Replaced all println statements with log.info() calls - Provides structured, filterable logging for test execution - Maintains test output visibility while enabling proper log management

mzitnik · 2025-11-17T13:04:34Z

spark-4.0/clickhouse-spark/src/main/scala/org/apache/spark/sql/clickhouse/SparkUtils.scala


-  def toArrowSchema(schema: StructType, timeZoneId: String): Schema = ArrowUtils.toArrowSchema(schema, timeZoneId, true)
+  def toArrowSchema(schema: StructType, timeZoneId: String): Schema =
+    ArrowUtils.toArrowSchema(schema, timeZoneId, true, false)


looking at the API https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/util/ArrowUtils.html#toArrowSchema(org.apache.spark.sql.types.StructType,java.lang.String,boolean,boolean)

Do we know how this boolean largeVarTypes = false will affect us?

largeVarTypes = false uses 32-bit offsets (max 2GB per string/binary value) which is the Spark 3.x behavior, while true would use 64-bit offsets for massive values but likely break compatibility.

mzitnik · 2025-11-17T13:22:27Z

@ShimonSte
Looks like we are missing TPC_DS tests for Spark 4

- Updated build-and-test.yml: Added Java 8 & 17 matrix to run-tests-with-specific-clickhouse - Updated cloud.yml: Test all Spark 3.x versions with both Java 8 & 17 - Updated tpcds.yml: Test all Spark 3.x versions with both Java 8 & 17 - Spark 4.0 continues to use Java 17 only (requires Java 11+) - Updated artifact names to include Java version for better debugging

ANTLR 4.13.2 (required for Spark 4.0) needs Java 11+ to compile. The run-tests job already handles Java 8 testing for Spark 3.x versions. This job now only uses Java 17 to avoid ANTLR compilation errors.

pan3793 · 2025-11-18T10:31:16Z

gradle.properties

 spark_33_scala_212_version=2.12.15
 spark_34_scala_212_version=2.12.17
 spark_35_scala_212_version=2.12.18
+spark_40_scala_212_version=2.12.18


spark 4.0 does not support scala 2.12

pan3793 · 2025-11-18T10:32:18Z

gradle.properties


 # Align with Apache Spark, and don't bundle them in release jar.
 commons_lang3_version=3.12.0
 commons_codec_version=1.16.0


the apache commons libs have good backward compatibility, let's upgrade them to align with latest stable Spark version.

pan3793 · 2025-11-18T10:33:09Z

gradle.properties

 kyuubi_version=1.9.2
 testcontainers_scala_version=0.41.2
 scalatest_version=3.2.16
 flexmark_version=0.62.2


also suggest upgrading scalatest and flexmark to align with spark 4.0.1

We should keep flexmark at the current version as newer versions do not support java 8.
I think it is time we start talking about deprecating java 8, I'll raise it up as an issue.

Unfortunately, in the big data world, especially the Hadoop ecosystem still heavily stuck with Java 8, I would suggest deferring such deprecation until we hit significant issues.

Additional notes: Spark is not a service, but more like a library or programming framework that mixes user code at runtime, so it's impossible to make aggressive upgrades and ask all users to tune all their jobs. In practice, some Spark jobs may be unable to upgrade to a new Spark version in their lifecycle.

Great point. Makes sense to keep java as-is for now. Thanks for the context!

pan3793 · 2025-11-18T10:34:46Z

.github/workflows/tpcds.yml

+          # Spark 4.0 only supports Scala 2.13
+          - spark: '4.0'
+            scala: '2.12'
+          # Spark 4.0 requires Java 11+


Suggested change

# Spark 4.0 requires Java 11+

# Spark 4.0 requires Java 17+

- Fix CI comments: Spark 4.0 requires Java 17+, not Java 11+ - Update commons-codec to 1.17.2 (align with Spark 4.0.1) - Update scalatest to 3.2.19 (align with Spark 4.0.1) - Update flexmark to 0.64.8 (latest stable)

Merged run-tests and run-tests-with-specific-clickhouse into a single job that tests all combinations of: - ClickHouse versions: 25.3, 25.6, 25.7, latest - Java versions: 8, 17 - Scala versions: 2.12, 2.13 - Spark versions: 3.3, 3.4, 3.5, 4.0 With proper exclusions: - Spark 4.0 only supports Scala 2.13 - Spark 4.0 requires Java 17+ Total: 52 test jobs (down from 64 with exclusions) - Spark 3.x: 16 jobs each (4 CH × 2 Java × 2 Scala) - Spark 4.0: 4 jobs (4 CH × Java 17 × Scala 2.13)

Changed from: 25.3, 25.6, 25.7, latest Changed to: 25.6, 25.7, 25.8, 25.9, latest This adds testing for newer ClickHouse versions (25.8, 25.9) while maintaining coverage of recent stable releases. Total test jobs increases from 52 to 65 (5 CH versions instead of 4).

Flexmark 0.64.8 requires Java 11+ which breaks tests running with Java 8. Version 0.62.2 is the last version compatible with Java 8.

Use findProperty with fallback values in build.gradle instead of getProperty. This allows us to omit spark_40_scala_212_version from gradle.properties since Spark 4.0 doesn't support Scala 2.12. If the property is missing, it falls back to sensible defaults: - scala_212_version: 2.12.18 - scala_213_version: 2.13.8

pan3793

overall LGTM, (I didn't go through all files, and I suppose most scala files are just copy-pasted, maybe with a little tuning to pass compile.)

ShimonSte force-pushed the spark-4-support branch from e3c873b to db9ed7a Compare November 16, 2025 13:28

ShimonSte requested review from BentsiLeviav and mzitnik November 16, 2025 13:37

ShimonSte marked this pull request as draft November 16, 2025 14:32

ShimonSte marked this pull request as ready for review November 16, 2025 14:33

mzitnik reviewed Nov 16, 2025

View reviewed changes

ShimonSte added 6 commits November 17, 2025 08:29

ShimonSte force-pushed the spark-4-support branch from cb77141 to 631d235 Compare November 17, 2025 10:25

ShimonSte force-pushed the spark-4-support branch from 631d235 to 6d449b3 Compare November 17, 2025 10:39

mzitnik reviewed Nov 17, 2025

View reviewed changes

ShimonSte added 2 commits November 17, 2025 16:52

Fix CI: Remove Java 8 from run-tests-with-specific-clickhouse

fc343e8

ANTLR 4.13.2 (required for Spark 4.0) needs Java 11+ to compile. The run-tests job already handles Java 8 testing for Spark 3.x versions. This job now only uses Java 17 to avoid ANTLR compilation errors.

pan3793 reviewed Nov 18, 2025

View reviewed changes

ShimonSte added 5 commits November 19, 2025 09:16

Fix comments and update dependency versions

a5d3234

- Fix CI comments: Spark 4.0 requires Java 17+, not Java 11+ - Update commons-codec to 1.17.2 (align with Spark 4.0.1) - Update scalatest to 3.2.19 (align with Spark 4.0.1) - Update flexmark to 0.64.8 (latest stable)

Revert flexmark to 0.62.2 for Java 8 compatibility

48d80e0

Flexmark 0.64.8 requires Java 11+ which breaks tests running with Java 8. Version 0.62.2 is the last version compatible with Java 8.

ShimonSte requested review from mzitnik and pan3793 November 20, 2025 06:30

pan3793 approved these changes Nov 20, 2025

View reviewed changes

mzitnik approved these changes Nov 20, 2025

View reviewed changes

ShimonSte merged commit 7f00753 into main Nov 20, 2025
244 of 294 checks passed

ShimonSte deleted the spark-4-support branch November 20, 2025 06:58

Spark 4 support #452

Spark 4 support #452

Uh oh!

Conversation

ShimonSte commented Nov 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Add Spark 4.0 Support

Overview

Changes

Core Implementation

Compatibility

Uh oh!

windsurf-bot bot commented Nov 16, 2025

Uh oh!

windsurf-bot bot commented Nov 16, 2025

Uh oh!

mzitnik left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mzitnik commented Nov 17, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pan3793 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ShimonSte commented Nov 16, 2025 •

edited

Loading