Skip to content

Conversation

@mzitnik
Copy link
Collaborator

@mzitnik mzitnik commented Oct 26, 2025

Summary

Checklist

Delete items not relevant to your PR:

  • Unit and integration tests covering the common scenarios were added
  • A human-readable description of the changes was provided to include in CHANGELOG
  • For significant changes, documentation in https://github.com/ClickHouse/clickhouse-docs was updated with further explanations or tutorials

@mzitnik mzitnik marked this pull request as ready for review October 29, 2025 05:41
@mzitnik mzitnik requested a review from BentsiLeviav October 29, 2025 05:41
Copy link

@windsurf-bot windsurf-bot bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Other comments (20)

💡 To request another review, post a new comment with "/windsurf-review".

Comment on lines +228 to +229
// , codec
client.syncInsertOutputJSONEachRow(database, table, format, new ByteArrayInputStream(data)) match {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's a commented-out parameter and explanatory comment that should be removed before merging. If the codec parameter is no longer needed in the method call due to API changes, we should just remove it cleanly without leaving commented code.

lazy val resp: QueryResponse = nodeClient.queryAndCheck(scanQuery, format)

def totalBlocksRead: Long = resp.getSummary.getStatistics.getBlocks
def totalBlocksRead: Long = 0L // resp.getSummary.getStatistics.getBlocks
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The totalBlocksRead method now returns a hardcoded 0L instead of getting the actual value from the response. This will cause the BLOCKS_READ metric to always report 0, which could be misleading for monitoring and debugging purposes.

lazy val resp: QueryResponse = nodeClient.queryAndCheck(scanQuery, format)

def totalBlocksRead: Long = resp.getSummary.getStatistics.getBlocks
def totalBlocksRead: Long = 0L // resp.getSummary.getStatistics.getBlocks
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The totalBlocksRead method now returns a hardcoded 0L instead of getting the actual block count from the response. This will cause inaccurate metrics reporting. Consider updating this method to retrieve the block count from the new QueryResponse API.

@BentsiLeviav BentsiLeviav self-requested a review November 2, 2025 09:52
mzitnik and others added 10 commits November 2, 2025 16:54
* Wake up ClickHouse Cloud instance before tests (#429)

* fix: Handle FixedString as plain text in JSON reader for all Spark versions

Problem:
ClickHouse returns FixedString as plain text in JSON format, but the
connector was trying to decode it as Base64, causing InvalidFormatException.

Solution:
Use pattern matching with guard to check if the JSON node is textual.
- If textual (FixedString): decode as UTF-8 bytes
- If not textual (true binary): decode as Base64

Applied to Spark 3.3, 3.4, and 3.5.

---------

Co-authored-by: Bentsi Leviav <[email protected]>
Co-authored-by: Shimon Steinitz <[email protected]>
* Wake up ClickHouse Cloud instance before tests (#429)

* feat: Add comprehensive read test coverage for Spark 3.3, 3.4, and 3.5

Add shared test trait ClickHouseReaderTestBase with 48 test scenarios covering:
- All primitive types (Boolean, Byte, Short, Int, Long, Float, Double)
- Large integers (UInt64, Int128, UInt128, Int256, UInt256)
- Decimals (Decimal32, Decimal64, Decimal128)
- Date/Time types (Date, Date32, DateTime, DateTime32, DateTime64)
- String types (String, UUID, FixedString)
- Enums (Enum8, Enum16)
- IP addresses (IPv4, IPv6)
- JSON data
- Collections (Arrays, Maps)
- Edge cases (empty strings, long strings, empty arrays, nullable variants)

Test suites for Binary and JSON read formats.

Test results: 96 tests per Spark version (288 total)
- Binary format: 47/48 passing
- JSON format: 47/48 passing
- Overall: 94/96 passing per version (98% pass rate)

Remaining failures are known bugs with fixes on separate branches.

* feat: Add comprehensive write test coverage for Spark 3.3, 3.4, and 3.5

Add shared test trait ClickHouseWriterTestBase with 17 test scenarios covering:
- Primitive types (Boolean, Byte, Short, Int, Long, Float, Double)
- Decimal types
- String types (regular and empty strings)
- Date and Timestamp types
- Collections (Arrays and Maps, including empty variants)
- Nullable variants

Test suites for JSON and Arrow write formats.
Note: Binary write format is not supported (only JSON and Arrow).

Test results: 34 tests per Spark version (102 total)
- JSON format: 17/17 passing (100%)
- Arrow format: 17/17 passing (100%)
- Overall: 34/34 passing per version (100% pass rate)

Known behavior: Boolean values write as BooleanType but read back as ShortType (0/1)
due to ClickHouse storing Boolean as UInt8.

* style: Apply spotless formatting

* style: Apply spotless formatting for Spark 3.3 and 3.4

Remove trailing whitespace from test files to pass CI spotless checks.

* fix: Change write format from binary to arrow in BinaryReaderSuite

The 'binary' write format option doesn't exist. Changed to 'arrow'
which is a valid write format option.

Applied to Spark 3.3, 3.4, and 3.5.

* test: Add nullable tests for ShortType, IntegerType, and LongType

Added missing nullable variant tests to ensure comprehensive coverage:
- decode ShortType - nullable with null values (Nullable(Int16))
- decode IntegerType - nullable with null values (Nullable(Int32))
- decode LongType - nullable with null values (Nullable(Int64))

These tests verify that nullable primitive types correctly handle NULL
values in both Binary and JSON read formats.

Applied to Spark 3.3, 3.4, and 3.5.

Total tests per Spark version: 51 (was 48)
Total across all versions: 153 (was 144)

* Refactor ClickHouseReaderTestBase: Add nullable tests and organize alphabetically

- Add missing nullable test cases for: Date32, Decimal32, Decimal128, UInt16, UUID, DateTime64
- Organize all 69 tests alphabetically by data type for better maintainability
- Ensure comprehensive coverage with both nullable and non-nullable variants for all data types
- Apply changes consistently across Spark 3.3, 3.4, and 3.5

* ci: Skip cloud tests on forks where secrets are unavailable

Add repository check to cloud workflow to prevent failures on forks
that don't have access to ClickHouse Cloud secrets. Tests will still
run on the main repository where secrets are properly configured.

* Refactor and enhance Reader/Writer tests for all Spark versions

- Add BooleanType tests to Reader (2 tests) with format-aware assertions
- Add 6 new tests to Writer: nested arrays, arrays with nullable elements,
  multiple Decimal precisions (18,4 and 38,10), Map with nullable values, and StructType
- Reorder all tests lexicographically for better organization
- Writer tests increased from 17 to 33 tests
- Reader tests increased from 69 to 71 tests
- Remove section header comments for cleaner code
- Apply changes to all Spark versions: 3.3, 3.4, and 3.5
- All tests now properly sorted alphabetically by data type and variant

* style: Apply spotless formatting to Reader/Writer tests

---------

Co-authored-by: Bentsi Leviav <[email protected]>
Co-authored-by: Shimon Steinitz <[email protected]>
- Fix DecimalType: Handle both BigInteger (Int256/UInt256) and BigDecimal (Decimal types)
- Fix ArrayType: Direct call to BinaryStreamReader.ArrayValue.getArrayOfObjects()
- Fix StringType: Handle UUID, InetAddress, and EnumValue types
- Fix DateType: Handle both LocalDate and ZonedDateTime
- Fix MapType: Handle all util.Map implementations

Removed reflection and defensive pattern matching for better performance.
All 34 Binary Reader test failures are now fixed (71/71 tests passing).

Fixes compatibility with new Java client API in update-java-client-version branch.
- Add Decimal(18,4) test with 0.001 tolerance for JSON/Arrow formats
- Documents precision limitation for decimals with >15-17 significant digits
- Uses tolerance-based assertions to account for observed precision loss
- Binary format preserves full precision (already tested in Binary Reader suite)
- All 278 tests passing
@CLAassistant
Copy link

CLAassistant commented Nov 9, 2025

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
2 out of 3 committers have signed the CLA.

✅ mzitnik
✅ ShimonSte
❌ Shimon Steinitz


Shimon Steinitz seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

@ShimonSte ShimonSte force-pushed the update-java-client-version branch from 7ea939c to 8beb2bc Compare November 9, 2025 14:29
- Convert mutable.ArraySeq to Array in ClickHouseJsonReader to ensure immutable collections
- Add test workaround for Spark's Row.getSeq behavior in Scala 2.13
- Fix Spotless formatting: remove trailing whitespace in ClickHouseBinaryReader
- Applied to all Spark versions: 3.3, 3.4, 3.5
@ShimonSte ShimonSte force-pushed the update-java-client-version branch from 982511d to 2687b1f Compare November 10, 2025 09:13
@mzitnik mzitnik force-pushed the update-java-client-version branch from 98365d4 to 18b4fcb Compare November 13, 2025 09:45
@BentsiLeviav BentsiLeviav merged commit 32da21c into main Nov 16, 2025
40 of 52 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants