Add Spark 4.0 debugging examples

- StreamingRateExample: Streaming app using rate source for continuous debugging
- SimpleBatchExample: Simple batch app for basic debugging scenarios
- Comprehensive README with setup and usage instructions
- Gradle and SBT build configurations for IDE support
- Examples allow setting breakpoints in connector code during execution

Author: ShimonSte

spark-4.0/examples/README.md

@@ -0,0 +1,179 @@
+# Spark 4.0 ClickHouse Connector Examples
+
+This directory contains example applications for debugging and testing the ClickHouse connector with Spark 4.0.
+
+## Prerequisites
+
+1. **ClickHouse Server Running**
+   ```bash
+   # Using Docker
+   docker run -d --name clickhouse-server \
+     -p 8123:8123 -p 9000:9000 \
+     --ulimit nofile=262144:262144 \
+     clickhouse/clickhouse-server
+   ```
+
+2. **Build the Connector**
+   ```bash
+   cd /Users/shimonsteinitz/Projects/spark-clickhouse-connector
+   ./gradlew -Dspark_binary_version=4.0 -Dscala_binary_version=2.13 :clickhouse-spark-4.0_2.13:build
+   ```
+
+## Running Examples in IDE (for Debugging)
+
+### IntelliJ IDEA / VS Code with Metals
+
+1. **Import the project** as a Gradle project
+
+2. **Set up Run Configuration**:
+   - Main class: `examples.StreamingRateExample` or `examples.SimpleBatchExample`
+   - VM options:
+     ```
+     -Dspark_binary_version=4.0
+     -Dscala_binary_version=2.13
+     ```
+   - Working directory: `spark-4.0/examples`
+   - Classpath: Include `clickhouse-spark-4.0_2.13` module
+
+3. **Set Breakpoints** in connector code:
+   - Write path: `com.clickhouse.spark.write.ClickHouseWriter`
+   - Read path: `com.clickhouse.spark.read.ClickHouseReader`
+   - Catalog operations: `com.clickhouse.spark.ClickHouseCatalog`
+
+4. **Run in Debug Mode** and step through the connector code
+
+## Examples
+
+### 1. SimpleBatchExample
+
+A straightforward batch processing example that:
+- Creates sample employee data
+- Writes to ClickHouse
+- Reads back and performs aggregations
+
+**Good for debugging**:
+- Table creation logic
+- Batch write operations
+- Read operations
+- Schema inference
+
+**Run**:
+```bash
+spark-submit \
+  --class examples.SimpleBatchExample \
+  --master local[*] \
+  --jars clickhouse-spark-runtime-4.0_2.13.jar \
+  examples/SimpleBatchExample.scala
+```
+
+### 2. StreamingRateExample
+
+A streaming application that:
+- Uses Spark's rate source (generates synthetic data)
+- Enriches data with multiple columns
+- Writes to ClickHouse in micro-batches every 5 seconds
+- Generates 10 rows per second
+
+**Good for debugging**:
+- Streaming write operations
+- Micro-batch processing
+- Continuous data ingestion
+- Performance under load
+
+**Run**:
+```bash
+spark-submit \
+  --class examples.StreamingRateExample \
+  --master local[*] \
+  --jars clickhouse-spark-runtime-4.0_2.13.jar \
+  examples/StreamingRateExample.scala
+```
+
+**Monitor the stream**:
+```sql
+-- In ClickHouse client
+SELECT count(*) FROM default.streaming_events;
+
+SELECT 
+  event_type, 
+  count(*) as cnt, 
+  avg(metric_value) as avg_metric
+FROM default.streaming_events 
+GROUP BY event_type;
+
+SELECT 
+  toStartOfMinute(event_time) as minute,
+  count(*) as events_per_minute
+FROM default.streaming_events
+GROUP BY minute
+ORDER BY minute DESC
+LIMIT 10;
+```
+
+## Debugging Tips
+
+### Enable Debug Logging
+
+Add to your SparkSession configuration:
+```scala
+.config("spark.sql.catalog.clickhouse.option.log.level", "DEBUG")
+```
+
+Or set log level programmatically:
+```scala
+spark.sparkContext.setLogLevel("DEBUG")
+```
+
+### Useful ClickHouse Queries
+
+```sql
+-- Check table structure
+DESCRIBE TABLE default.streaming_events;
+
+-- Check table engine and settings
+SHOW CREATE TABLE default.streaming_events;
+
+-- Monitor inserts
+SELECT 
+  table,
+  sum(rows) as total_rows,
+  sum(bytes) as total_bytes
+FROM system.parts
+WHERE database = 'default' AND table IN ('streaming_events', 'employees')
+GROUP BY table;
+
+-- Check recent parts
+SELECT 
+  partition,
+  name,
+  rows,
+  bytes_on_disk,
+  modification_time
+FROM system.parts
+WHERE database = 'default' AND table = 'streaming_events'
+ORDER BY modification_time DESC
+LIMIT 10;
+```
+
+## Troubleshooting
+
+### Connection Issues
+
+If you see connection errors:
+```scala
+// Verify ClickHouse is accessible
+curl http://localhost:8123/ping
+```
+
+### Clean Up
+
+```sql
+-- Drop tables
+DROP TABLE IF EXISTS default.streaming_events;
+DROP TABLE IF EXISTS default.employees;
+```
+
+```bash
+# Remove checkpoint directory
+rm -rf /tmp/clickhouse-streaming-checkpoint
+```

spark-4.0/examples/SimpleBatchExample.scala

@@ -0,0 +1,56 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+plugins {
+    id "scala"
+    id "idea"
+}
+
+dependencies {
+    implementation "org.scala-lang:scala-library:${scala_version}"
+
+    // Spark APIs are needed at runtime when running from IDE/Gradle
+    implementation "org.apache.spark:spark-sql_${scala_binary_version}:${spark_40_version}"
+    implementation "org.apache.spark:spark-streaming_${scala_binary_version}:${spark_40_version}"
+
+    // Use connector project classes, and include shaded runtime on classpath
+    implementation project(":clickhouse-spark-4.0_${scala_binary_version}")
+    runtimeOnly project(":clickhouse-spark-runtime-4.0_${scala_binary_version}")
+}
+
+sourceCompatibility = JavaVersion.VERSION_17
+
+tasks.withType(ScalaCompile).configureEach {
+    scalaCompileOptions.additionalParameters = [
+            "-deprecation",
+            "-feature"
+    ]
+}
+
+// Application convenience tasks
+tasks.register("runStreaming", JavaExec) {
+    group = "application"
+    description = "Run StreamingRateExample"
+    classpath = sourceSets.main.runtimeClasspath
+    mainClass = "examples.StreamingRateExample"
+    jvmArgs "-Dspark_binary_version=4.0", "-Dscala_binary_version=2.13"
+}
+
+tasks.register("runBatch", JavaExec) {
+    group = "application"
+    description = "Run SimpleBatchExample"
+    classpath = sourceSets.main.runtimeClasspath
+    mainClass = "examples.SimpleBatchExample"
+    jvmArgs "-Dspark_binary_version=4.0", "-Dscala_binary_version=2.13"
+}

spark-4.0/examples/StreamingRateExample.scala

@@ -0,0 +1,35 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+name := "clickhouse-spark-examples"
+
+version := "1.0"
+
+scalaVersion := "2.13.8"
+
+val sparkVersion = "4.0.1"
+
+libraryDependencies ++= Seq(
+  "org.apache.spark" %% "spark-sql" % sparkVersion % "provided",
+  "org.apache.spark" %% "spark-streaming" % sparkVersion % "provided"
+)
+
+// For local development, include the connector from the parent project
+unmanagedClasspath in Compile ++= {
+  val base = baseDirectory.value / ".." / "clickhouse-spark" / "build" / "classes"
+  Seq(
+    Attributed.blank(base / "scala" / "main"),
+    Attributed.blank(base / "java" / "main")
+  )
+}