Release v2.8.0

Lightweight
Docker: altinity/clickhouse-sink-connector:2.8.0-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.8.0-lt/images/sha256-3799c182de1e91a1dd1434d2f2035a6bdbb036d9a1f74bad64c8b537446db7f5

What's New

• Multi-platform(ARM) builds
• Oracle support(Experimental)
• Version History(Temporal tables) (Experimental)
• Schema Override
• support for more MySQL DDL statements.
• Debezium upgrade to 3.1.3.Final

What's Changed

• Fix table name substitution for creating schema history table. by @subkanthi in #1103
• 2.7.1 by @subkanthi in #1104
• Add max_memory_usage support in clickhouse_table_checksum.py by @aadant in #1119
• Support for partition expression in checksum command lines by @aadant in #1118
• Fixed parser when generated columns have isnotnull by @subkanthi in #1115
• schema override functionality. by @subkanthi in #1072
• Update postgres docker configuration. by @subkanthi in #1123
• Upgraded debezium to 3.1.3.Final by @subkanthi in #1117
• Fix time order for inserted rows panel by @mohaidoss in #1095
• [1097] Support longtext MySQL DDL. by @subkanthi in #1101
• Added oracle docker files. by @subkanthi in #1112
• Fixed typo in commented-out config.yml parameter by @PlzStandBy in #1105
• Added logic to write history tables in a separate connector mode. by @subkanthi in #1063
• Implement blocking put for back-pressure in DebeziumChangeEvent in sink connector lightweight by @captainpro-eng in #1108
• Updated Debezium configuration. by @subkanthi in #1147
• Added integration test for binlog history. by @subkanthi in #1143
• Support create database on cluster by @subkanthi in #1135
• Mergetree history fixes. by @subkanthi in #1149
• Replaced bitnami mysql docker images with official mysql images. by @subkanthi in #1152
• Fixed mapping of bigint unsigned. by @subkanthi in #1153
• Enable kafka connect json and added scripts for running without schem… by @subkanthi in #1130
• Fix alter table add column not null tests. by @subkanthi in #1155

New Contributors

• @PlzStandBy made their first contribution in #1105
• @captainpro-eng made their first contribution in #1108

Full Changelog: 2.7.1...2.8.0

Release v2.7.1

Lightweight
Docker: altinity/clickhouse-sink-connector:2.7.1-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.7.1-lt/images/sha256-6aeb1dcbcd7b2f0e0f43f3f31cdc198a594977b2c2e338552465ad50eebd02ef

What's Changed

• 2.7.0 by @subkanthi in #1058
• Some documentation improvements for the production_setup page by @mohaidoss in #1094
• Fixed shell script path to get release tag. by @subkanthi in #1099
• Added maven wrapper by @subkanthi in #1100
• Fix table name substitution for creating schema history table. by @subkanthi in #1103

New Contributors

• @mohaidoss made their first contribution in #1094

Full Changelog: 2.7.0...2.7.1

Release v2.7.0

Lightweight
Docker: altinity/clickhouse-sink-connector:2.7.0-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.7.0-lt/images/sha256-2e0f1a0487a71a11b19200320707545ec5c5992b0abc37ad09ec954c36352f37

Breaking Changes

Configuration Changes
offset.storage.jdbc.offset.table.name to offset.storage.jdbc.table.name
offset.storage.jdbc.offset.table.ddl to offset.storage.jdbc.table.ddl
offset.storage.jdbc.offset.table.delete to offset.storage.jdbc.table.delete

What's new

• Debezium upgrade to 3.1.2.Final
• Various bug fixes

What's Changed

• fix: clickhouse_loader port and secure options by @philippevezina in #1041
• Added workaround to fix the debezium timestamp for DATETIME4, DATETIM… by @subkanthi in #1057
• [1049] Fixed typo in config file. by @subkanthi in #1050
• Added logic to display the retry count. by @subkanthi in #1023
• Fix integration tests 2 7 0 by @subkanthi in #1067
• Reenable disabled tests. by @subkanthi in #1033
• Fix/1046 nullpointerexception dbsource by @subkanthi in #1068
• [1046] Fix NullPointerException and Refactor Connection Pool Initialization Logic by @chimeyrock999 in #1048
• Reverted back network_mode: host with ports. by @subkanthi in #1070
• Rolled back network_mode: Host by @subkanthi in #1081
• Revert "Fix/1046 nullpointerexception dbsource" by @subkanthi in #1084
• NULL Primary Key from MySQL needs to be translated to CH. by @subkanthi in #1087
• Fixed encoding logic when password has special characters. by @subkanthi in #1086
• Added logic to persist error logs in a separate database. by @subkanthi in #1054
• Replaced redpanda images. by @subkanthi in #1073
• Upgraded debezium to 3.1.2.Final, Added integration test to definer SQL. by @subkanthi in #1055
• Fixed REST API test to primarily test sink-connector-client. by @subkanthi in #1047

New Contributors

• @philippevezina made their first contribution in #1041
• @chimeyrock999 made their first contribution in #1048

Full Changelog: 2.6.0...2.7.0

Release v2.6.0

Lightweight
Docker: altinity/clickhouse-sink-connector:2.6.0-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.6.0-lt/images/sha256-1114ee20b6ba76f78c2d4858ee3805115753aeecd26221875d68538ba1aeb6a8

What's New

MySQL SET Data Type #881
ARM support for docker builds (Experimental) - #955
Debezium upgraded to 3.1.0.AlphaFinal - #1001
PostgreSQL time with timezone mapping added - #951

What's Changed

• 2.5.1 by @subkanthi in #995
• Renable disabled integration tests. by @subkanthi in #1004
• Added connection pool documentation by @subkanthi in #1000
• Update README.md by @subkanthi in #1009
• Update connection_pool.md by @subkanthi in #1010
• Updated config doc by @subkanthi in #1011
• Pass jdbc parameters to datasource, removed okhttp dependencies. by @subkanthi in #1012
• Update debezium version to 3.1.0.AlphaFinal by @subkanthi in #1001
• Add ddl translation rest api by @subkanthi in #936
• Added support for create table for MySQL Set Data type by @subkanthi in #881
• Added step to install ps command for sink-connector lightweight by @subkanthi in #878
• 866 incorrect precision and scale parsing due to column comment interference by @subkanthi in #868
• Build cross-platform docker image for arm and amd by @subkanthi in #955
• Added test case for parsing ghost SQL statements by @subkanthi in #817
• Added integration test to validate replication of DDL with partitions. by @subkanthi in #953
• Add steps to start a second connector. by @subkanthi in #959
• Update README.md by @subkanthi in #1016
• Local environment setup(IntelliJ) by @subkanthi in #1017
• set errors.max.retries in config.yml by @subkanthi in #1018
• Revert database in config.yml to test by @subkanthi in #1020
• 993 datetime columns with non utc tz on source and target are not replicated properly by @subkanthi in #1022
• Added logic to set clickhouse settings from configuration parameter. by @subkanthi in #1024
• Added retry logic for acknowledging records. by @subkanthi in #1015
• Added mapping of time with timezone postgres datatype to clickhouse string and added integration test. by @subkanthi in #951
• Update ClickHouseDataTypeMapper.java: fix for working with bytebuffer by @salisbury-espinosa in #897
• Revert "Added retry logic for acknowledging records." by @subkanthi in #1031
• maven checkstyle fixes. by @subkanthi in #1030
• Fix datetime integration tests. by @subkanthi in #1029
• Added network_mode:host to docker compose by @subkanthi in #1038
• Testflows tests for 2 6 0 by @Selfeer in #1036
• Instructions to install JAR file for kafka sink connector by @subkanthi in #879
• Added integration test to cover MySQL demo. by @subkanthi in #984
• Added logic to CREATE on Cluster when replicated mode is enabled by @subkanthi in #880
• Update ddl docs by @subkanthi in #1025
• K8s manifest files by @subkanthi in #1032
• Update PreparedStatementExecutor.java: fix updates from postgres (update in postgres -> kafka -> clickhouse) by @salisbury-espinosa in #900
• Added monitoring of sink connector using docker compose by @subkanthi in #1044

Full Changelog: 2.5.1...2.6.0

Release v2.5.1

Lightweight
Docker: altinity/clickhouse-sink-connector:2.5.1-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.5.1-lt/images/sha256-ac7372e68da990fc0cee91923a779a4eca31407496aa0a15b2ec4a4a52acc165

What's new:

error.max.retries: All the clickhouse database calls will be retried based on the configuration variable.
Hikari connection pool: All the clickhouse db calls will be routed through Hikari connection pool. The number of connections are configurable.https://github.com/Altinity/clickhouse-sink-connector/blob/develop/doc/connection_pool.md

What's Changed

Postgres Numeric -> BigDecimal exception
Resolved an exception occurring when mapping PostgreSQL NUMERIC data types to Java's BigDecimal, enhancing compatibility with PostgreSQL sources.
Issue #980​

DateTime64 truncation limit behavior different from ClickHouse
Addressed inconsistencies in DateTime64 truncation behavior between the connector and ClickHouse, ensuring uniform data handling.
Issue #971​

DDL Translation error: modify column datatype with DEFAULT NULL
Fixed errors in Data Definition Language (DDL) translation when modifying column data types with DEFAULT NULL constraints, improving schema evolution support.
Issue #965
​
Hikari Connection Pool
​The pull request #893 introduces the integration of the HikariCP connection pool. This enhancement aims to efficiently manage ClickHouse connections, addressing potential connection leaks identified in issue #867. By implementing HikariCP, the connector benefits from improved performance and reliability through effective connection pooling.

• Update configuration.md by @subkanthi in #957
• Update README.md by @subkanthi in #961
• Update getLatestRelease.sh- remove extra quote. by @subkanthi in #962
• Update quickstart.md to explain export statement. by @subkanthi in #963
• Update architecture.md by @subkanthi in #964
• Created replica.status.view for postgres. by @subkanthi in #970
• #380 - Added configuration settings to prevent freeze of debezium loop. by @subkanthi in #974
• Updated version of upload-artifact to v4. by @subkanthi in #978
• Add jdbc params -keepalive by @subkanthi in #988
• Update Troubleshooting.md by @subkanthi in #989
• Added support to disable connection pool. by @subkanthi in #992
• Set nanoseconds in DateTime64 max to zeros and not nines by @subkanthi in #972
• Added unit test to validate modify column followed by default NULL by @subkanthi in #966
• Fix to limit truncate bigdecimal values. by @subkanthi in #981
• Use Hikari Connection Pool for handling Clickhouse connections by @subkanthi in #893

Full Changelog: 2.5.0...2.5.1

Release v2.5.0

Lightweight
Docker: altinity/clickhouse-sink-connector:2.5.0-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.5.0-lt/images/sha256-60d5edd7d89eeee4ec64ec802c95c223096793654cce256d58c4e15219c4d0e2?context=explore

Kafka
Docker: altinity/clickhouse-sink-connector:2.5.0-kafka
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.5.0-kafka/images/sha256-4cc61265b51b4a1bdfbed244c4b074daedb852c8a665ca1b9cf19a70243eb197?context=explore

What's new:

MongoDB support: Restored mongodb config.yml and instructions to setup sink connector to replicate MongoDB databases.
Debezium upgrade to 3.0.0. - Updated Debezium to the latest stable 3.0.0.
Ignore DDL statements based on regular expressions: ignore DDL from MySQL based on regular expressions. ignore.ddl.regex
Kafka Debezium upgrade: MySQL Debezium connector was upgraded to latest stable 3.0.0.
Performance improvements: Removed additional calls to clickhouse to get table schema.
MySQL DDL Support: Alter table rename without table keyword

What's Changed

• Update README.md to include initial data dump/load by @subkanthi in #821
• 794 change logging level of struct empty not a valid cdc record + record to info by @subkanthi in #825
• 801 records are not acknowledged or the offsets are not updated in singlethreaded mode by @subkanthi in #827
• Added release notes for 2.3.1 release by @subkanthi in #830
• Release 2.3.1 by @subkanthi in #824
• Single threaded mode diagram by @subkanthi in #823
• Add Altinity Slack badge to top of readme. by @hodgesrm in #844
• Update config.yml by @subkanthi in #846
• Added production setup for MySQL by @subkanthi in #767
• 2.4.0 by @subkanthi in #798
• Database override it rrmt by @subkanthi in #877
• 861 renaming a table doesnt work for databases that are mapped to another name in clickhouse by @subkanthi in #862
• Update configuration.md by @subkanthi in #884
• Update log4j2.xml changes to have sink connector in INFO level. by @subkanthi in #892
• Remove additional call to get column data types from clickhouse by @subkanthi in #889
• Update README.md by @subkanthi in #905
• Update architecture.md by @subkanthi in #910
• Update incremental_snapshot.md by @subkanthi in #911
• Added logic to support rename column DDL with database override. by @subkanthi in #912
• Added logic to create schema history table on connector startup. by @subkanthi in #914
• Added test to cover alter table rename without table keyword by @subkanthi in #909
• Update debezium-connector-setup.sh for 'delete' operation by @salisbury-espinosa in #902
• Fix integration tests in 2.4.1 branch by @subkanthi in #920
• 895 ignore ddl based on regex by @subkanthi in #896
• Add DDL test for CREATE TABLE like by @subkanthi in #929
• 384 remove references to older config by @subkanthi in #883
• Add retries for creating debezium storage database by @subkanthi in #926
• Updated debezium to 3.0.0.Final and kafka connect dependencies to the… by @subkanthi in #887
• Removed kube-rbac-proxy references by @subkanthi in #940
• Mongodb integration test by @subkanthi in #921
• Kafka version, upgrade mysql debezium to 3.0.4.Final, set mysql to 8.0 by @subkanthi in #939

New Contributors

• @salisbury-espinosa made their first contribution in #902

Full Changelog: 2.4.0...2.4.1t

Release v2.4.0

Lightweight
Docker: altinity/clickhouse-sink-connector:2.4.0-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.4.0-lt/images/sha256-d509cafcadd11de0fd25b935efb0ee9e49429e9150ca9950c89d194e7a0b7f77?context=explore

Kafka
Docker: altinity/clickhouse-sink-connector:2.4.0-kafka
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.4.0-kafka/images/sha256-21e31d08819c6af5c52c9cf80fefe6cba979d87da41a689c66f5cbed4f4b0e5a?context=explore

What's new:

ddl retry: When the ddl.retry: "true" variable is added to config.yml, if for some reason DDL's translated by sink connector fails to execute on ClickHouse, logic has been added to retry on an exponential backoff loop.
MySQL POINT & POLYGON data types: The Polygon and Point datatypes will be mapped to ClickHouse.
MySQL DDL Support: Added support for DROP CONSTRAINT and fixed bug with dropping NULL in ALTER TABLE.
sink-connector-client Added commands to purge schema history and offset table.
Exclude Generated/Alias and Materialized: Added logic to exclude Materialized and Alias columns.

What's Changed

• Update state_storage.md by @subkanthi in #771
• Update README.md to include state storage link by @subkanthi in #772
• Updated offset state storage documentation. by @subkanthi in #774
• Update state_storage.md to include schema storage. by @subkanthi in #775
• Update state_storage.md to include postgresql offsets by @subkanthi in #776
• Testflows new kafka tests by @Selfeer in #738
• Update quickstart.md to start sink connector service by @subkanthi in #785
• Added logic to get latest release from github by @subkanthi in #786
• Update quickstart.md with script to set environment variable. by @subkanthi in #787
• Update quickstart_postgres.md by @subkanthi in #788
• Added updates to script by @subkanthi in #789
• Added single.threaded flag to Mariadb test to validate replication in… by @subkanthi in #781
• Update production_setup.md by @subkanthi in #796
• Changed logging level to info for STRUCT EMPTY not a valid CDC record by @subkanthi in #795
• Update production_setup.md to include max_paritions_per_insert by @subkanthi in #804
• Added logic to commit batch in single threaded mode by @subkanthi in #802
• Update production_setup.md , fixed broken link by @subkanthi in #809
• Added support for DROP CONSTRAINT by @subkanthi in #807
• Update config.ym by @subkanthi in #813
• Removed logic of duplicating instances of DebeziumOffsetStorage by @subkanthi in #833
• Support MySQL point by @poweroftrue in #815
• Support MySQL Polygons by @poweroftrue in #853
• Added Integration test for testing POINT data type in MySQL by @subkanthi in #851
• Upgrade debezium to 2.7.1.Final by @subkanthi in #792
• Set log retention max size to 100M by @subkanthi in #805
• Added integration test to validate database.override.map in snapshot.mode=initial mode by @subkanthi in #818
• Log docker tag in startup by @subkanthi in #841
• Upgrade clickhouse.jdbc.version to 0.6.5 by @subkanthi in #832
• Single threaded documentation by @subkanthi in #822
• add ddl.retry by @Selfeer in #865
• Refactored DDL processing logic by @subkanthi in #839
• Testflows updates by @Selfeer in #873
• Added test to replicate generated columns in postgreSQL by @subkanthi in #777
• Added function to retrieve a map of column name and is_nullable as bo… by @subkanthi in #852
• Added test for MySQL DDL with partitions by @subkanthi in #784
• 773 add sink connector client step to purge schema history so that newer tables can be added by @subkanthi in #779

New Contributors

• @poweroftrue made their first contribution in #815

Full Changelog: 2.3.0...2

Release v2.3.1

Lightweight
Docker: altinity/clickhouse-sink-connector:2.3.1-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.3.1-lt/images/sha256-4cfbc7711a0c8fd47cda0385476026ce8456fb6a0eaeac7c9455f220c044cf07?context=explore

Kafka
Docker: altinity/clickhouse-sink-connector:2.3.1-kafka
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.3.1-kafka/images/sha256-577822ac4cb14dc7e48fba32239638915742ca2a758d518e3084ef81c375892a?context=explore

What's Changed

• Update state_storage.md by @subkanthi in #771
• Update README.md to include state storage link by @subkanthi in #772
• Updated offset state storage documentation. by @subkanthi in #774
• Update state_storage.md to include schema storage. by @subkanthi in #775
• Update state_storage.md to include postgresql offsets by @subkanthi in #776
• Update quickstart.md to start sink connector service by @subkanthi in #785
• Added logic to get latest release from github by @subkanthi in #786
• Update quickstart.md with script to set environment variable. by @subkanthi in #787
• Update quickstart_postgres.md by @subkanthi in #788
• Added updates to script by @subkanthi in #789
• Added single.threaded flag to Mariadb test to validate replication in… by @subkanthi in #781
• Update production_setup.md by @subkanthi in #796
• Changed logging level to info for STRUCT EMPTY not a valid CDC record by @subkanthi in #795
• Update production_setup.md to include max_paritions_per_insert by @subkanthi in #804
• Update production_setup.md , fixed broken link by @subkanthi in #809
• Update config.ym by @subkanthi in #813
• Update README.md to include initial data dump/load by @subkanthi in #821
• 794 change logging level of struct empty not a valid cdc record + record to info by @subkanthi in #825
• 801 records are not acknowledged or the offsets are not updated in singlethreaded mode by @subkanthi in #827

Full Changelog: 2.3.0...2.3.1

Release v2.3.0

Lightweight
Docker: altinity/clickhouse-sink-connector:2.3.0-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.3.0-lt/images/sha256-0634822631c25a88a1d5fe662000dacdcf85adac6fd907765f784ac693b92df0?context=explore

Kafka
Docker: altinity/clickhouse-sink-connector:2.3.0-kafka
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.3.0-kafka/images/sha256-4e228117805895f78608db8c101071d8286f0e23370d677ba268d0dd1ff7f291?context=explore

What's Changed

• Update Monitoring.md to include metrics.port by @subkanthi in #735
• Added integration test for PostgreSQL keepermap by @subkanthi in #728
• Enable metrics by default for MySQL and postgres by @subkanthi in #750
• Added documentation to set the replication start position by @subkanthi in #753
• Added documentation for using JAR file for postgres replication by @subkanthi in #754
• Update quickstart.md by @subkanthi in #763
• Update quickstart.md update docker tag. by @subkanthi in #764
• Removed schema history configuration settings for postgres by @subkanthi in #755
• 725 we cant start grafana in sink connector we have cert issue by @subkanthi in #760
• Added JMX exporter to export JMX metrics from debezium. by @subkanthi in #757
• Disable validation of source database by @subkanthi in #716
• Added Integration test to validate truncate event replication in post…gresql by @subkanthi in #759
• Fix set lsn to accept string and not long by @subkanthi in #752
• Added logic in retrying database in case of failure by @subkanthi in #761
• 630 postgres heartbeat setup documentation by @subkanthi in #765
• 628 add integration test for mariadb by @subkanthi in #673
• Added functionality to replicate in single threaded mode based on configuration without using a Queue by @subkanthi in #756
• Convert localDateTime to String in show_replica_status API call by @subkanthi in #736

Full Changelog: 2.2.1...2.3.0

Release v2.2.1

Lightweight
Docker: altinity/clickhouse-sink-connector:2.2.1-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.2.1-lt/images/sha256-40c1fca45b6c7c66417e9b4010776b6092a8f2ba2cd5bb59e461c4de17b9c384?context=explore

Kafka
Docker: altinity/clickhouse-sink-connector:2.2.1-kafka
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.2.1-kafka/images/sha256-78571eec2d5d9103bf75a993851e051d739cf110b07160e9f294d5d3baf7f494?context=explore

What's Changed

• Update README.md by @subkanthi in #687
• Added log4j2.xml by @subkanthi in #688
• Update logging.md by @subkanthi in #690
• Update logging.md by @subkanthi in #691
• Update quickstart_kafka.md by @subkanthi in #696
• Create incremental_snapshot.md- Document steps to add newer tables. by @subkanthi in #697
• Update README.md to include incremental snapshot document by @subkanthi in #698
• Update incremental_snapshot.md - fix typo by @subkanthi in #699
• Added config with keepermap storage and removed FINAL in CREATE VIEW … by @subkanthi in #695
• Added ClickHouse trademark by @hodgesrm in #702
• Update incremental_snapshot.md by @subkanthi in #706
• Add rrmt check kafka by @Selfeer in #707
• Update incremental_snapshot.md to include schema changes by @subkanthi in #709
• Snapshot out of memory documentation by @subkanthi in #710
• Update production_setup.md for incremental snapshot size by @subkanthi in #711
• 713 numeric datatype in postgres mapped to decimal102 in clickhouse by @subkanthi in #715
• Ignore retrieval of scale/precision for enum data types by @subkanthi in #693
• Added logic to support ALTER TABLE with ALGORITHM by @subkanthi in #719

Full Changelog: 2.2.0...2.2.1