Release v2.5.1

Lightweight
Docker: altinity/clickhouse-sink-connector:2.5.1-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.5.1-lt/images/sha256-ac7372e68da990fc0cee91923a779a4eca31407496aa0a15b2ec4a4a52acc165

What's new:

error.max.retries: All the clickhouse database calls will be retried based on the configuration variable.
Hikari connection pool: All the clickhouse db calls will be routed through Hikari connection pool. The number of connections are configurable.https://github.com/Altinity/clickhouse-sink-connector/blob/develop/doc/connection_pool.md

What's Changed

Postgres Numeric -> BigDecimal exception
Resolved an exception occurring when mapping PostgreSQL NUMERIC data types to Java's BigDecimal, enhancing compatibility with PostgreSQL sources.
Issue #980​

DateTime64 truncation limit behavior different from ClickHouse
Addressed inconsistencies in DateTime64 truncation behavior between the connector and ClickHouse, ensuring uniform data handling.
Issue #971​

DDL Translation error: modify column datatype with DEFAULT NULL
Fixed errors in Data Definition Language (DDL) translation when modifying column data types with DEFAULT NULL constraints, improving schema evolution support.
Issue #965
​
Hikari Connection Pool
​The pull request #893 introduces the integration of the HikariCP connection pool. This enhancement aims to efficiently manage ClickHouse connections, addressing potential connection leaks identified in issue #867. By implementing HikariCP, the connector benefits from improved performance and reliability through effective connection pooling.

• Update configuration.md by @subkanthi in #957
• Update README.md by @subkanthi in #961
• Update getLatestRelease.sh- remove extra quote. by @subkanthi in #962
• Update quickstart.md to explain export statement. by @subkanthi in #963
• Update architecture.md by @subkanthi in #964
• Created replica.status.view for postgres. by @subkanthi in #970
• #380 - Added configuration settings to prevent freeze of debezium loop. by @subkanthi in #974
• Updated version of upload-artifact to v4. by @subkanthi in #978
• Add jdbc params -keepalive by @subkanthi in #988
• Update Troubleshooting.md by @subkanthi in #989
• Added support to disable connection pool. by @subkanthi in #992
• Set nanoseconds in DateTime64 max to zeros and not nines by @subkanthi in #972
• Added unit test to validate modify column followed by default NULL by @subkanthi in #966
• Fix to limit truncate bigdecimal values. by @subkanthi in #981
• Use Hikari Connection Pool for handling Clickhouse connections by @subkanthi in #893

Full Changelog: 2.5.0...2.5.1

Release v2.5.0

Lightweight
Docker: altinity/clickhouse-sink-connector:2.5.0-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.5.0-lt/images/sha256-60d5edd7d89eeee4ec64ec802c95c223096793654cce256d58c4e15219c4d0e2?context=explore

Kafka
Docker: altinity/clickhouse-sink-connector:2.5.0-kafka
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.5.0-kafka/images/sha256-4cc61265b51b4a1bdfbed244c4b074daedb852c8a665ca1b9cf19a70243eb197?context=explore

What's new:

MongoDB support: Restored mongodb config.yml and instructions to setup sink connector to replicate MongoDB databases.
Debezium upgrade to 3.0.0. - Updated Debezium to the latest stable 3.0.0.
Ignore DDL statements based on regular expressions: ignore DDL from MySQL based on regular expressions. ignore.ddl.regex
Kafka Debezium upgrade: MySQL Debezium connector was upgraded to latest stable 3.0.0.
Performance improvements: Removed additional calls to clickhouse to get table schema.
MySQL DDL Support: Alter table rename without table keyword

What's Changed

• Update README.md to include initial data dump/load by @subkanthi in #821
• 794 change logging level of struct empty not a valid cdc record + record to info by @subkanthi in #825
• 801 records are not acknowledged or the offsets are not updated in singlethreaded mode by @subkanthi in #827
• Added release notes for 2.3.1 release by @subkanthi in #830
• Release 2.3.1 by @subkanthi in #824
• Single threaded mode diagram by @subkanthi in #823
• Add Altinity Slack badge to top of readme. by @hodgesrm in #844
• Update config.yml by @subkanthi in #846
• Added production setup for MySQL by @subkanthi in #767
• 2.4.0 by @subkanthi in #798
• Database override it rrmt by @subkanthi in #877
• 861 renaming a table doesnt work for databases that are mapped to another name in clickhouse by @subkanthi in #862
• Update configuration.md by @subkanthi in #884
• Update log4j2.xml changes to have sink connector in INFO level. by @subkanthi in #892
• Remove additional call to get column data types from clickhouse by @subkanthi in #889
• Update README.md by @subkanthi in #905
• Update architecture.md by @subkanthi in #910
• Update incremental_snapshot.md by @subkanthi in #911
• Added logic to support rename column DDL with database override. by @subkanthi in #912
• Added logic to create schema history table on connector startup. by @subkanthi in #914
• Added test to cover alter table rename without table keyword by @subkanthi in #909
• Update debezium-connector-setup.sh for 'delete' operation by @salisbury-espinosa in #902
• Fix integration tests in 2.4.1 branch by @subkanthi in #920
• 895 ignore ddl based on regex by @subkanthi in #896
• Add DDL test for CREATE TABLE like by @subkanthi in #929
• 384 remove references to older config by @subkanthi in #883
• Add retries for creating debezium storage database by @subkanthi in #926
• Updated debezium to 3.0.0.Final and kafka connect dependencies to the… by @subkanthi in #887
• Removed kube-rbac-proxy references by @subkanthi in #940
• Mongodb integration test by @subkanthi in #921
• Kafka version, upgrade mysql debezium to 3.0.4.Final, set mysql to 8.0 by @subkanthi in #939

New Contributors

• @salisbury-espinosa made their first contribution in #902

Full Changelog: 2.4.0...2.4.1t

Release v2.4.0

Lightweight
Docker: altinity/clickhouse-sink-connector:2.4.0-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.4.0-lt/images/sha256-d509cafcadd11de0fd25b935efb0ee9e49429e9150ca9950c89d194e7a0b7f77?context=explore

Kafka
Docker: altinity/clickhouse-sink-connector:2.4.0-kafka
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.4.0-kafka/images/sha256-21e31d08819c6af5c52c9cf80fefe6cba979d87da41a689c66f5cbed4f4b0e5a?context=explore

What's new:

ddl retry: When the ddl.retry: "true" variable is added to config.yml, if for some reason DDL's translated by sink connector fails to execute on ClickHouse, logic has been added to retry on an exponential backoff loop.
MySQL POINT & POLYGON data types: The Polygon and Point datatypes will be mapped to ClickHouse.
MySQL DDL Support: Added support for DROP CONSTRAINT and fixed bug with dropping NULL in ALTER TABLE.
sink-connector-client Added commands to purge schema history and offset table.
Exclude Generated/Alias and Materialized: Added logic to exclude Materialized and Alias columns.

What's Changed

• Update state_storage.md by @subkanthi in #771
• Update README.md to include state storage link by @subkanthi in #772
• Updated offset state storage documentation. by @subkanthi in #774
• Update state_storage.md to include schema storage. by @subkanthi in #775
• Update state_storage.md to include postgresql offsets by @subkanthi in #776
• Testflows new kafka tests by @Selfeer in #738
• Update quickstart.md to start sink connector service by @subkanthi in #785
• Added logic to get latest release from github by @subkanthi in #786
• Update quickstart.md with script to set environment variable. by @subkanthi in #787
• Update quickstart_postgres.md by @subkanthi in #788
• Added updates to script by @subkanthi in #789
• Added single.threaded flag to Mariadb test to validate replication in… by @subkanthi in #781
• Update production_setup.md by @subkanthi in #796
• Changed logging level to info for STRUCT EMPTY not a valid CDC record by @subkanthi in #795
• Update production_setup.md to include max_paritions_per_insert by @subkanthi in #804
• Added logic to commit batch in single threaded mode by @subkanthi in #802
• Update production_setup.md , fixed broken link by @subkanthi in #809
• Added support for DROP CONSTRAINT by @subkanthi in #807
• Update config.ym by @subkanthi in #813
• Removed logic of duplicating instances of DebeziumOffsetStorage by @subkanthi in #833
• Support MySQL point by @poweroftrue in #815
• Support MySQL Polygons by @poweroftrue in #853
• Added Integration test for testing POINT data type in MySQL by @subkanthi in #851
• Upgrade debezium to 2.7.1.Final by @subkanthi in #792
• Set log retention max size to 100M by @subkanthi in #805
• Added integration test to validate database.override.map in snapshot.mode=initial mode by @subkanthi in #818
• Log docker tag in startup by @subkanthi in #841
• Upgrade clickhouse.jdbc.version to 0.6.5 by @subkanthi in #832
• Single threaded documentation by @subkanthi in #822
• add ddl.retry by @Selfeer in #865
• Refactored DDL processing logic by @subkanthi in #839
• Testflows updates by @Selfeer in #873
• Added test to replicate generated columns in postgreSQL by @subkanthi in #777
• Added function to retrieve a map of column name and is_nullable as bo… by @subkanthi in #852
• Added test for MySQL DDL with partitions by @subkanthi in #784
• 773 add sink connector client step to purge schema history so that newer tables can be added by @subkanthi in #779

New Contributors

• @poweroftrue made their first contribution in #815

Full Changelog: 2.3.0...2

Release v2.3.1

Lightweight
Docker: altinity/clickhouse-sink-connector:2.3.1-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.3.1-lt/images/sha256-4cfbc7711a0c8fd47cda0385476026ce8456fb6a0eaeac7c9455f220c044cf07?context=explore

Kafka
Docker: altinity/clickhouse-sink-connector:2.3.1-kafka
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.3.1-kafka/images/sha256-577822ac4cb14dc7e48fba32239638915742ca2a758d518e3084ef81c375892a?context=explore

What's Changed

• Update state_storage.md by @subkanthi in #771
• Update README.md to include state storage link by @subkanthi in #772
• Updated offset state storage documentation. by @subkanthi in #774
• Update state_storage.md to include schema storage. by @subkanthi in #775
• Update state_storage.md to include postgresql offsets by @subkanthi in #776
• Update quickstart.md to start sink connector service by @subkanthi in #785
• Added logic to get latest release from github by @subkanthi in #786
• Update quickstart.md with script to set environment variable. by @subkanthi in #787
• Update quickstart_postgres.md by @subkanthi in #788
• Added updates to script by @subkanthi in #789
• Added single.threaded flag to Mariadb test to validate replication in… by @subkanthi in #781
• Update production_setup.md by @subkanthi in #796
• Changed logging level to info for STRUCT EMPTY not a valid CDC record by @subkanthi in #795
• Update production_setup.md to include max_paritions_per_insert by @subkanthi in #804
• Update production_setup.md , fixed broken link by @subkanthi in #809
• Update config.ym by @subkanthi in #813
• Update README.md to include initial data dump/load by @subkanthi in #821
• 794 change logging level of struct empty not a valid cdc record + record to info by @subkanthi in #825
• 801 records are not acknowledged or the offsets are not updated in singlethreaded mode by @subkanthi in #827

Full Changelog: 2.3.0...2.3.1

Release v2.3.0

Lightweight
Docker: altinity/clickhouse-sink-connector:2.3.0-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.3.0-lt/images/sha256-0634822631c25a88a1d5fe662000dacdcf85adac6fd907765f784ac693b92df0?context=explore

Kafka
Docker: altinity/clickhouse-sink-connector:2.3.0-kafka
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.3.0-kafka/images/sha256-4e228117805895f78608db8c101071d8286f0e23370d677ba268d0dd1ff7f291?context=explore

What's Changed

• Update Monitoring.md to include metrics.port by @subkanthi in #735
• Added integration test for PostgreSQL keepermap by @subkanthi in #728
• Enable metrics by default for MySQL and postgres by @subkanthi in #750
• Added documentation to set the replication start position by @subkanthi in #753
• Added documentation for using JAR file for postgres replication by @subkanthi in #754
• Update quickstart.md by @subkanthi in #763
• Update quickstart.md update docker tag. by @subkanthi in #764
• Removed schema history configuration settings for postgres by @subkanthi in #755
• 725 we cant start grafana in sink connector we have cert issue by @subkanthi in #760
• Added JMX exporter to export JMX metrics from debezium. by @subkanthi in #757
• Disable validation of source database by @subkanthi in #716
• Added Integration test to validate truncate event replication in post…gresql by @subkanthi in #759
• Fix set lsn to accept string and not long by @subkanthi in #752
• Added logic in retrying database in case of failure by @subkanthi in #761
• 630 postgres heartbeat setup documentation by @subkanthi in #765
• 628 add integration test for mariadb by @subkanthi in #673
• Added functionality to replicate in single threaded mode based on configuration without using a Queue by @subkanthi in #756
• Convert localDateTime to String in show_replica_status API call by @subkanthi in #736

Full Changelog: 2.2.1...2.3.0

Release v2.2.1

Lightweight
Docker: altinity/clickhouse-sink-connector:2.2.1-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.2.1-lt/images/sha256-40c1fca45b6c7c66417e9b4010776b6092a8f2ba2cd5bb59e461c4de17b9c384?context=explore

Kafka
Docker: altinity/clickhouse-sink-connector:2.2.1-kafka
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.2.1-kafka/images/sha256-78571eec2d5d9103bf75a993851e051d739cf110b07160e9f294d5d3baf7f494?context=explore

What's Changed

• Update README.md by @subkanthi in #687
• Added log4j2.xml by @subkanthi in #688
• Update logging.md by @subkanthi in #690
• Update logging.md by @subkanthi in #691
• Update quickstart_kafka.md by @subkanthi in #696
• Create incremental_snapshot.md- Document steps to add newer tables. by @subkanthi in #697
• Update README.md to include incremental snapshot document by @subkanthi in #698
• Update incremental_snapshot.md - fix typo by @subkanthi in #699
• Added config with keepermap storage and removed FINAL in CREATE VIEW … by @subkanthi in #695
• Added ClickHouse trademark by @hodgesrm in #702
• Update incremental_snapshot.md by @subkanthi in #706
• Add rrmt check kafka by @Selfeer in #707
• Update incremental_snapshot.md to include schema changes by @subkanthi in #709
• Snapshot out of memory documentation by @subkanthi in #710
• Update production_setup.md for incremental snapshot size by @subkanthi in #711
• 713 numeric datatype in postgres mapped to decimal102 in clickhouse by @subkanthi in #715
• Ignore retrieval of scale/precision for enum data types by @subkanthi in #693
• Added logic to support ALTER TABLE with ALGORITHM by @subkanthi in #719

Full Changelog: 2.2.0...2.2.1

Release v2.2.0

Lightweight
Docker: altinity/clickhouse-sink-connector:2.2.0-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.2.0-lt/images/sha256-c2e2cf6d6dc59043a0cbc097ac908381779a2eaf3508ff2a0062f3beb8f2c4cb?context=explore

Breaking Changes

The environment variable LOGGING_LEVEL is no longer used for setting logging level of the application,from this release it will be set in the Root level="info" line in log4j2.xml.

    <Root level="info" additivity="false">
            <AppenderRef ref="console" />

Examples(Debug)

    <Root level="debug" additivity="false">
            <AppenderRef ref="console" />

Examples(Error)

    <Root level="error" additivity="false">
            <AppenderRef ref="console" />

Logging is configured using https://github.com/Altinity/clickhouse-sink-connector/blob/develop/sink-connector-lightweight/docker/log4j2.xml and is mounted to the docker container in https://github.com/Altinity/clickhouse-sink-connector/blob/develop/sink-connector-lightweight/docker/clickhouse-sink-connector-lt-service.yml. Refer to https://github.com/Altinity/clickhouse-sink-connector/blob/develop/doc/logging.md on setup.

What's Changed

• Upgraded to debezium 2.7.0 and update new function definition for MyS… by @subkanthi in #587
• Added log.error statements for all RunTimeException by @subkanthi in #588
• Update development.md by @subkanthi in #604
• Add TesTflows tests support for ReplicatedReplacingMergeTree engine by @Selfeer in #610
• Revert back debezium version to 2.5.0.Beta1 by @subkanthi in #616
• Added support to use user provided rmt delete column by @subkanthi in #593
• Update quickstart README by @hoanganht1k27 in #618
• Fix runners 24 4 by @Selfeer in #621
• Fixed renaming of is_deleted column when the source columns have backticks by @subkanthi in #620
• Added logic to include database name in the logs by @subkanthi in #613
• Added Integration test to validate mapping of MySQL JSON data types t… by @subkanthi in #592
• Fix fails testflows by @Selfeer in #629
• Fix broken CI/CD for TestFlows by @Selfeer in #633
• 631 add database name or UUID to rrmt engine definition when tables are auto created by @subkanthi in #632
• Resolve 2 2 0 conflicts by @Selfeer in #651
• 631 add database name or UUID to rrmt engine definition when tables are auto created by @subkanthi in #643
• 2.1.0 by @subkanthi in #647
• Database name overrides by @subkanthi in #624
• Remove clickhouse server database by @subkanthi in #614
• Added Integration test to validate mismatch of mysql columns by @subkanthi in #627
• Log exception cause when there is an exception starting connector by @subkanthi in #652
• Added production setup architecture by @subkanthi in #638
• 575 add integration test multiple schema for postgresql by @subkanthi in #579
• Return error code when there is an error retrieving show_replica_status by @subkanthi in #658
• Added logic to OVERRIDE CREATE DATABASE SQL by @subkanthi in #653
• Changes to fix grafana datasource by @subkanthi in #639
• Change MySQL docker image used in Java Integration test from latest t… by @subkanthi in #670
• Added logic to move the CREATE VIEW operation to after the connector … by @subkanthi in #668
• Avoid printing error logs when there are no records in the queue by @subkanthi in #674
• Updated documentation on handling datetime/timestamp fields by @subkanthi in #662
• Upgraded to debezium 2.7.0 and update new function definition for MyS… by @subkanthi in #590

New Contributors

• @hoanganht1k27 made their first contribution in #618

Full Changelog: 2.1.0...2.2.0

Release v2.1.0

Lightweight
Docker: altinity/clickhouse-sink-connector:2.1.0-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.1.0-lt/images/sha256-eb436979852212c7a0f624c1cc2f870ba4b173027be8e21654cdc9179b6d5590?context=explore

Kafka
Docker: altinity/clickhouse-sink-connector:2.1.0-kafka
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.1.0-kafka/images/sha256-65d4cbdcdfd159d6d7a1702635cd44da3135fa37dbab637a7a958e3339122b92?context=explore

What's Changed

Breaking Changes.

The configuration clickhouse.server.database is now deprecated with the multiple database support.
By default the source MySQL/postgres database name will be used as the ClickHouse destination database name.

Changes

• Release 2.0.2 by @subkanthi in #510
• Added release notes for 2.0.2 by @subkanthi in #527
• add 1 second delay after query execution by @Selfeer in #537
• Update README.md by @subkanthi in #539
• Update Monitoring.md by @subkanthi in #541
• Change index_granularity to 8192 instead of 8198. by @subkanthi in #534
• Refactor TestFlows tests related to Lightweight by @Selfeer in #543
• Update config.yml to include database.server.id by @subkanthi in #544
• Update Troubleshooting.md by @subkanthi in #545
• Update Monitoring.md to include insert duration query and part log query by @subkanthi in #546
• Removed references to deduplication.policy in kafka configuration by @subkanthi in #547
• Use sequence number + timestamp in non-gtid mode for version column. by @subkanthi in #550
• Added logic to support multiple databases by @subkanthi in #535
• 523 handle scenario when records could be inserted with the same timestampnon gtid mode by @subkanthi in #552
• [528] Added logic to create view for replica_source_info table by @subkanthi in #549
• Replaced slf4j calls with log4j2 api calls by @subkanthi in #553
• Can't load table from Postgres to Clickhouse containing nullable numeric column by @ZlobnyiSerg in #529
• Kafka fixes for multiple database. by @subkanthi in #555
• Added integration test to perform updates on PK to verify incrementin… by @subkanthi in #554
• Enable postgres tests by @subkanthi in #556
• Grafana - Fix prometheus targets by @subkanthi in #558
• Fixed logic of creating sequence number based on debezium timestamp, … by @subkanthi in #557
• Removed excessive logging statements by @subkanthi in #565
• Changed CREATE VIEW to CREATE VIEW IF NOT EXISTS by @subkanthi in #567
• Fix alter drop column by @subkanthi in #560
• Force RMT to old version for Integration tests by @subkanthi in #571
• Changed from ts_ms to debezium ts_ms for adding sequence numbers by @subkanthi in #569
• Fixed alter table change column not null DDL query by @subkanthi in #573
• Changes to make sure the threads are exited when the CLI stop command… by @subkanthi in #525
• remove old broken tests by @Selfeer in #585

New Contributors

• @ZlobnyiSerg made their first contribution in #529

Full Changelog: 2.0.2...2.1.0

Release v2.0.2

Lightweight
Docker: altinity/clickhouse-sink-connector:2.0.2-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.0.2-lt/images/sha256-1c0db9877331a0aaceb2e0f2838db3a5156a3cfbfdb83a92235b6287b576c5d0?context=explore

Kafka
Docker: altinity/clickhouse-sink-connector:2.0.2-kafka
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.0.2-kafka/images/sha256-66e786bbd1230c26abdd0bad70b549abdb1040eab02a25b41c7c7217f51cf0bf?context=explore

What's Changed

• Added release notes for 2.0.1 by @subkanthi in #504
• Update README.md with latest release 2.0.1 by @subkanthi in #505
• 506 datetime limits were updated in jdbc libraryclickhouse java by @subkanthi in #508
• Update config.yml to set database.connectionTimezone by @subkanthi in #514
• Removed log4j 1.x dependencies by @subkanthi in #509
• Updated troubleshooting documentation by @subkanthi in #517
• Added sample application to create 5000 sample records. by @DougTidwell in #521
• Various fixes to the MySQL python commands by @aadant in #515
• ReplicatedReplacingmergetree support for kafka by @subkanthi in #522
• When MAX_QUEUE_SIZE is configured, the this.records.add(batch) method may throw an IllegalStateException and terminate the task. by @UsenPang in #512
• Release 2.0.2 by @subkanthi in #510

New Contributors

• @DougTidwell made their first contribution in #521
• @UsenPang made their first contribution in #512

Full Changelog: 2.0.1...2.0.2

Release v2.0.1

Lightweight
Docker: altinity/clickhouse-sink-connector:2.0.1-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.0.1-lt/images/sha256-99f38d1b5c34ac8bbcc86a86ca1eadca34e984c77ff1e2b3a54d5ee612029037?context=explore

What's Changed

• Updated README with 2.0.0 release assets by @subkanthi in #481
• 482 rearrange main readme documentation by @subkanthi in #483
• Doc updates for testflows by @Selfeer in #485
• update requirements by @Selfeer in #486
• Update documents for testflows by @Selfeer in #496
• Changed from ConcurrentLinkedQueue to LinkedBlockedQueue with the size configurable by @subkanthi in #491
• Added logic to auto create RRMT tables by @subkanthi in #492
• Separate testflows requirements by @Selfeer in #498
• update requirements by @Selfeer in #499
• update requirements by @Selfeer in #500
• Fix integration tests by @subkanthi in #502
• Release 2.0.1 by @subkanthi in #497

Known issues

• Error running DDL Query - when trying to create a new ReplicatedReplacingMergeTree table #503