|
| 1 | +## Handling PostgreSQL WAL Growth with Debezium Connectors |
| 2 | +Credits: https://medium.com/@pawanpg0963/postgres-replication-lag-using-debezium-connector-4ba50e330cd6 |
| 3 | + |
| 4 | +One of the common problems with PostgreSQL is the WAL size increasing. This issue can be observed when using Debezium connectors for change data capture. |
| 5 | +The WAL size increases due to the connector not sending any data to the replication slot. |
| 6 | +This can be observed by checking the replication slot lag using the following query: |
| 7 | +```sql |
| 8 | +postgres=# SELECT slot_name, pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS replicationSlotLag, |
| 9 | +pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn)) AS confirmedLag, active FROM pg_replication_slots; |
| 10 | + slot_name | replicationslotlag | confirmedlag | active |
| 11 | +-----------+--------------------+--------------+-------- |
| 12 | + db1_slot | 20 GB | 16 GB | t |
| 13 | + db2_slot | 62 MB | 42 MB | t |
| 14 | +(2 rows) |
| 15 | +``` |
| 16 | + |
| 17 | +This issue can be addressed using the `heartbeat.interval.ms` configuration |
| 18 | + |
| 19 | +### Solution |
| 20 | +Create a new table in postgres (Heartbeat) table. |
| 21 | +```sql |
| 22 | +CREATE TABLE heartbeat.pg_heartbeat ( |
| 23 | +random_text TEXT, |
| 24 | +last_update TIMESTAMP |
| 25 | +); |
| 26 | +``` |
| 27 | +Add the table to the existing publisher used by the connector: |
| 28 | +```sql |
| 29 | +INSERT INTO heartbeat.pg_heartbeat (random_text, last_update) VALUES ('test_heartbeat', NOW()); |
| 30 | +``` |
| 31 | +Add the table to the existing publisher used by the connector: |
| 32 | +``` |
| 33 | +ALTER PUBLICATION db1_pub ADD TABLE heartbeat.pg_heartbeat; |
| 34 | +ALTER PUBLICATION db2_pub ADD TABLE heartbeat.pg_heartbeat; |
| 35 | +``` |
| 36 | +Grant privileges to the schema heartbeat and table pg_heartbeat to the replication user used by the connector. |
| 37 | +Add the following configuration to `config.yml` |
| 38 | +``` |
| 39 | +heartbeat.interval.ms=10000 |
| 40 | +heartbeat.action.query="UPDATE heartbeat.pg_heartbeat SET last_update=NOW();" |
| 41 | +``` |
0 commit comments