Postgres CDC in Production: Handling Failover, Retries, and Duplicates

PostgreSQL CDC is easy to demonstrate and surprisingly difficult to operate. A connector can be healthy while a replication slot is retaining dangerous amounts of WAL, a failover can succeed while the downstream sink replays the last few events, and a retry can either protect you from data loss or trigger the same business side effect twice. The problem is not that CDC is unreliable by definition. The problem is that several independent systems must agree on what “processed” means.

This guide explains how to run Postgres CDC in production when the normal path breaks. It connects PostgreSQL logical decoding, Debezium offsets, Kafka delivery, failover slots, retry boundaries, duplicate-safe sinks, transaction ordering, snapshots, and operational monitoring into one practical reliability model. You will learn what PostgreSQL 17+ failover slots solve, why duplicates are often the safer failure mode, how to choose the right idempotency pattern, and which signals prove that a pipeline has actually recovered.

The examples use PostgreSQL and Debezium terminology, but the underlying rules apply to any CDC stack with a durable source log, an asynchronous connector, and a downstream consumer. Treat configuration examples as version- and environment-sensitive: validate them in staging against your exact PostgreSQL, Debezium, transport, and sink versions before using them during a production incident.

Scope and assumptions:

The examples in this guide target PostgreSQL 17+ with the current Debezium PostgreSQL connector. Behavior can differ across older PostgreSQL versions, managed services, connector releases, converters, and sink implementations. Validate every configuration in a staging environment that matches your production topology.

Why CDC in production fails at the boundaries

A local demo makes PostgreSQL change data capture look simple: create a replication slot, start Debezium, publish events to Kafka, and consume them downstream. The difficult cases begin when those components disagree about progress.

The database knows one position in its write-ahead log. The connector persists another position in its offset store. Kafka tracks records and consumer offsets. The sink tracks what it has committed, and an external API may have already acted on an event before the consumer crashes. Each layer can be healthy while the overall pipeline is not.

That is why the most useful production question is not “Does CDC support exactly-once delivery?” It is: What happens if the process dies immediately after each irreversible step?

PostgreSQL’s own logical decoding documentation gives the key detail: a logical slot normally emits each change once, but its current position is persisted at checkpoint. After a database crash, the slot can return to an earlier LSN and send recent changes again. The client is responsible for preventing harmful effects from processing those messages twice.

This is not a defect to hide with a vague “retry” setting. It is a recovery contract. A system that replays an event after an uncertain failure is often safer than one that moves its checkpoint early and silently loses the event.

Production rule: Prefer a recoverable duplicate over an unrecoverable gap. Then make the sink idempotent enough that replay is boring.

The four-state reliability model

Treat Postgres CDC as four linked state machines rather than one pipeline. During an incident, ask which state advanced, which state did not, and whether the two can be reconciled.

The distinction matters during failover. A synchronized source slot can preserve the ability to continue reading changes from a promoted standby, but it does not know whether your sink committed the last event before the connector crashed. The source can be continuous while the downstream application still sees a replay.

What PostgreSQL failover slots solve

PostgreSQL 17 introduced the current failover-slot workflow that Debezium documents for PostgreSQL 17 and later. With the connector configured for a failover-enabled slot, and with the primary configured to synchronize that slot to the standby, a promoted standby can provide the slot needed for the connector to continue reading changes.

The important word is configured. A failover slot is not a magic backup of every possible CDC state. PostgreSQL’s logical replication failover procedure says that slot synchronization is asynchronous. You must confirm that the required slots exist on the standby and are ready before promotion. The standby also needs to be ahead of the subscriber for a successful transition.

For a Debezium PostgreSQL connector, the current documentation describes these core settings and conditions:

1. The source primary runs PostgreSQL 17 or later.
2. The connector configuration sets slot.failover=true.
3. The primary’s synchronized_standby_slots includes the Debezium slot name.
4. The slot is persistent, synchronized, and not invalidated on the standby.
5. The connector’s connection route reaches the new primary after promotion.

The following is a PostgreSQL configuration fragment, not a complete production configuration. Apply it through your normal configuration-management process and verify the standby-side synchronization settings in the current PostgreSQL documentation.

# Debezium PostgreSQL connector properties
name=orders-cdc

connector.class=io.debezium.connector.postgresql.PostgresConnector
database.hostname=postgres-writer.example.internal
plugin.name=pgoutput
slot.name=debezium_orders
slot.failover=true
publication.name=dbz_orders

# postgresql.conf on the primary
# Include the logical slot used by the connector.
synchronized_standby_slots = 'debezium_orders'

After the configuration is applied, inspect the slot on the relevant server. The following query uses current pg_replication_slots columns documented by PostgreSQL:

SQL:

SELECT
    slot_name,
    active,
    restart_lsn,
    confirmed_flush_lsn,
    wal_status,
    safe_wal_size,
    invalidation_reason,
    failover,
    synced
FROM pg_replication_slots
WHERE slot_name = 'debezium_orders';

Before a planned promotion, the PostgreSQL documentation uses a readiness expression equivalent to the following:

SQL:

SELECT
    slot_name,
    (synced AND NOT temporary AND invalidation_reason IS NULL)
        AS failover_ready
FROM pg_replication_slots
WHERE slot_name = 'debezium_orders';

A result of failover_ready = true is a necessary readiness signal, not a full disaster-recovery test. It does not prove that the connector can authenticate through the new route, that the promoted server contains every required WAL record, or that the sink will not replay the last few events. Those are separate gates.

What failover slots do not solve

Failover slots address source continuity. They do not solve four other problems:

• The endpoint problem: The connector still needs a connection string, DNS record, proxy, or service endpoint that reaches the new primary.
• The checkpoint problem: An event can be committed to the sink while the connector’s offset is still behind it. That event may be replayed.
• The side-effect problem: An upsert may be harmless on replay, while an email, payment, webhook, or inventory reservation is not.
• The version problem: A PostgreSQL 10–16 fleet cannot be treated as if it had PostgreSQL 17+ failover-slot behavior.

For planned failover, stop or fence writes and CDC consumers according to your topology, validate the standby slot state, promote the standby, move the connector route, and then verify both the stream and the sink. Do not declare success because the connector reports RUNNING.

PostgreSQL version matrix

The advice below separates documented capability from deployment work. It is a planning matrix, not a substitute for your PostgreSQL and Debezium version matrix.

Debezium’s PostgreSQL connector documentation and its 3.0.5 release notes should be checked together. A property introduced in a recent connector release does not make an older database, managed service, or deployment topology equivalent to the current reference architecture.

The retry timeline that creates duplicates

The most useful mental model for retries is a timeline. Assume the connector reads event E, the consumer writes it to a sink, and the process dies before its source position is durably advanced.

The dangerous ordering is the reverse: advance the checkpoint, then write to the sink. A crash in that gap can create silent loss. The exact mechanics vary by connector, Kafka Connect worker, transport, and sink, so treat the sequence as a correctness rule for your consumer transaction, not as a claim that every component exposes the same acknowledgement API.

A second source of duplicates is snapshot overlap. Debezium documents that an ad hoc snapshot can have a delay between the signal and the time streaming stops, so events may be emitted after the snapshot that duplicate records captured by the snapshot. Your deduplication design must therefore handle both crash replay and snapshot/read-event overlap.

Duplicate-safe sink patterns

No single deduplication method fits every CDC workload. Start by deciding whether the sink is maintaining current state, recording an audit history, or triggering an external side effect.

Pattern 1: Idempotent upsert for current state

If the target represents the latest state of a row, an upsert keyed by the source primary key is a sensible foundation. It is not a complete duplicate strategy for every event type, but it makes a replay of the same state harmless.

The following illustrative PostgreSQL sink example assumes that the target stores the source LSN in a pg_lsn column. Adapt the types and mapping to your sink; do not paste this into a non-PostgreSQL database.

SQL:

INSERT INTO customer_state (
    customer_id,
    email,
    status,
    last_source_lsn
)
VALUES (
    :customer_id,
    :email,
    :status,
    :source_lsn::pg_lsn
)
ON CONFLICT (customer_id) DO UPDATE
SET
    email = EXCLUDED.email,
    status = EXCLUDED.status,
    -- Note: pg_lsn is PostgreSQL-specific; this example is not portable SQL.
    last_source_lsn = EXCLUDED.last_source_lsn
WHERE customer_state.last_source_lsn IS NULL
   OR customer_state.last_source_lsn < EXCLUDED.last_source_lsn;

The LSN guard prevents an older replayed update from replacing a newer state. It assumes that all events for the row come from the same PostgreSQL source and that the connector preserves a comparable source position. If your system merges multiple sources, use a business version, source ID plus version, or another explicitly defined ordering contract.

Pattern 2: Unique event ledger for audit and commands

A current-state upsert cannot tell you whether a business event has already triggered a side effect. For that, persist a stable event identity with the effect in the same database transaction.

SQL:

CREATE TABLE applied_cdc_events (
    event_id text PRIMARY KEY,
    applied_at timestamptz NOT NULL DEFAULT now()
);

The consumer should begin a transaction, attempt to insert the event ID with ON CONFLICT DO NOTHING, and apply the business effect only when the insert succeeds. Commit the ledger row and the effect together. If the process crashes before commit, the event is eligible for retry; if it crashes after commit, the duplicate event finds the existing key and does not repeat the effect.

This pattern protects effects that are represented in the same transactional database. It cannot make an already-sent external HTTP request atomic with your database. For external systems, use an idempotency key accepted by the receiver, a durable outbox, or a reconciliation process.

Pattern 3: Outbox events with explicit identity

The Debezium Outbox Event Router uses an event ID, an aggregate ID, an event type, and a payload in its documented example. The event ID is available in the emitted message headers and can be used to remove duplicates. The aggregate ID becomes the message key in the default pattern, which helps keep related events in the same Kafka partition.

That is a different contract from raw row-level CDC. Raw CDC answers “what changed in this table?” An outbox event answers “what business fact should another service act on?” Mixing the two without naming the contract can create duplicate business events, confusing topic ownership, and difficult replay rules.

Ordering, transactions, and the outbox boundary

PostgreSQL commits transactions; Kafka transports records; consumers apply effects. Those are related ordering domains, not one universal order.

For state replication, you often need per-entity ordering more than global ordering. Use a stable key that routes all changes for an entity to the same partition, then keep consumer-side parallelism consistent with that choice.

The Debezium transaction metadata documentation describes BEGIN and END metadata events and a transaction identifier built from the PostgreSQL transaction ID and the LSN of the operation.

Transaction metadata can help a downstream processor buffer or group changes when application semantics require transaction boundaries. It does not make a multi-system workflow atomic. A sink that reads a BEGIN marker still needs a durable transaction model and a policy for what happens when the stream is interrupted inside that boundary.

Two practical rules follow:

1. Do not promise global ordering unless you can prove it. State the exact guarantee: per key, per partition, per table, or transaction-aware processing.
2. Do not use an outbox as decoration. Capture the outbox table selectively, define the event ID, choose the routing key, and decide how consumers retire or reconcile processed events.

If the pipeline has bidirectional CDC, define domain ownership before enabling it. A write replicated from service A to service B can be captured again and sent back to A, creating an update loop or reconciliation storm. One system should be authoritative for a domain at a given stage of a migration, or the design must include explicit loop-prevention metadata and conflict rules.

Snapshots, schema changes, and long transactions

CDC correctness begins before the first streamed event. Debezium’s PostgreSQL connector performs a consistent initial snapshot because PostgreSQL normally does not retain the entire database history in WAL. It then continues streaming from the position associated with the snapshot.

A connector that stops during an initial snapshot can begin a new snapshot when it restarts. That is why snapshot events must be treated as a distinct operation type, not blindly mixed with live updates.

Debezium’s incremental snapshots reduce the need to stop streaming for the entire capture, but they still create a collision window: a streamed update can arrive before the corresponding snapshot READ event, and the connector must resolve which record should be emitted.

Use a snapshot plan that answers four questions:

1. How much source storage can be consumed while the snapshot runs?
2. What happens if the connector fails halfway through?
3. How will the sink distinguish a snapshot READ from a live UPDATE?
4. What schema and primary-key changes are forbidden during the capture window?

If the downstream target is a lakehouse or a Parquet landing zone, snapshot and cutover should be treated as a table-ownership change rather than a file-copy exercise. Our guide on migrating Parquet data to Iceberg expands the same validation, backfill, rollback, and ownership questions for the sink side of the pipeline.

Schema evolution deserves its own gate. The current Debezium PostgreSQL connector documentation notes that logical decoding does not support DDL changes as row-change events. Do not assume that adding, renaming, or removing a column will be handled by the same path as an INSERT or UPDATE. Coordinate source DDL, connector compatibility, schema registry policy, and sink migrations.

Long transactions create another boundary problem. The source may not expose a useful committed event until the transaction commits, then the connector can receive a burst. Monitor commit-time lag, not only average event latency. A pipeline can look idle for minutes and then appear to “fall behind” when a large transaction becomes visible.

Schema evolution, throughput, and alternative architectures

Reliability and performance are not separate concerns in a CDC pipeline. A queue that absorbs a Kafka slowdown can protect the source, but it also increases memory use and recovery distance. A schema that is technically valid can still be unusable by an older sink. Treat both as compatibility contracts that must be tested before production rollout.

Schema Registry is a contract gate, not a magic migration layer

With Kafka Connect converters that use a registry-backed format, a Debezium event can carry a schema identifier that the consumer resolves from the registry. The exact wire behavior depends on the converter and serialization format; JSON without schemas, Avro, Protobuf, and JSON Schema do not have identical evolution rules.

The Debezium PostgreSQL connector documentation explains the converter-dependent schema representation, while the Confluent Schema Registry compatibility documentation defines the compatibility modes.

A useful operating model is to treat a source DDL migration, a registered event schema, and a sink deployment as one change. For example, changing an INT field to BIGINT may be a widening conversion in one schema format, but its acceptance and reader behavior still depend on the serializer, field definition, compatibility mode, and sink language.

Do not infer safety from the database migration alone. If the sink evolves from files into a table contract, our explanation of how Iceberg and Parquet solve different layers provides useful downstream context; it does not replace testing the CDC serializer and sink.

Confluent’s default is BACKWARD, and it is non-transitive. That default is not automatically the right choice for every CDC topic. Decide whether consumers replay from the beginning, whether old sinks remain active during a rollout, and whether the topic is a state log or an event stream. An incompatible registration should fail at the registry gate; a schema that passes registration can still fail later in a sink that cannot deserialize or apply the new type.

A safe rollout for a type change is usually: add a new field with a compatible representation, dual-write or backfill it, deploy readers that understand both fields, validate historical replay, and only then retire the old field. If the change is genuinely incompatible, create a new subject or topic and migrate consumers instead of forcing two incompatible contracts into one stream.

Performance tuning: find the slow boundary before changing knobs

When a connector falls behind, first identify where progress stops. Compare the source LSN position, Debezium’s source lag, the connector queue, Kafka producer throughput and request latency, worker CPU and garbage collection, broker partition throughput, and sink consumer lag.

If the source LSN advances normally but the worker’s queue stays full, the problem is likely downstream of PostgreSQL. If the queue is empty and source lag grows, investigate the database, network, connector task, or replication connection instead.

The current Debezium PostgreSQL connector reference documents these useful controls:

A conservative example is:

JSON:

{
  "max.batch.size": "2048",
  "max.queue.size": "8192",
  "max.queue.size.in.bytes": "67108864",
  "snapshot.fetch.size": "10240"
}

These are starting points, not universal performance recommendations. The Debezium connector reference requires max.queue.size to be larger than max.batch.size, and documents the queue as a backpressure mechanism when Kafka is unavailable or slower than ingestion. Change one class of variable at a time, replay a representative workload, and record source lag, queue occupancy, heap use, producer latency, and sink catch-up time.

Do not treat queue growth as proof that PostgreSQL is slow. A Kafka Connect worker can be CPU-bound by serialization or transformations, constrained by producer acknowledgements, paused by a rebalance, or sharing resources with unrelated connectors.

Likewise, increasing the queue can hide a worker bottleneck until the process reaches its heap limit. If the worker is the bottleneck, scale or isolate workers and tasks, reduce expensive transforms, review producer and broker limits, and fix sink throughput before increasing source-side retention.

Large transactions: the OOM warning that a batch setting cannot solve

A PostgreSQL transaction is atomic at the source. A connector may process its row changes in batches, but that does not turn one source transaction into several independent transactions. There is no confirmed current Debezium PostgreSQL property named max.transaction.size that safely splits an already-committed PostgreSQL transaction for downstream consumers.

Large transactions can still create memory pressure because the connector, serializer, queue, broker, or sink may handle a large burst before the system reaches a stable checkpoint.

The practical controls are workload design and bounded resources: break application bulk work into smaller source transactions when the business operation allows it, size the Kafka Connect heap deliberately, cap queue bytes as well as record count, tune snapshot reads independently, and load-test the largest realistic transaction. Monitor GC pauses, resident memory, queue occupancy, connector restarts, and the time required to drain a burst.

Do not “fix” an OOM by blindly raising max.queue.size. If the queue is already the memory consumer, a larger queue makes the failure later but potentially larger. If the business transaction must remain atomic, protect the worker with heap headroom and backpressure, then verify that the sink can apply the burst without violating its own timeout and idempotency rules.

Choosing an alternative to Debezium plus Kafka Connect

Debezium plus Kafka Connect is a strong fit when you want an open event stream, Kafka-native fan-out, connector-level offsets, and control over the worker and sink ecosystem. It is not the only reasonable architecture. The right choice depends on whether the primary requirement is database migration, stream processing, or a long-lived event platform.

The AWS DMS PostgreSQL source documentation describes PostgreSQL logical replication slots, source requirements, and CDC limitations. The AWS DMS CDC documentation distinguishes full-load-plus-CDC from CDC-only tasks and notes that CDC latency is not guaranteed to be real-time. The Apache Flink CDC PostgreSQL connector documentation describes snapshot and incremental change reading, checkpoint-related options, and the connector’s PostgreSQL configuration.

When CDC lands in an analytical table rather than a raw Kafka topic, the next decision is table-format and catalog ownership. Our guide to Iceberg with Snowflake production architecture covers the downstream questions around external catalogs, schema evolution, multi-engine access, and maintenance ownership. If the destination decision is still open, our Iceberg vs. Delta Lake vs. Hudi comparison frames that choice by workload and operating model rather than by feature checklists.

Choose by operating model, not by a simplistic “fastest tool” claim. If your team already runs Kafka and needs many independent consumers, Debezium may be the cleanest boundary. If the immediate goal is a managed migration with minimal platform ownership, DMS may be more appropriate. If CDC is one stage in a stateful streaming computation, Flink CDC may reduce architectural handoffs. In every case, test failover, retries, schema changes, duplicates, and recovery from the actual checkpoint boundary.

For a concrete alternative to a self-managed Kafka-centered path, see our PostgreSQL-to-ClickHouse CDC architecture. It is especially useful for comparing capture, durable intermediate storage, and destination materialization, but its Estuary and ClickHouse behavior should not be generalized into a Debezium guarantee.

Production gotchas most guides omit

The failure model becomes more realistic when you include the details that are easy to miss in a first implementation. These are not reasons to enable every setting globally. They are signals that tell you where a “healthy” CDC pipeline can still lose context, retain WAL, or confuse a downstream consumer.

1. Low-traffic databases may need heartbeats

A connector can be running while its captured database produces too few relevant events to advance the slot frequently. This is especially important when several PostgreSQL databases share one instance: WAL is shared at the instance level, while logical replication slots are associated with individual databases. Current Debezium PostgreSQL documentation describes periodic heartbeats and an action query for this situation.

A table-based heartbeat must be included in the PostgreSQL publication. An alternative is an action query that emits a logical message. For example, the replication-slot guidance from Gunnar Morling shows this Debezium configuration pattern:

JSON:

{
  "heartbeat.interval.ms": "60000",
  "heartbeat.action.query": "SELECT pg_logical_emit_message(false, 'heartbeat', now()::varchar)"
}

The exact heartbeat method depends on your connector version, output plugin, permissions, and publication. If you use pg_logical_emit_message, the Debezium database user needs permission to execute that PostgreSQL function. Test the resulting message and confirm that the slot’s progress changes; do not assume that a connector heartbeat automatically proves that all captured tables are healthy.

2. TOAST values can be intentionally unavailable

PostgreSQL stores oversized values such as large text or jsonb values using TOAST storage. When an unchanged TOASTed column is not part of the table’s replica identity, it may not be present in an UPDATE or DELETE change event. Debezium cannot safely fetch that missing value from the database after the fact because the row may have changed again, so it emits the configured unavailable.value.placeholder value instead.

That placeholder is configurable. Do not hard-code __debezium_unavailable_value as if it were universal. Teach the sink to recognize the configured sentinel, or select a replica-identity strategy that matches the sink’s need for full row images.

REPLICA IDENTITY FULL makes previous values of all columns available for UPDATE and DELETE events according to the Debezium replica-identity documentation. It is a compatibility choice, not a free repair. Larger old-row images can increase event and WAL volume, so measure the source, transport, and sink impact on the specific tables that need it. Prefer a suitable primary key or index identity when that is sufficient for the consumer.

3. A slot can hold back catalog cleanup, not only WAL

Replication slots retain resources that their consumers may still need. PostgreSQL documents that a slot can retain WAL and system-catalog rows; in extreme situations, retention can contribute to catalog bloat or force the database to shut down to protect against transaction ID wraparound. Debezium also lists catalog bloat as a risk of leaving a slot unused for too long.

This is why slot monitoring should include catalog_xmin, restart_lsn, confirmed_flush_lsn, wal_status, and safe_wal_size. The correct response is not always “restart the connector.” First identify whether the slot is still required, whether the consumer can catch up, and whether the source has enough disk and transaction-age headroom to wait.

4. A primary-key change can change the Kafka ordering key

When an UPDATE changes a primary key, Debezium emits a DELETE for the old key followed by a CREATE for the new key, and includes __debezium.newkey and __debezium.oldkey headers. The current connector documentation describes this behavior explicitly.

That means per-key ordering has a boundary: the old and new keys are different.

If your partitioner maps them to different Kafka partitions, a consumer cannot treat the pair as one continuously keyed entity. If primary-key mutation is allowed in the source model, handle the delete/create pair deliberately, preserve the old and new identities, and do not use a primary-key-only ordering assumption across the change.

5. Keep the replication connection session-pinned

PgBouncer transaction pooling assigns a server connection only for the duration of a transaction, while PostgreSQL’s replication protocol uses a replication-mode connection and streams data through a long-lived session.

PgBouncer’s pooling documentation describes the transaction-pooling boundary directly.

Do not put a Debezium streaming replication connection behind a transaction pooler and assume it behaves like an ordinary short-lived SQL client.

Give the connector a direct, session-pinned route to PostgreSQL, or use a pooler mode and configuration that your exact connector and protocol path explicitly support. Keep ordinary application connection pooling separate from the replication connection.

6. Deletes produce tombstones as a second message

A hard delete is not necessarily one Kafka record. Debezium emits a delete event and, by default, follows it with a same-key tombstone whose value is null. As the Debezium PostgreSQL connector documentation explains, the tombstone lets Kafka log compaction remove older records for that key. Consumers must therefore distinguish at least three cases: a normal row event with a non-null value, a delete event with delete metadata, and a tombstone with a null value.

A deserializer that assumes every record has a row-shaped value can fail on the tombstone or accidentally treat it as an empty row. If a sink does not need tombstones, configure tombstones.on.delete deliberately rather than filtering nulls without understanding the compaction consequence.

7. max_slot_wal_keep_size is a safety valve, not a recovery strategy

PostgreSQL’s current replication configuration documentation sets max_slot_wal_keep_size to -1 by default, which allows replication slots to retain an unlimited amount of WAL. A configured limit caps the WAL that slots may retain at checkpoint time. If a slot's restart_lsn falls too far behind, required WAL may be removed and the slot may no longer be able to continue replication.

For example, a deliberately chosen cap might look like this:

# postgresql.conf — example only; size it from workload and recovery objectives
max_slot_wal_keep_size = 50GB

This protects disk availability at the cost of a possible history gap. If the slot becomes unusable, recovery may require a new slot and a fresh snapshot; the cap does not reconstruct the missing changes. Size it from WAL generation rate, disk headroom, acceptable recovery time, and the time needed to restore the consumer. Alert before the cap is reached by watching the pg_replication_slots fields documented by PostgreSQL.

8. Managed PostgreSQL can change the failover answer

Do not transfer a community PostgreSQL 17 failover-slot procedure directly to every hosted service. In the AWS Aurora scenario described in the AWS Database Blog, sync_replication_slots is not exposed as a modifiable parameter and the described writer-failover path still requires slot recreation and Debezium reconnection. That is a provider-specific caveat, not a universal statement about managed PostgreSQL.

The operational rule is simple: check the provider’s failover, slot, endpoint, and parameter-support documentation before declaring a connector topology failover-ready.

Observability that catches the real failure

A green connector status is not a reliability proof. Build a dashboard that joins source, connector, transport, and sink signals.

Set thresholds from your WAL generation rate, disk headroom, recovery time objective, and sink recovery time. Do not copy a community alert such as “30 minutes inactive” without calculating what 30 minutes means for your workload.

Failure injection should be part of acceptance testing. Stop the connector after a sink commit but before offset flush. Kill it before the sink commit. Interrupt the network after an uncertain acknowledgement. Promote the standby during low traffic. Replace a standby and wait for slot resynchronization. Then verify not only that the connector resumes, but also that the sink has the right final state and that a one-time side effect ran once.

Production recovery runbooks

Runbook A: Planned primary promotion

Before promotion, confirm that the standby is receiving physical WAL, the required logical slots exist on the standby, synced is true, the slot is persistent and not invalidated, and the standby is ahead of the downstream consumer. Confirm that the connector’s endpoint will resolve to the promoted primary. If the topology has a planned maintenance procedure, pause writes or fence the old primary according to that procedure.

Promote the standby, update or validate the writer endpoint, and start the connector against the new primary. Record the last known source LSN and the first LSN observed after reconnect. Expect a replay window if the stored position precedes the sink’s last commit. Measure the duplicate count and verify the sink’s idempotency behavior.

Runbook B: Connector crash or network interruption

Do not immediately delete the slot or offsets. First capture the connector logs, stored offset, slot state, restart_lsn, confirmed_flush_lsn, WAL status, and sink acknowledgement history. Check whether the required WAL remains available. Restart the connector using the same slot and offset configuration, then compare the replayed records with the sink ledger or version guard.

If you create a new slot without preserving the old source position, you may have created a gap. Debezium’s documentation warns in its upgrade guidance that a new slot can provide only changes after its creation; it cannot supply historical changes before that point. Slot replacement is therefore a recovery decision, not a routine restart step.

Runbook C: WAL growth or slot invalidation

Treat rapid WAL growth as a source-capacity incident. Identify the slot responsible, confirm whether a connector is active, compare restart_lsn and confirmed_flush_lsn, and check wal_status, safe_wal_size, and invalidation_reason. Restore the consumer if the history is still available. If the slot is no longer needed, remove it only after confirming ownership and retention consequences.

Never solve a full disk by dropping an unknown slot during an incident without recording what data that slot represented. A fast cleanup can turn a recoverable backlog into an irreversible history gap.

Runbook D: Schema or event-contract mismatch

Pause the affected consumer if it can apply incorrect state. Identify whether the event was a snapshot record, a row change, an outbox event, a heartbeat, or transaction metadata. Compare the source schema, connector schema, serialized message, and sink migration. Resume only after you have a compatibility plan and a replay strategy.

Common mistakes

Treating “exactly once” as an end-to-end property

A source slot, connector, transport, and sink can each expose a different durability boundary. Say exactly what is guaranteed and where. In most production designs, the practical goal is at-least-once delivery plus idempotent application, not a slogan that hides external side effects.

Advancing offsets before durable application

This is the easiest way to trade visible duplicates for silent loss. Make the acknowledgement order explicit in code and test the crash window.

Using only a primary key for every kind of deduplication

A primary key can make a state upsert repeatable. It does not stop an audit event from being inserted twice or an HTTP request from being sent twice. Choose the identity and atomicity boundary for the effect.

Assuming failover readiness from continuous traffic

A slot may look healthy while the system is busy and become unusable after a quiet period, standby replacement, or configuration drift. Test failover when traffic is low and after the standby has been rebuilt.

Running two event contracts without documenting ownership

Raw table CDC and outbox events can both be valid, but consumers must know which one represents state and which one represents a business fact. Give them distinct topics, schemas, ownership, and replay rules.

Monitoring only connector uptime

A connector can be RUNNING while source lag, retained WAL, transport backlog, or sink failures grow. Reliability monitoring must cross the connector boundary.

Handing production secrets to a tutorial

The examples here use placeholders. Keep credentials out of connector files where your platform supports a secret provider, use a dedicated replication user with the minimum required privileges, and verify the current Debezium PostgreSQL permissions guidance before deployment.

A production-ready definition of “recovered”

A CDC pipeline is not recovered when the connector reconnects. It is recovered when the source slot is valid, the connector is reading from the intended position, the transport is draining, the sink has applied the final state, and any replayed event has been absorbed without repeating an irreversible side effect.

That definition changes the way you design the system. Failover slots are worth using because they reduce the chance of a source-history break. Retries are worth keeping because they reduce the chance of silent loss. Duplicates are worth expecting because uncertainty is unavoidable at distributed acknowledgement boundaries. The durable answer is not to pretend the boundaries disappear; it is to make every boundary observable and every replay safe.

About The Author

A Gadallh

Ahmed Gadallah is the Founder and Editor of Vertex Frontier, where he publishes research-driven articles on AI, data science, cloud computing, cybersecurity, software engineering, and emerging technologies, with a focus on technical accuracy, clarity, and practical insights.

View all articles by A Gadallh →