Skip to content
javahacks.dev
Microservices

How to Use the Outbox Pattern and CDC With Debezium

Publish events reliably with the outbox pattern: write an outbox row in the same transaction and stream it to Kafka with Debezium change data capture.

6 min read
Summarize in AI
Flat illustration of an editor window full of code, with the Hibernate, Apache Maven, Spring, JUnit5 and OpenJDK logos.

A service that saves an order and then publishes OrderPlaced to Kafka has a dual write problem. If the process dies between the two steps, the database and the broker disagree, and no retry loop can tell which step happened. The outbox pattern removes the second write from the transaction. This uses PostgreSQL 17, Debezium 3 and Kafka 4 with Spring Boot 4 on JDK 25.

The outbox table

SQL
create table outbox (
    id           uuid primary key,
    aggregatetype text not null,
    aggregateid  text not null,
    type         text not null,
    payload      jsonb not null,
    created_at   timestamptz not null default now()
);

Writing in the same transaction

java
@Service
class OrderService {
    private final OrderRepository orders;
    private final OutboxRepository outbox;

    @Transactional
    public Order place(NewOrder request) {
        Order order = orders.save(Order.from(request));
        outbox.save(new OutboxEvent(
                UUID.randomUUID(), "order", order.getId().toString(),
                "OrderPlaced", Json.of(new OrderPlaced(order.getId(), order.total()))));
        return order;
    }
}

Both rows commit together or neither does. The rule behind this is the transaction boundary in JDBC transactions. No message leaves the service here, which is the point.

Streaming with Debezium

Debezium runs as a Kafka Connect connector that tails the PostgreSQL write-ahead log. The Outbox Event Router transformation turns each inserted row into a Kafka message:

JSON
{
  "name": "orders-outbox",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "db",
    "database.dbname": "shop",
    "table.include.list": "public.outbox",
    "plugin.name": "pgoutput",
    "topic.prefix": "shop",
    "transforms": "outbox",
    "transforms.outbox.type": "io.debezium.transforms.outbox.EventRouter",
    "transforms.outbox.route.by.field": "aggregatetype",
    "transforms.outbox.route.topic.replacement": "events.${routedByValue}",
    "transforms.outbox.table.field.event.key": "aggregateid",
    "transforms.outbox.table.field.event.payload": "payload"
  }
}

The router sends order rows to the topic events.order, with aggregateid as the message key, so all events for one order land in one partition and keep their order, see partitions and keys. The database needs wal_level=logical, and the connector uses a replication slot, which keeps WAL until it is consumed. A stopped connector therefore makes the WAL grow, so monitor the slot's lag.

Cleaning up

The outbox is a transit table. Delete rows once captured, either by inserting and deleting in the same transaction (Debezium reads the insert from the log, so the table stays empty) or with a periodic job that removes old rows. An ever-growing outbox slows inserts.

Consuming

Delivery is at least once. A connector restart can resend events, so a consumer deduplicates on the event id:

java
@KafkaListener(topics = "events.order")
void handle(ConsumerRecord<String, String> record) {
    String eventId = new String(record.headers().lastHeader("id").value());
    if (!processed.markIfNew(eventId)) return;
    billing.charge(Json.read(record.value(), OrderPlaced.class));
}

The patterns are in idempotent consumers and reliable Kafka consumers.

Polling as a simpler alternative

Without Kafka Connect, a scheduled job can read unsent rows and publish them, marking them sent. It is simple and adds latency and load. Debezium gives low latency without queries on the table, at the price of operating Connect.

Delivery is at least once, so consumers must cope

Change data capture publishes each outbox row, and under failure it may publish one twice: after a connector restart the last offset is behind the last event that reached Kafka, and the event repeats. A consumer must treat an event it has already handled as a no-op. The simplest approach is to carry a unique event id in every message and to record processed ids in a table with a unique constraint, in the same transaction as the work the event triggers. A second delivery then fails on the constraint and is skipped. Without this, a repeated OrderPaid charges the customer twice, and the bug appears only when the connector restarts.

Ordering

Kafka orders events within a partition and not across partitions. Use the aggregate id, such as the order id, as the message key, so all events for one order go to one partition and stay in sequence. Debezium's outbox event router takes the key from a column you name. Events for different orders may arrive in any order, which is fine unless a consumer assumes otherwise.

The operational risk nobody mentions

A database with logical replication keeps its write-ahead log until the connector has read it. If the connector is down for a long time, PostgreSQL retains the log for its replication slot, and the disk fills. Monitor the lag and the retained size of the slot, alert well before the disk is at risk, and have a procedure for dropping a slot that will not recover. MySQL has the same shape with binary log retention. This is the failure that turns "we added an event stream" into an outage on the primary database, and it deserves a dashboard on day one.

Schema evolution of the payload

The payload column is a contract between services. Version it, either with a field in the event or with the event type name, and make additive changes only, so old consumers keep working. Use a schema registry with compatibility checks if you have several consuming teams, because a renamed field that nobody was told about breaks a consumer in another team at night. Keep payloads small and self-contained: include the data a consumer needs, so that it does not call back to the producing service for every event.

Testing the whole path

An outbox is a chain of four parts: your transaction, the connector, the broker and the consumer. Test each of them, and then test them together once. A Testcontainers setup can start the database, Kafka and Kafka Connect with the Debezium image, register the connector from a JSON file, insert an order in a transaction, and read the event from a topic with a consumer, with a timeout. The test takes tens of seconds, so run it in a separate CI stage, and it catches the mistakes that unit tests cannot: a wrong table name in the connector configuration, a missing column in the router settings, a serializer mismatch. Add a second test that kills the connector mid-stream and restarts it, and assert that events arrive at least once and that the consumer ends in the right state.

Cleaning up the table

The outbox table grows with every business transaction. Delete rows after Debezium has read them, either in the same transaction that inserts (Debezium reads the log, so an insert followed by a delete in one transaction still produces the event), or with a periodic job that removes rows older than a retention window. The first approach keeps the table tiny and costs one extra write. Choose a retention that covers the longest connector outage you plan to tolerate if you use the periodic delete, so a restart never needs a row that is already gone.

Where this does not apply

If the service has no database transaction to share, for example it only forwards events, the pattern has nothing to attach to. In sagas, the outbox publishes each step, see the saga pattern. The pattern is described at microservices.io, and the router in the Debezium documentation.

Also inHibernate
Taggedhow to
Share

Keep reading

Get new posts by email

We send one email when a new post is published. No spam.

Unsubscribe in one click. No spam, ever.