Skip to content
javahacks.dev
Microservices

Duplicate Messages Idempotent Consumer Design in Practice

Handle duplicate and out of order messages: a processed message table, natural keys, versioning and a design that ignores delivery order.

6 min read
Summarize in AI
Flat illustration of a producer, a topic with partitions and consumers, with the Apache Kafka, Spring Boot, RabbitMQ, Docker and Redis logos.

A broker that promises never to lose a message must sometimes send it twice: after a timeout, a consumer crash before the commit, a rebalance, a producer retry. A design that assumes exactly one delivery breaks in production. This applies to Kafka, RabbitMQ, SQS and any similar system.

Make the effect idempotent

An operation is idempotent when doing it twice has the same effect as once.

OperationIdempotent?Fix
set status = 'PAID'YesNone
balance = balance + 10NoDedupe by message id
insert into payment ...NoUnique key, ignore conflicts
Send emailNoRecord that it was sent
delete where id = ?YesNone

The easiest case is an update that sets a value. The hard ones add, append or notify.

The processed-messages table

SQL
create table processed_message (
    message_id   varchar(64) primary key,
    consumer     varchar(64) not null,
    processed_at timestamptz not null default now()
);
java
@Transactional
public void handle(PaymentReceived event) {
    int inserted = jdbc.sql("""
            insert into processed_message (message_id, consumer)
            values (:id, 'ledger') on conflict do nothing
            """).param("id", event.messageId()).update();

    if (inserted == 0) {
        log.info("duplicate {} ignored", event.messageId());
        return;
    }
    ledger.credit(event.account(), event.amount());      // same transaction
}

The effect and the record of it commit together. If the transaction rolls back, the id is not recorded, and a retry processes it. If a duplicate arrives later, the insert conflicts and the handler returns. The message id comes from the producer: generate a UUID when creating the event, not when sending, so a retry of the same event has the same id.

Clean old rows after a period longer than the broker's retention, and consider a unique constraint on a business key instead when the event itself is unique, such as an order id.

Upserts as natural idempotence

SQL
insert into order_summary (order_id, status, version)
values (:id, :status, :version)
on conflict (order_id) do update
   set status = excluded.status, version = excluded.version
 where order_summary.version < excluded.version;

Repeating the statement leaves the same row. The where clause adds protection against stale messages, covered next.

Out of order messages

A broker keeps order only within a partition, and only if the producer keeps related events on one partition by using the same key. Even then, retries and reprocessing can reorder.

java
void apply(OrderStatusChanged event) {
    OrderView view = views.find(event.orderId());
    if (view != null && view.version() >= event.version()) {
        return;                                          // stale: a newer state is already applied
    }
    views.save(event.orderId(), event.status(), event.version());
}

Each event carries a version or sequence number from the source of truth. The consumer applies it only if it is newer than what it holds. Timestamps are weaker than versions, because clocks drift and can tie. The key choice and partitioning are in Kafka partitions and keys.

When an event arrives before its prerequisite

A PaymentCaptured for an order that the consumer has not yet seen (because the OrderPlaced is delayed) cannot be applied. Choices: park it in a retry topic and try later, see dead letter and retry topics, or store a placeholder row that the later event completes.

Side effects

For emails and calls to other services, store an "attempted" record before calling, pass an idempotency key to the provider, and reconcile by checking status. The HTTP side is in idempotency keys.

Test it

Write a test that delivers the same message twice, and one that delivers a newer then an older version. Run both through the real handler and a real database, the way the testing strategy in Java testing suggests. These two tests find most bugs.

A short-lived guard in Redis

For work where a database table per message is too heavy, a key with an expiry does the same job. SET with NX succeeds only for the first caller:

java
@Component
class DedupeGuard {

    private final StringRedisTemplate redis;

    DedupeGuard(StringRedisTemplate redis) {
        this.redis = redis;
    }

    boolean firstTime(String messageId) {
        Boolean created = redis.opsForValue()
                .setIfAbsent("seen:" + messageId, "1", Duration.ofDays(7));
        return Boolean.TRUE.equals(created);
    }
}

The key lives for seven days, which should exceed the longest replay window, and then disappears, so the store stays small. The weakness is the gap between the guard and the effect: if the process crashes after the key is set and before the work is done, the retry finds the key and skips the work, and the message is lost. A guard held outside the database that does the work cannot be atomic with it. Use it for effects that are cheap to lose, such as a cache refresh, and use a database table in the same transaction for payments.

Versions solve ordering and duplicates together

When messages carry a version or a sequence number for the entity, one rule handles duplicates, stale arrivals and reordering: apply an update only if its version is higher than the stored one.

SQL
update account
set balance = ?, version = ?
where id = ? and version < ?;

A duplicate has the same version and updates nothing. A late message with an older version updates nothing. The number of affected rows says whether the message did anything, and zero is a normal outcome, not an error. This works for state-carrying events, such as "balance is now 120", and not for deltas, such as "add 20", where each message must be applied exactly once.

Why duplicates occur

CauseWhere it happens
Consumer crashes after processing, before the offset or ackEvery broker
Rebalance moves a partition mid-batchKafka consumer groups
Visibility timeout shorter than processing timeSQS and similar queues
Producer retries after a timeout when the first send succeededAny broker without producer idempotence
An operator replays a dead-letter queueEverywhere
A bug sends the same event twiceYour own code

The sixth is the one that no setting prevents, and it is the reason to design the consumer for duplicates even on a broker with strong guarantees.

Poison messages

A message that always fails is retried forever and blocks the partition behind it. Set a limit on attempts and send the message to a dead-letter destination with the error attached:

java
@Bean
DefaultErrorHandler errorHandler(KafkaTemplate<Object, Object> template) {
    var recoverer = new DeadLetterPublishingRecoverer(template);
    return new DefaultErrorHandler(recoverer, new FixedBackOff(1000L, 3));
}

Three attempts one second apart, then the record goes to orders.DLT with headers that carry the exception. Someone must read that topic: alert on its size, and write the replay path, which is safe because the consumer is idempotent.

Prove the idempotence in a test

The whole design reduces to one test: send the same message twice and check that the effect happened once.

java
@Test
void secondDeliveryHasNoEffect() {
    var message = new PaymentReceived("m-1", "acc-1", new BigDecimal("25.00"));
    handler.handle(message);
    handler.handle(message);
    assertThat(accounts.balanceOf("acc-1")).isEqualByComparingTo("25.00");
}

Add one that sends two different messages in reverse order, and one that crashes between the guard and the effect, with a transaction that rolls both back. These three tests cover the failure modes that duplicates cause in production.

Agree on message identity with the producer

An idempotent consumer needs an identifier that is stable across retries. Agree with the producing team on what it is: a UUID generated once when the business event is created and stored with it, not one generated at send time. A producer that creates a new id on each retry defeats every consumer's deduplication, and the symptom is duplicates that no guard catches.

Put the rule in the event contract, with a test on the producer side that sends the same event twice and checks that the id is the same.

Where this does not apply

If you can use Kafka transactions and every step stays within Kafka, the broker removes duplicates, see exactly once. Anything touching an outside system still needs the techniques here. The reference is the idempotent consumer pattern, and the reliability notes in Kafka describe the delivery guarantees.

Share

Keep reading

Get new posts by email

We send one email when a new post is published. No spam.

Unsubscribe in one click. No spam, ever.