Skip to content
javahacks.dev
Microservices

Dead Letter Queue Kafka Spring With Retry Topics Setup

Add retries and a dead letter topic to a Spring Kafka consumer: DefaultErrorHandler, backoff, DeadLetterPublishingRecoverer and RetryableTopic.

6 min read
Summarize in AI
Flat illustration of settings switches, a slider and a gear, with the Apache Kafka, Spring, Spring Boot and Apache Maven logos.

A consumer that cannot process a message has three bad options: crash, skip the message and lose it, or retry it forever and block everything behind it. A dead letter queue offers a fourth. After a bounded number of attempts, the failing record is moved to a separate topic with the reason attached, and the consumer carries on. The message is kept for inspection and replay, and the partition keeps moving. Spring Kafka supports this with two mechanisms, a blocking retry with a dead letter topic and a non-blocking retry topic chain. This post sets up both, covers the poison pill that breaks deserialisation, shows how to choose retries by failure type and how to operate the topic afterwards. It uses Spring Kafka 4.

Blocking retries and a DLT

java
@Bean
DefaultErrorHandler errorHandler(KafkaTemplate<Object, Object> template) {
    var recoverer = new DeadLetterPublishingRecoverer(template);
    var handler = new DefaultErrorHandler(recoverer, new FixedBackOff(1000L, 3));
    handler.addNotRetryableExceptions(IllegalArgumentException.class);
    return handler;
}

When the listener throws, the error handler retries the same record in place, up to three more times with one second between, and then hands it to the recoverer. The recoverer publishes it to a topic named after the original with a .DLT suffix, so orders becomes orders.DLT, on the same partition number by default. The exception class, message and stack trace travel as record headers, for example kafka_dlt-exception-message and kafka_dlt-original-topic, so a person reading the dead letter sees what failed and where it came from. Spring Boot picks up a DefaultErrorHandler bean and applies it to the listener containers it creates.

The word blocking is the catch. While the handler retries a record, the partition is stuck: no later record from it is processed. With a one-second delay and three attempts that is four seconds per bad record. For a handful of failures it is fine, and for a burst of them the lag grows.

Poison pills

A poison pill is a record that fails before your code sees it, usually because the bytes cannot be deserialised. The consumer throws on every poll, cannot advance past the record, and loops forever. The fix is to let a wrapper catch the deserialisation failure and hand it to the error handler as a normal failure:

properties
spring.kafka.consumer.value-deserializer=org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
spring.kafka.consumer.properties.spring.deserializer.value.delegate.class=org.springframework.kafka.support.serializer.JsonDeserializer

ErrorHandlingDeserializer wraps the real deserialiser. When it fails, it records the error in a header and passes a null value, and the container routes the record to the dead letter topic without ever invoking the listener. Without this wrapper, one malformed message stops the consumer group until someone intervenes. Make it a default in every consumer you write. The related behaviour when partitions move between consumers is covered in rebalances and offsets.

Non-blocking retries

For failures that may heal in a while, such as a downstream service that is briefly down, a long blocking backoff would stall the partition. Retry topics move the wait out of the main flow:

java
@RetryableTopic(
        attempts = "4",
        backoff = @Backoff(delay = 1000, multiplier = 2.0),
        dltStrategy = DltStrategy.FAIL_ON_ERROR)
@KafkaListener(topics = "orders")
void handle(OrderPlaced event) {
    processor.process(event);
}

@DltHandler
void dead(OrderPlaced event, @Header(KafkaHeaders.EXCEPTION_MESSAGE) String reason) {
    log.error("gave up on order {}: {}", event.id(), reason);
}

Spring creates retry topics, such as orders-retry-1000 and orders-retry-2000, and a dead letter topic. A failing record is published to the first retry topic and the main partition moves on immediately. A consumer of the retry topic delays the record until its time and processes it again. After the last attempt it goes to the dead letter topic, where the @DltHandler receives it. The cost is more topics, a changed order of processing, since retried records arrive later than their successors, and extra configuration. Choose it when ordering within a key is not critical and failures are frequent. The retry topic mechanism is described in the retry topic documentation.

Choose retries by failure type

Retrying a failure that will never succeed wastes time and fills the dead letter topic late. Classify exceptions:

FailureRetry?Example
Transient infrastructureYes, with backoffTimeout calling a service, a lock timeout
Permanent data errorNo, straight to the DLTValidation failure, unknown enum value
Deserialisation failureNoMalformed payload
Bug in the consumerNo; fix the code, then replayNullPointerException

Register permanent exceptions with addNotRetryableExceptions, and the handler skips retries for them. A consumer that is not idempotent makes retries dangerous, because a record that half-succeeded runs again; see handling duplicate messages and the design advice in reliable Kafka consumers.

Operate the dead letter topic

A dead letter topic that nobody reads is a silent data loss with extra steps. Alert on its message count: any message arriving is worth a notification, and a growing count is an incident. Give it a retention long enough to investigate, days to weeks. Provide a way to look at the records, with their headers, in a tool the team uses. Then decide what happens to each: fix the data and republish it to the original topic, fix the consumer bug and replay the lot, or discard it with a note. Build the replay as a small, tested tool, not a one-off command typed under pressure, and keep a record of what you replayed. Make sure the consumer is idempotent before you replay anything.

Testing the failure path

A dead letter setup that has never been exercised will fail the first time it is needed. Test it with an embedded or containerised broker. Publish a message that makes the listener throw, wait for the dead letter topic to receive it, and assert on the headers:

java
@Test
void poisonRecordEndsUpInTheDlt() {
    kafkaTemplate.send("orders", "k1", "not valid json");

    ConsumerRecord<String, String> dead = KafkaTestUtils.getSingleRecord(dltConsumer, "orders.DLT", Duration.ofSeconds(15));

    assertThat(dead.headers().lastHeader("kafka_dlt-original-topic")).isNotNull();
}

Add a test that a permanent error is not retried: count listener invocations and assert it was called once. And one that a transient error is retried the configured number of times before it lands in the topic. These three tests cover the behaviours that decide whether a bad record is handled or silently lost.

Sizing and keys

The dead letter topic should have at least as many partitions as the source topic when you keep the default behaviour of publishing to the same partition number, otherwise the publish fails. Keep the original record key, which the recoverer does by default, so a replay lands in the same partition order. Decide on retention and compaction up front: compacting a dead letter topic keeps only the latest record per key and can discard evidence of earlier failures, so use time-based retention.

Where this does not apply

Messages whose loss is acceptable, such as metrics samples, do not need the machinery. A consumer that processes records in a database transaction with its offset needs different handling. The configuration options are in the error handling chapter of the Spring Kafka reference.

Share

Keep reading

Get new posts by email

We send one email when a new post is published. No spam.

Unsubscribe in one click. No spam, ever.