Skip to content
javahacks.dev
Microservices

How to Use RabbitMQ Acks Retries and Dead Letters in Spring

Configure RabbitMQ with Spring AMQP so messages are never lost or stuck: acks, publisher confirms, bounded retries and dead letter queues.

9 min read
Summarize in AI
Flat illustration of services connected around a gateway, with the Spring and Spring Boot logos.

RabbitMQ gives a Java service a dependable work queue, as long as three questions have answers: when does a message count as done, what happens when handling it fails and where does a message go when it can never succeed. The defaults answer none of them well. A consumer with automatic acknowledgement can lose messages when it crashes mid-processing. A handler that throws can put the same message back on the queue forever. A message the broker cannot route disappears. This post configures Spring AMQP so each question has a deliberate answer: acknowledgements, publisher confirms, bounded retries, dead lettering and a path to replay. It uses Spring Boot 4 and RabbitMQ 4. For the equivalent design on Kafka, see dead letter queues with Kafka and Spring, and for choosing between the two, synchronous versus asynchronous communication.

Acknowledgements: when is a message done

A RabbitMQ consumer receives a message and tells the broker when it is finished by acknowledging it. Until then the broker holds the message as unacknowledged and redelivers it if the consumer's channel closes. The mode decides who acknowledges:

ModeWho acksBehaviour
NONENobody; the broker treats delivery as successFastest and loses messages if the consumer dies mid-handling
AUTO (Spring's default)The listener container, after the listener returns normallyAcks on success; on an exception it rejects (and requeues by default)
MANUALYour code, with the ChannelFull control; you must ack or the message stays unacked

AUTO is a good default. The container acknowledges when your method returns and rejects when it throws, and the thrown exception type decides what happens next, which the retry section covers. Set it explicitly:

src/main/resources/application.propertiesproperties
spring.rabbitmq.listener.simple.acknowledge-mode=auto
spring.rabbitmq.listener.simple.prefetch=20
spring.rabbitmq.listener.simple.concurrency=4
spring.rabbitmq.listener.simple.max-concurrency=8

prefetch limits how many unacknowledged messages one consumer may hold. The default of 250 means a slow consumer hoards 250 messages while others sit idle; a lower value, 10 to 50, spreads work fairly and bounds redelivery after a crash. It is RabbitMQ's back pressure control. Use MANUAL mode only when acknowledgement depends on something outside the method return, such as an asynchronous completion.

The topology: queue, exchange and dead letter exchange

Declare the queues and their dead letter routing in code so they are created with the application and reviewed with it:

src/main/java/dev/javahacks/amqp/AmqpTopology.javajava
@Configuration
public class AmqpTopology {

    public static final String EXCHANGE = "orders";
    public static final String QUEUE = "orders.billing";
    public static final String DLX = "orders.dlx";
    public static final String DLQ = "orders.billing.dlq";

    @Bean DirectExchange ordersExchange() { return new DirectExchange(EXCHANGE, true, false); }
    @Bean DirectExchange deadLetterExchange() { return new DirectExchange(DLX, true, false); }

    @Bean
    Queue billingQueue() {
        return QueueBuilder.durable(QUEUE)
                .quorum()                                           // replicated, safe queue type
                .deadLetterExchange(DLX)
                .deadLetterRoutingKey(QUEUE)
                .build();
    }

    @Bean Queue billingDeadLetterQueue() { return QueueBuilder.durable(DLQ).quorum().build(); }

    @Bean Binding billingBinding() { return BindingBuilder.bind(billingQueue()).to(ordersExchange()).with("order.paid"); }
    @Bean Binding dlqBinding() { return BindingBuilder.bind(billingDeadLetterQueue()).to(deadLetterExchange()).with(QUEUE); }
}

The main queue carries x-dead-letter-exchange and x-dead-letter-routing-key arguments. When a message on it is rejected without requeue, expires or exceeds a length limit, the broker republishes it to the dead letter exchange, which routes it to the dead letter queue. Quorum queues are replicated with the Raft protocol and are the recommended type for data you cannot lose; classic mirrored queues are deprecated and removed in RabbitMQ 4. Note that the arguments of an existing queue cannot be changed: if the queue already exists without the dead letter arguments, declaring it again with different ones fails with PRECONDITION_FAILED - inequivalent arg 'x-dead-letter-exchange', and the fix is to delete and recreate it or to use a policy, which applies settings to queues by name pattern without redeclaring them.

Publisher confirms: did the broker take it

The send side has its own failure mode. rabbitTemplate.convertAndSend returns as soon as the message is handed to the client library. If the broker never received it, or received it and could not route or persist it, nothing tells you. Two features close the gap:

properties
spring.rabbitmq.publisher-confirm-type=correlated
spring.rabbitmq.publisher-returns=true
spring.rabbitmq.template.mandatory=true

Confirms make the broker acknowledge each publish after it has taken responsibility (for a quorum queue, after replicating it to a majority). Returns with mandatory=true hand back a message that matched no queue, which would otherwise be dropped silently. Wire the callbacks:

src/main/java/dev/javahacks/amqp/OrderPublisher.javajava
@Component
public class OrderPublisher {
    private static final Logger log = LoggerFactory.getLogger(OrderPublisher.class);
    private final RabbitTemplate rabbit;

    public OrderPublisher(RabbitTemplate rabbit) {
        this.rabbit = rabbit;
        rabbit.setConfirmCallback((correlation, ack, cause) -> {
            if (!ack) {
                log.error("broker rejected message {}: {}", correlation != null ? correlation.getId() : "?", cause);
            }
        });
        rabbit.setReturnsCallback(returned ->
                log.error("unroutable message {} to {} with key {}: {}",
                        returned.getMessage().getMessageProperties().getMessageId(),
                        returned.getExchange(), returned.getRoutingKey(), returned.getReplyText()));
    }

    public void paid(OrderPaid event) {
        var correlation = new CorrelationData(event.orderId());
        rabbit.convertAndSend(AmqpTopology.EXCHANGE, "order.paid", event, message -> {
            message.getMessageProperties().setMessageId(event.orderId());
            message.getMessageProperties().setDeliveryMode(MessageDeliveryMode.PERSISTENT);
            return message;
        }, correlation);
    }
}

A negative confirm or a return is the cue to store the event for resending, as in the failure handler pattern used with Kafka. The CorrelationData id links a confirm to the message it concerns. As with Kafka, a process crash between the business transaction and the publish still loses the event, so for events that must follow a database change use the outbox pattern in the Kafka producer guide, which applies equally here.

Retries: bounded, with backoff

When a handler throws, AUTO mode rejects the message. With the default default-requeue-rejected=true, the broker puts it straight back at the front of the queue, and the consumer receives it again immediately: an infinite loop that burns CPU and blocks everything behind it. Two controls end the loop.

First, retry inside the container with a limit and backoff, and then give up cleanly. Spring Boot configures a RetryInterceptor:

properties
spring.rabbitmq.listener.simple.retry.enabled=true
spring.rabbitmq.listener.simple.retry.max-attempts=4
spring.rabbitmq.listener.simple.retry.initial-interval=1s
spring.rabbitmq.listener.simple.retry.multiplier=2
spring.rabbitmq.listener.simple.retry.max-interval=10s
spring.rabbitmq.listener.simple.default-requeue-rejected=false

After four attempts (1, 2, 4 seconds apart) the message is rejected without requeue, so the broker dead letters it to the DLQ. Both properties together are the essential combination: retry for transient failures, reject without requeue for the rest. Some exceptions are not worth retrying at all, such as a malformed payload that will fail the same way each time. Throwing AmqpRejectAndDontRequeueException skips the retries and sends the message to the dead letter exchange at once:

java
@RabbitListener(queues = AmqpTopology.QUEUE)
public void on(OrderPaid event) {
    if (event.orderId() == null) {
        throw new AmqpRejectAndDontRequeueException("order id missing");      // permanent: no retry
    }
    billing.charge(event);                                                  // transient failures retry
}

Spring's ConditionalRejectingErrorHandler already does this for message conversion errors (MessageConversionException), since an unparseable body can never succeed. Define your own set of exceptions to be treated as fatal where it makes sense.

Second, understand what the in-container retry costs. It happens in memory on the consumer thread, so the thread is occupied during the backoff sleeps, and if the application restarts mid-retry the attempt count resets (the broker redelivers the message from the start). For short backoffs of seconds that is acceptable. For retries that must wait minutes, use a delayed retry queue.

Delayed retry with a wait queue

A wait queue holds a failed message for a fixed time and then dead letters it back to the main queue, so the delay is enforced by the broker and survives restarts:

java
@Bean
Queue billingRetryQueue() {
    return QueueBuilder.durable("orders.billing.retry")
            .quorum()
            .ttl(60_000)                                     // wait one minute
            .deadLetterExchange(AmqpTopology.EXCHANGE)       // then route back to the original exchange
            .deadLetterRoutingKey("order.paid")
            .build();
}

The failure handler publishes the failed message to the retry queue, with a retry counter in a header, instead of rejecting it:

java
@RabbitListener(queues = AmqpTopology.QUEUE)
public void on(Message message, OrderPaid event) {
    try {
        billing.charge(event);
    } catch (TransientBillingException e) {
        int attempts = Optional.ofNullable(message.getMessageProperties().getHeader("x-attempts"))
                .map(h -> (Integer) h).orElse(0);
        if (attempts >= 5) {
            throw new AmqpRejectAndDontRequeueException("gave up after " + attempts, e);     // to the DLQ
        }
        rabbit.convertAndSend("", "orders.billing.retry", event, m -> {
            m.getMessageProperties().setHeader("x-attempts", attempts + 1);
            return m;
        });
    }
}

A message that fails goes to the wait queue, sits for a minute, returns to the main queue and is tried again, up to five times, then falls into the DLQ. Per message TTL is possible but suffers from head-of-line blocking (a message with a long TTL at the front blocks shorter ones behind it), so use one retry queue per delay tier, such as 10 seconds, 1 minute and 10 minutes, if you need escalating delays. The delayed message exchange plugin is an alternative where available.

The dead letter queue is a to do list

A message in the DLQ is one that failed beyond the automated recovery. It is not storage; it is work for a person. Make it operable:

Keep the reason. RabbitMQ adds an x-death header recording the queue, reason (rejected, expired, maxlen) and count. Spring's RepublishMessageRecoverer can add the exception message and stack trace as headers when it republishes after retries are exhausted, which is far more helpful than the bare header:

java
@Bean
MessageRecoverer republishRecoverer(RabbitTemplate rabbit) {
    return new RepublishMessageRecoverer(rabbit, AmqpTopology.DLX, AmqpTopology.QUEUE);
}

With this bean, a message that exhausts its retries is republished to the DLX with x-exception-message and x-exception-stacktrace headers, readable in the management UI.

Alert on depth. The metric rabbitmq_queue_messages{queue="orders.billing.dlq"} above zero for more than a few minutes should page or at least ticket. A DLQ nobody watches is where data goes to be forgotten.

Provide a replay path. After the cause is fixed, move messages back. The shovel plugin or a small admin endpoint does it:

java
@PostMapping("/admin/dlq/replay")
public int replay(@RequestParam(defaultValue = "100") int max) {
    int moved = 0;
    Message m;
    while (moved < max && (m = rabbit.receive(AmqpTopology.DLQ, 1000)) != null) {
        rabbit.send(AmqpTopology.EXCHANGE, "order.paid", m);
        moved++;
    }
    return moved;
}

Protect that endpoint, and make the consumer idempotent, because a replayed message may already have had side effects; see handling duplicate messages with an idempotent consumer.

Test the failure paths

The happy path is the least interesting test. With Testcontainers' RabbitMQContainer, assert what happens on failure:

java
@SpringBootTest
@Testcontainers
class BillingListenerFailureIT {

    @Container @ServiceConnection
    static RabbitMQContainer rabbit = new RabbitMQContainer("rabbitmq:4.0-management");

    @Autowired RabbitTemplate template;
    @MockitoBean BillingService billing;

    @Test
    void aPermanentlyFailingMessageEndsUpInTheDeadLetterQueue() {
        doThrow(new IllegalStateException("card service down")).when(billing).charge(any());

        template.convertAndSend(AmqpTopology.EXCHANGE, "order.paid", new OrderPaid("o-1", 1000));

        Message dead = template.receive(AmqpTopology.DLQ, 20_000);
        assertThat(dead).isNotNull();
        verify(billing, times(4)).charge(any());                    // four attempts, then dead lettered
    }

    @Test
    void aMalformedMessageSkipsRetriesAndGoesStraightToTheDeadLetterQueue() {
        template.convertAndSend(AmqpTopology.EXCHANGE, "order.paid", "not json at all");

        assertThat(template.receive(AmqpTopology.DLQ, 10_000)).isNotNull();
        verifyNoInteractions(billing);
    }
}

The attempts assertion pins the retry configuration, so a later edit that changes it breaks a test and not production.

Common mistakes

**Auto acknowledgement with NONE.** Convenient and lossy.

Default requeue. A poison message that loops. Set default-requeue-rejected=false.

Classic non-durable queues. A broker restart loses them. Use durable, quorum queues and persistent messages for data that matters.

No prefetch limit. One consumer hoards the backlog, and a crash redelivers hundreds of messages at once.

Ignoring returns. A typo in a routing key makes every publish vanish, and nothing logs.

Long synchronous work in the listener. It ties up the consumer thread and the unacknowledged window. Hand off carefully, or lower the work per message.

When this does not apply

If you need to replay history, many independent consumers reading the same stream or very high throughput, a log such as Kafka is the better fit. RabbitMQ shines for task queues, routing and request reply with modest volume. If the broker is managed by a cloud provider, its offering (Amazon MQ, CloudAMQP) may restrict plugins and queue types. The official references are the RabbitMQ documentation on reliability, dead letter exchanges and the Spring AMQP reference.

Taggedhow to
Share

Keep reading

Get new posts by email

We send one email when a new post is published. No spam.

Unsubscribe in one click. No spam, ever.