Skip to content
javahacks.dev
Testing

How to Test Kafka Consumers With Testcontainers in Spring

Test a Spring Kafka consumer against a real broker in Testcontainers: send a record, wait for the effect, and cover retries and the dead letter topic.

6 min read
Summarize in AI
Flat illustration of a test report with one failing test, with the Apache Kafka, Spring and Selenium logos.

A Kafka consumer is easy to test badly. A test that mocks the listener's collaborators proves the method body works and says nothing about deserialisation, the topic configuration, the consumer group, offset handling, retries or the dead letter topic, which are exactly where consumers fail in production. A real broker in a container removes that gap. With Testcontainers and Spring Boot's service connection support, the broker starts for the test, the application connects to it with no properties, and the test sends a real record and waits for the effect. This post builds that test for a Spring Kafka consumer on Spring Boot 4, Spring Kafka 4 and JDK 25, then covers the failure path, duplicates and keeping the suite fast.

The consumer under test

src/main/java/com/example/shipping/ShipmentListener.javajava
@Component
class ShipmentListener {

    private final ShipmentService shipments;

    ShipmentListener(ShipmentService shipments) { this.shipments = shipments; }

    @KafkaListener(topics = "orders", groupId = "shipping")
    void on(OrderPlaced event) {
        shipments.createFor(event.orderId());
    }
}

The listener reads OrderPlaced records from orders as part of the group shipping. ShipmentService.createFor writes a row. The properties for deserialisation (a JSON deserialiser with trusted packages) are in application.yml. What we want to prove is that a record on the topic becomes a row.

The test

src/test/java/com/example/shipping/ShipmentListenerIT.javajava
@SpringBootTest
@Testcontainers
class ShipmentListenerIT {

    @Container
    @ServiceConnection
    static KafkaContainer kafka = new KafkaContainer(DockerImageName.parse("apache/kafka:3.9.0"));

    @Autowired KafkaTemplate<String, OrderPlaced> template;
    @Autowired ShipmentRepository shipments;

    @Test
    void creates_a_shipment_when_an_order_is_placed() {
        template.send("orders", "o-1", new OrderPlaced("o-1", "12 High St"));

        await().atMost(Duration.ofSeconds(15)).untilAsserted(() ->
            assertTrue(shipments.existsByOrderId("o-1")));
    }
}

@Container on a static field starts one broker for the class, and @ServiceConnection tells Spring Boot to read its bootstrap address and configure spring.kafka.bootstrap-servers from it: no @DynamicPropertySource is needed, as explained in ServiceConnection. The application context starts with the listener connected to the container. The test sends a record through the application's own KafkaTemplate, so serialisation is exercised as in production, and then waits with Awaitility for the visible effect, a database row. Poll the effect, and do not sleep: the consumer joins its group, receives a partition assignment and polls, which takes a variable time on a fresh broker, so a fixed sleep is either too long or flaky. See Awaitility for asynchronous tests and the wider treatment in testing async code.

The 15-second limit is an upper bound, and the test returns as soon as the row exists. The first test in a build pays the cost of the container pull and start, usually a few seconds, which later classes avoid by sharing the container.

Topic and offset setup

Kafka creates a topic when the first record arrives if auto-creation is on, which Testcontainers' broker enables by default, with default partitions. Be explicit when the test depends on partitioning:

java
@Bean
NewTopic orders() {
    return TopicBuilder.name("orders").partitions(3).replicas(1).build();
}

Spring Kafka's KafkaAdmin creates NewTopic beans at startup. For consumers that start with an offset reset of latest, a record sent before the consumer was assigned partitions is missed, and the test fails intermittently. Either set spring.kafka.consumer.auto-offset-reset=earliest for tests, or wait for assignment before sending, with ContainerTestUtils.waitForAssignment(container, partitions) using the KafkaListenerEndpointRegistry. The first is simpler and mirrors a new consumer group in production. The meaning of keys and partitions, and why records for one order must share a key, is in partitions and keys.

Use a unique group id or topic name per test class if tests share a broker and must not see each other's records, or clean up by recreating topics. Records left from an earlier test appear to a later consumer with earliest and can produce confusing failures.

Failure paths

The happy path is half of what a consumer does. Test what happens when processing throws. With a retry and dead letter configuration:

java
@Bean
DefaultErrorHandler errorHandler(KafkaTemplate<Object, Object> template) {
    var recoverer = new DeadLetterPublishingRecoverer(template);
    return new DefaultErrorHandler(recoverer, new FixedBackOff(200L, 2));
}
java
@Test
void sends_a_poison_record_to_the_dead_letter_topic() {
    doThrow(new IllegalStateException("boom")).when(shipmentService).createFor("bad");

    template.send("orders", "bad", new OrderPlaced("bad", "?"));

    ConsumerRecord<String, String> dead = KafkaTestUtils.getSingleRecord(deadLetterConsumer, "orders.DLT", Duration.ofSeconds(15));
    assertEquals("bad", dead.key());
}

With a FixedBackOff of two retries, the listener is called three times, then the record is published to orders.DLT and the consumer moves on. Assert that the failing record reaches the dead letter topic and that a record sent after it is still processed, which proves the partition is not blocked by the poison one. The pattern and its pitfalls are in dead letter queues with Spring Kafka. Also test deserialisation errors: send a record with invalid JSON, using a producer with a StringSerializer, and check what happens. An ErrorHandlingDeserializer wraps the real deserialiser so that a bad payload becomes a handled failure; without it the consumer can loop on the record forever.

Duplicates

Kafka delivers at least once, so the consumer will see the same record twice after a rebalance or a crash, and the effect must be the same. Test it directly: send the same record twice, wait, and assert that exactly one shipment exists.

java
template.send("orders", "o-2", event);
template.send("orders", "o-2", event);

await().untilAsserted(() -> assertEquals(1, shipments.countByOrderId("o-2")));

This test fails for a naive consumer and passes for one that deduplicates, for example with a unique constraint on the order id and an idempotent insert. Because the check is cheap, it belongs in every consumer's tests. The techniques are in idempotent consumers and writing a reliable Kafka consumer. Awaitility's untilAsserted retries until the count is 1, so a short extra wait with during can confirm it stays at 1.

Speed

Starting a broker per test class dominates the time. Share one container across the suite: declare it in an abstract base class or in a static initialiser so that all classes use it, or enable container reuse so that it survives between runs on a developer machine, covered in reusing containers. Share the Spring context too: classes with identical configuration reuse the cached context. Keep the consumer tests few, and test the handler logic in plain unit tests, so the integration tests only cover the wiring and the failure behaviour. A reasonable ratio is a handful of container tests per consumer and many unit tests.

Alternatives

@EmbeddedKafka from spring-kafka-test starts an in-JVM broker with no Docker. It is faster to start and simple to configure, and it differs from a real broker in version and some behaviour, so it can pass where production fails. Use it when containers are not available. A mock consumer or the TopologyTestDriver suits tests of Kafka Streams topologies, not listener wiring. The Spring side is documented in Spring Kafka testing.

Where this does not apply

For logic inside the listener, test the service, not the broker. For the compatibility of message formats between teams, add contract tests, as in Pact, instead of end-to-end tests. The container module is described at java.testcontainers.org/modules/kafka.

Taggedhow to
Share

Keep reading

Get new posts by email

We send one email when a new post is published. No spam.

Unsubscribe in one click. No spam, ever.