Skip to content
javahacks.dev
Micronaut

How to Use Micronaut With Kafka Producers and Consumers

Send and receive Kafka messages in Micronaut with declarative producers and listeners, set serializers and offsets and test with a real broker.

6 min read
Summarize in AI
Flat illustration of startup time of a JVM compared with a native image, with the Apache Kafka, Spring Boot, Docker and Kubernetes logos.

Micronaut Kafka wraps the Kafka client in declarative interfaces: you describe a producer as an interface with annotations, you write a consumer as a class with annotated methods, and the framework generates the implementation at compile time. There is no reflection and no runtime proxy, which keeps startup fast and works in a native image. What the declarative style hides are the delivery guarantees, which are the same as Kafka's and which you must still choose. This post sets up a producer and a consumer on Micronaut 4 and JDK 25, then covers offsets and acknowledgements, failure handling and testing. For the same job in Quarkus, see Kafka in Quarkus.

Configuration

pom.xmlXML
<dependency>
  <groupId>io.micronaut.kafka</groupId>
  <artifactId>micronaut-kafka</artifactId>
</dependency>
src/main/resources/application.ymlYAML
kafka:
  bootstrap:
    servers: localhost:9092

Everything under kafka.* is passed to the Kafka client, so settings such as kafka.producers.default.acks and kafka.consumers.default.max-poll-records work as in the Kafka documentation. Named configurations let you tune individual clients: kafka.consumers.shipping.auto-offset-reset: earliest applies to a consumer whose clientId is shipping. In development, Micronaut Test Resources can start a Kafka container for you when no server is configured, as it does for databases.

A producer

src/main/java/com/example/orders/OrderProducer.javajava
@KafkaClient
public interface OrderProducer {

    @Topic("orders")
    CompletableFuture<RecordMetadata> send(@KafkaKey String orderId, OrderPlaced event);

    @Topic("orders")
    void sendAndForget(@KafkaKey String orderId, OrderPlaced event);
}

@KafkaClient marks an interface for which Micronaut generates an implementation. @Topic names the destination. The parameter annotated @KafkaKey becomes the record key, which decides the partition; records with the same key keep their order, as explained in partitions and keys. Any other parameter is the value, serialised by the configured serialiser, with Jackson JSON as the default for objects. The return type controls the behaviour: a CompletableFuture<RecordMetadata> lets the caller wait for the broker's acknowledgement and see the partition and offset, while void is fire-and-forget. Inject the interface like any bean: OrderProducer producer.

Fire-and-forget is unsafe for anything that matters. A failed send is logged and the caller never learns of it. For important events, wait on the future, handle the failure, and consider the outbox pattern when the event must be published if and only if a database change committed.

A consumer

src/main/java/com/example/shipping/ShippingListener.javajava
@KafkaListener(groupId = "shipping", offsetReset = OffsetReset.EARLIEST)
public class ShippingListener {

    private final ShipmentService shipments;

    ShippingListener(ShipmentService shipments) { this.shipments = shipments; }

    @Topic("orders")
    void onOrder(@KafkaKey String orderId, OrderPlaced event) {
        shipments.create(orderId, event);
    }
}

@KafkaListener marks a class whose @Topic methods consume records. The groupId is the Kafka consumer group: instances with the same group share the topic's partitions, and different groups each see every record. offsetReset = EARLIEST decides where a group with no stored offset starts; the default LATEST skips everything sent before the group first connected, which looks like a consumer that "misses messages" in a new environment. Arguments are bound by type: the key, the value, Consumer, ConsumerRecord, and metadata like @Header or Acknowledgement. By default the listener processes records on a single thread per consumer, in order; threads creates several consumers in the same process, each with its own partitions.

Delivery guarantees

Out of the box, the offset is committed automatically on a timer after records are polled (enable.auto.commit behaviour with OffsetStrategy.AUTO). If the process dies after processing a record and before the commit, the record is delivered again, and if the commit happens before processing finishes and then the process dies, the record is lost. Neither is acceptable by accident. Choose a strategy:

java
@KafkaListener(groupId = "shipping", offsetStrategy = OffsetStrategy.SYNC_PER_RECORD)

SYNC_PER_RECORD commits after each record is handled successfully, giving at-least-once delivery with the smallest window of duplicates, at the cost of a commit per record. SYNC and ASYNC commit after each batch, DISABLED leaves it to you, and with an Acknowledgement parameter in the method signature you call ack.ack() when you are done. At-least-once is the practical target: make the handler idempotent, for instance by recording the processed record's key and offset in the same database transaction as its effect, so a replay does nothing. The patterns are in idempotent consumers. Exactly-once across Kafka and a database needs transactions on both sides or the outbox pattern, and is rarely worth the complexity inside a single service.

Failure handling

When a listener method throws, the default is to log the error and continue with the next record, which means the failed record is skipped. That is the Micronaut default and it is a quiet way to lose data. Implement an exception handler to decide:

java
@KafkaListener(groupId = "shipping", errorStrategy = @ErrorStrategy(value = ErrorStrategyValue.RETRY_ON_ERROR, retryCount = 3, retryDelay = "1s"))
public class ShippingListener { /* ... */ }

RETRY_ON_ERROR re-polls the failing record after a delay up to a retry count; RESUME_AT_NEXT_RECORD skips it; RETRY_EXPONENTIALLY_ON_ERROR backs off. For records that can never succeed, such as malformed payloads, retrying is a waste and blocks the partition behind the poison record. Publish them to a dead letter topic and move on, and alert on that topic's depth. Deserialisation errors happen before your method runs, so configure the deserialiser to catch them; Micronaut passes them to the KafkaListenerExceptionHandler, where you can send the raw bytes to the dead letter topic.

Test it

A fast test replaces Kafka with Micronaut Test Resources' container, or with an in-memory stand-in for the service under test.

java
@MicronautTest
class ShippingListenerTest {

    @Inject OrderProducer producer;
    @Inject ShipmentRepository shipments;

    @Test
    void creates_a_shipment_for_a_placed_order() {
        producer.sendAndForget("o-1", new OrderPlaced("o-1", "12 High St"));

        await().atMost(Duration.ofSeconds(10)).untilAsserted(() ->
            assertTrue(shipments.existsByOrderId("o-1")));
    }
}

The consumer is asynchronous, so poll for the effect with Awaitility instead of sleeping, as described in testing async code. With a real broker in a container, this is a full round trip and takes seconds; keep a handful of such tests and unit-test the handler logic directly.

Serialisation

The default serialiser for objects is Jackson JSON, and for keys it is a string. Declare them explicitly for anything shared with other teams, because the producer and the consumer must agree on the format and on the class names if type information is included in the payload. Avoid putting Java class names in messages: a consumer that must deserialise com.example.orders.OrderPlaced couples every reader to your package layout, and renaming the class breaks old records still on the topic. Define the payload as a plain record with stable field names, add new fields as optional, and never reuse a field name with a different meaning. For stronger guarantees, use Avro or Protobuf with a schema registry, which enforces compatibility rules when producers register a new schema version, as covered in schema registries.

Batch consumption

A listener can receive records in a batch by declaring a List parameter and setting batch = true on @KafkaListener. Processing a batch in one database transaction is much faster than one transaction per record for write-heavy consumers, and the offset is committed after the batch. The cost is coarser failure handling: one bad record fails the whole batch, so catch exceptions per record inside the method, send the bad ones to a dead letter topic, and let the rest proceed.

Where this does not apply

If you need Kafka Streams or complex stream processing, use the Kafka Streams module or Flink; the declarative listener is for message-at-a-time handling. For a first Micronaut project, begin with the Micronaut tutorial. The reference is the Micronaut Kafka guide, and the Kafka concepts are in the Apache Kafka documentation.

Share

Keep reading

Get new posts by email

We send one email when a new post is published. No spam.

Unsubscribe in one click. No spam, ever.