Skip to content
javahacks.dev
Quarkus

How to Use Quarkus With Kafka and Reactive Messaging

Produce and consume Kafka messages in Quarkus with SmallRye Reactive Messaging: channels, @Incoming and @Outgoing, acknowledgement and Dev Services.

6 min read
Summarize in AI
Flat illustration of startup time of a JVM compared with a native image, with the Quarkus, Apache Kafka, Spring Boot, Docker and Kubernetes logos.

Quarkus does not expose the Kafka client directly in application code. It exposes channels, and a small piece of configuration binds a channel to a topic. The code consumes from orders-in and produces to shipments-out, and which topic that means, with which serialiser, is decided in application.properties. The abstraction comes from SmallRye Reactive Messaging, and it works well once you know the three behaviours that decide correctness: how messages are acknowledged, which thread runs your method, and what ordering Kafka guarantees. This post builds a consumer and a producer on Quarkus 3 and JDK 25, then covers failure handling and scaling.

Dependency and Dev Services

pom.xmlXML
<dependency>
  <groupId>io.quarkus</groupId>
  <artifactId>quarkus-messaging-kafka</artifactId>
</dependency>

In dev mode and in tests, Quarkus starts a Kafka broker in a container when no broker address is configured. This is Dev Services, and it needs a container runtime. The application finds the broker automatically, so mvn quarkus:dev gives you a working Kafka with no setup, and tests get a fresh one. The mechanism applies to databases as well, and is described in Dev Services. For production, set kafka.bootstrap.servers to the real brokers.

Producing

src/main/java/com/example/orders/OrderPublisher.javajava
@ApplicationScoped
public class OrderPublisher {

    @Inject
    @Channel("orders-out")
    Emitter<OrderPlaced> emitter;

    public CompletionStage<Void> publish(OrderPlaced event) {
        return emitter.send(event);
    }
}
src/main/resources/application.propertiesproperties
mp.messaging.outgoing.orders-out.connector=smallrye-kafka
mp.messaging.outgoing.orders-out.topic=orders
mp.messaging.outgoing.orders-out.value.serializer=io.quarkus.kafka.client.serialization.ObjectMapperSerializer

@Channel("orders-out") names the channel and Emitter<T> is the handle for sending from imperative code. emitter.send(payload) returns a CompletionStage that completes when the broker acknowledges the record, so a caller can wait for it, or ignore it and accept loss on failure. Configuration binds the outgoing channel orders-out to the topic orders with a serialiser; the Jackson-based one handles records and POJOs. A record key controls the partition: wrap the payload in a Message with OutgoingKafkaRecordMetadata to set a key, or use a keyed serialiser. Messages with the same key go to the same partition, which gives ordering for that key, as explained in partitions and keys.

Do not send from a request thread and ignore the result if the message matters. A message published after a database commit can be lost if the process crashes between the commit and the send; the outbox pattern in idempotent consumers and outboxes is the remedy.

Consuming

src/main/java/com/example/shipping/ShipmentService.javajava
@ApplicationScoped
public class ShipmentService {

    @Incoming("orders-in")
    @Blocking
    @Transactional
    public void on(OrderPlaced event) {
        shipments.create(event.orderId());
    }
}
properties
mp.messaging.incoming.orders-in.connector=smallrye-kafka
mp.messaging.incoming.orders-in.topic=orders
mp.messaging.incoming.orders-in.group.id=shipping
mp.messaging.incoming.orders-in.auto.offset.reset=earliest
mp.messaging.incoming.orders-in.value.deserializer=com.example.orders.OrderPlacedDeserializer

@Incoming("orders-in") makes the method a consumer of that channel. The group.id is the Kafka consumer group: instances with the same id share the topic's partitions, and different ids each get every message. auto.offset.reset=earliest decides where a new group starts; without it, a new group starts at the end and misses messages sent before it first connected. A deserialiser for each payload type is needed, and a Jackson ObjectMapperDeserializer<OrderPlaced> subclass is a three-line class.

Failure handling

The behaviour of a failing message is decided by acknowledgement. Reactive Messaging acknowledges a message when your method returns normally (with the default POST_PROCESSING strategy for payload-based methods) and negatively acknowledges it when your method throws. What happens on a nack is configured with failure-strategy:

properties
mp.messaging.incoming.orders-in.failure-strategy=dead-letter-queue
mp.messaging.incoming.orders-in.dead-letter-queue.topic=orders-dlq

fail is the default: the application marks itself unhealthy and stops consuming, which is safe and halts all processing for the partition until someone acts. ignore logs the failure and moves on, which loses the message. dead-letter-queue publishes the failed record to another topic with the error in headers and continues, which keeps the stream moving and leaves the bad messages for inspection; the concept is described in dead letter queues. Choose deliberately. A poison message, one that fails every time, blocks a partition forever under fail.

Kafka delivery is at-least-once with these settings: a crash after processing and before the offset is committed re-delivers the message. Make handlers idempotent, for example by recording processed event ids in the same transaction as the effect.

Blocking work

The Kafka consumer's events arrive on an I/O thread. Blocking that thread, with JDBC or an HTTP call, stalls the consumer and eventually triggers a rebalance when it misses its poll deadline. That is what @Blocking is for: it moves the method to a worker thread. It is required here because shipments.create writes to a database. Without it, Quarkus logs a warning about blocking the event loop in dev mode. With JDK 21 or later, @RunOnVirtualThread runs the method on a virtual thread instead, suitable for blocking I/O. Methods that are purely non-blocking, returning Uni, stay on the I/O thread. The default for @Blocking handlers is to process messages one at a time per channel to preserve order; @Blocking(ordered = false) allows concurrent processing and drops per-key ordering, so use it only when order does not matter.

Transforming

A method with both annotations consumes, transforms and produces, which suits a processing step:

java
@Incoming("orders-in")
@Outgoing("shipments-out")
public ShipmentRequested toShipment(OrderPlaced event) {
    return new ShipmentRequested(event.orderId(), event.address());
}

The returned object becomes the payload of the next message on shipments-out, which you configure as an outgoing channel. Return Message<T> instead to carry metadata or control acknowledgement yourself. A processor in the middle of a chain acknowledges the incoming message after the outgoing one is acknowledged, which propagates backpressure and failure upstream.

Ordering, partitions and scaling

Kafka guarantees order within a partition, and a consumer group has at most one active consumer per partition. Two instances of the service split the partitions between them; a third instance beyond the partition count sits idle. Choose the partition count for the throughput you will need, since raising it later changes how keys map to partitions. Keep related events on one key (the order id), so the shipping step sees OrderPlaced before OrderCancelled. When an instance leaves or joins, Kafka rebalances, and consumption pauses briefly; long processing times between polls cause unwanted rebalances, so set max.poll.interval.ms above your slowest message and keep handlers short.

Test with the Dev Services broker: send with an InMemoryConnector for unit tests (quarkus-test-vertx and the smallrye-reactive-messaging-in-memory helper replace the Kafka connector for a channel) and with the real container in integration tests.

Where this does not apply

If the service needs request-reply semantics, or exactly-once processing across systems, Kafka messaging with these defaults is not enough, and you need transactions or an outbox. The guides are at quarkus.io/guides/kafka and in the SmallRye Reactive Messaging documentation.

Taggedhow to
Share

Keep reading

Get new posts by email

We send one email when a new post is published. No spam.

Unsubscribe in one click. No spam, ever.