How to Write a Reliable Kafka Consumer in Java and Spring
Write a Kafka consumer that survives restarts, duplicates, poison messages and slow batches, with manual commits, idempotency, retries and a dead letter topic

A Kafka consumer that works in a demo fails in four ways in production. The process restarts between handling a message and committing its offset, so the message is delivered again. One message cannot be parsed, throws on every attempt and halts the partition behind it. A batch takes longer than the poll timeout, so the broker decides the consumer died and moves its partitions to another instance, which processes the same batch. And a rebalance in the middle of a batch leaves half of it committed. None of these is exotic. All are decided by a handful of settings and one design rule: assume every message arrives more than once.
This post builds a consumer in layers. First the delivery guarantee you actually get, then a plain Java loop with manual commits, then idempotent processing with a database constraint, then the Spring for Apache Kafka version with retries and a dead-letter topic, and finally the settings that control rebalances. The code targets Kafka 4 clients and Spring Boot 4 with Spring for Apache Kafka 4; the consumer API shown is unchanged from earlier versions.
The delivery guarantee you actually get
A consumer tracks its position as an offset per partition, committed to the internal __consumer_offsets topic. The order of two operations, processing and committing, decides what a crash costs:
| Order | Crash between the steps | Result |
|---|---|---|
| Commit, then process | Offset is saved, work never done | The message is lost (at most once) |
| Process, then commit | Work is done, offset not saved | The message is processed again (at least once) |
The second is the right default, because duplicates can be made harmless and lost messages cannot be recovered. It requires turning off automatic commits. With enable.auto.commit=true, the client commits on a timer (every 5 seconds by default), independent of whether your code finished, which brings back the first row. Kafka also offers exactly-once semantics inside Kafka, with transactions, for consume-transform-produce loops that write back to Kafka. When the consumer's side effect is a database write or an HTTP call, exactly once is not available, and idempotency is the answer. Plan for at least once.
A plain Java loop
Start without a framework so the mechanics are visible:
import java.time.Duration;
import java.util.List;
import java.util.Map;
import java.util.Properties;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.clients.consumer.OffsetAndMetadata;
import org.apache.kafka.common.TopicPartition;
import org.apache.kafka.common.serialization.StringDeserializer;
public class OrderConsumer {
public static void main(String[] args) {
Properties p = new Properties();
p.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
p.put(ConsumerConfig.GROUP_ID_CONFIG, "order-projector");
p.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false"); // we decide when
p.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest"); // only for a new group
p.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, "100"); // bound the batch
p.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
p.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(p)) {
consumer.subscribe(List.of("orders"));
while (true) {
ConsumerRecords<String, String> batch = consumer.poll(Duration.ofSeconds(1));
for (ConsumerRecord<String, String> record : batch) {
handle(record); // may throw
consumer.commitSync(Map.of(
new TopicPartition(record.topic(), record.partition()),
new OffsetAndMetadata(record.offset() + 1))); // next offset to read
}
}
}
}
static void handle(ConsumerRecord<String, String> record) {
System.out.printf("%s-%d@%d key=%s%n", record.topic(), record.partition(), record.offset(), record.key());
}
}Two details cause most of the bugs. The committed offset is the position of the next record to read, so you commit offset + 1; committing offset replays the last message after every restart. And committing after every record is the safest but slowest choice, since each commitSync is a network round trip. For throughput, commit once per batch after the loop, accepting that a crash replays up to max.poll.records messages. That cost is acceptable only because the next section makes replays harmless.
Make processing idempotent
A message processed twice must have the same effect as once. The reliable way is to record the identity of what you handled in the same database transaction as the effect, with a unique constraint doing the check:
CREATE TABLE processed_event (
event_id VARCHAR(64) PRIMARY KEY,
handled_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
);import org.springframework.dao.DuplicateKeyException;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
@Service
public class OrderProjector {
private final JdbcTemplate jdbc;
public OrderProjector(JdbcTemplate jdbc) {
this.jdbc = jdbc;
}
@Transactional
public void apply(String eventId, String orderId, long amountCents) {
try {
jdbc.update("INSERT INTO processed_event (event_id) VALUES (?)", eventId);
} catch (DuplicateKeyException duplicate) {
return; // already handled: do nothing
}
jdbc.update("UPDATE order_total SET cents = cents + ? WHERE order_id = ?", amountCents, orderId);
}
}The insert and the update commit together or not at all. A redelivery hits the primary key, the method returns and the total is not counted twice. The event id comes from the producer: put a unique id in the message, since the Kafka coordinates (topic-partition-offset) work only as long as nobody republishes into a new topic. For effects that are naturally idempotent, such as UPDATE ... SET status = 'SHIPPED', you can skip the table. The same idea for HTTP endpoints is in idempotent REST endpoints.
Do not take too long between polls
The broker decides a consumer is alive by two timers. The heartbeat thread proves the process is up (session.timeout.ms, 45 seconds by default). Separately, max.poll.interval.ms, 5 minutes by default, bounds the time between two poll() calls. If your batch of 100 messages takes longer than that to process, the client leaves the group, the partitions are reassigned and another instance processes the same batch while the first is still working. The log says:
Member consumer-order-projector-1 sending LeaveGroup request to coordinator ... because consumer poll timeout has expired.The fix is arithmetic. If one message takes up to 2 seconds in the worst case, max.poll.records=100 allows 200 seconds of work, which fits in 300. If one message can take 30 seconds, drop the batch to max.poll.records=10, or raise max.poll.interval.ms, or move slow work off the consumer thread. Do not process on another thread while the consumer thread polls and commits ahead of it, because that reintroduces the lost-message ordering from the table above.
Spring for Apache Kafka
The framework keeps the same rules and supplies the loop. Turn off auto-commit and let the container commit after the listener returns:
spring.kafka.bootstrap-servers=localhost:9092
spring.kafka.consumer.group-id=order-projector
spring.kafka.consumer.enable-auto-commit=false
spring.kafka.consumer.auto-offset-reset=earliest
spring.kafka.consumer.max-poll-records=100
spring.kafka.listener.ack-mode=record
spring.kafka.listener.concurrency=3import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Component;
@Component
public class OrderListener {
private final OrderProjector projector;
public OrderListener(OrderProjector projector) {
this.projector = projector;
}
@KafkaListener(topics = "orders")
public void on(ConsumerRecord<String, String> record) {
projector.apply(record.key(), parseOrderId(record.value()), parseAmount(record.value()));
}
}With ack-mode=record the container commits after each successful listener call. batch commits once after the whole poll, and manual hands you an Acknowledgment to call yourself. concurrency=3 starts three consumer threads; it helps only up to the number of partitions, because a partition is never shared within a group. Keep the partition and key design in mind, see partitions and keys.
Retries and the dead-letter topic
When the listener throws, what happens next is up to the error handler. Spring's default DefaultErrorHandler retries immediately 9 more times and then logs and skips the record, which is almost never what you want. Configure the policy explicitly: retry with a pause, then publish the record to a dead-letter topic and move on.
import org.apache.kafka.common.TopicPartition;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.kafka.core.KafkaOperations;
import org.springframework.kafka.listener.DeadLetterPublishingRecoverer;
import org.springframework.kafka.listener.DefaultErrorHandler;
import org.springframework.util.backoff.ExponentialBackOff;
@Configuration
public class KafkaErrorConfig {
@Bean
DefaultErrorHandler errorHandler(KafkaOperations<Object, Object> template) {
var recoverer = new DeadLetterPublishingRecoverer(template,
(record, ex) -> new TopicPartition(record.topic() + ".DLT", record.partition()));
var backOff = new ExponentialBackOff(500L, 2.0); // 0.5 s, 1 s, 2 s, ...
backOff.setMaxElapsedTime(10_000L); // give up after about 10 s
var handler = new DefaultErrorHandler(recoverer, backOff);
handler.addNotRetryableExceptions(IllegalArgumentException.class); // a bad payload will not heal
return handler;
}
}Spring Boot wires a single CommonErrorHandler bean into the listener container factory. The behavior is: a failure is retried with the delays above, and when the budget is spent the record is written to orders.DLT on the same partition number with headers describing the original topic, offset and exception, and the consumer moves on. Exceptions you list as not retryable go straight to the dead-letter topic, since parsing the same bad JSON ten times helps nobody. The partition in the example assumes the DLT has at least as many partitions as the source; create it that way.
The dead-letter topic is not a trash can. Give it an owner, an alert on its depth and a way to replay records after the cause is fixed. A DLT nobody reads converts an outage into silent data loss.
Deserialization failures
A record that cannot be deserialized fails before your listener runs, inside the consumer's poll, and without protection it throws on every poll for the same offset, so the partition is stuck forever. This is the poison pill. Wrap the deserializer so the failure is captured as a value and handed to the error handler instead:
spring.kafka.consumer.key-deserializer=org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
spring.kafka.consumer.value-deserializer=org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
spring.kafka.consumer.properties.spring.deserializer.value.delegate.class=org.apache.kafka.common.serialization.StringDeserializerThe delegate does the actual decoding. With ErrorHandlingDeserializer, a bad record reaches the error handler as a DeserializationException, which goes to the dead-letter topic with the raw bytes in a header so you can inspect it.
Rebalances
When a consumer joins or leaves, the group rebalances and partitions move. In the middle of a batch, that can mean another instance gets the partition before you committed. Three practices keep this safe. Use the cooperative assignor so a rebalance moves only the partitions that must move (partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor). Give each instance a stable group.instance.id so a quick restart does not trigger a rebalance at all. And rely on the idempotency above, because a rebalance is the moment duplicates are created.
Monitor lag
Lag is the distance between the latest offset in a partition and your committed offset. Read it from the broker:
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group order-projectorA healthy partition and a lagging one look like this:
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
order-projector orders 0 18204 18204 0
order-projector orders 1 17990 18310 320A lag that stays near zero is a consumer keeping up. A lag that grows steadily is a consumer slower than the producer, a stuck partition, or a rebalance loop. Alert on the trend, not on a single number. Long pauses from garbage collection can also trigger rebalances, see reading GC logs.
Where this does not apply
If you consume from Kafka and write back to Kafka, transactions and isolation.level=read_committed give exactly-once processing without a table, and Kafka Streams handles the plumbing. For ordering guarantees across keys, no consumer setting helps: ordering holds only within a partition. The setting reference is in the Kafka consumer configuration docs, and the Spring behaviors above are in the Spring for Apache Kafka reference.


