Skip to content
javahacks.dev
Microservices

Kafka Rebalance Offsets Java Consumer Handling Guide

Handle Kafka consumer rebalances and offset commits in Java: ConsumerRebalanceListener, manual commits, cooperative rebalancing and avoiding duplicates.

6 min read
Summarize in AI
Flat illustration of a producer, a topic with partitions and consumers, with the Apache Kafka, OpenJDK, Redis and RabbitMQ logos.

A Kafka consumer group shares partitions among its members. When membership changes, partitions move. Handling that move well decides whether you process records twice, lose them, or stall. This applies to Kafka clients 3.x and 4.0, which removed ZooKeeper, with the consumer group protocol improved in recent releases. The basics are in writing a reliable consumer.

Offsets

A consumer tracks an offset per partition: the next record it will read. Committing an offset writes that position to the __consumer_offsets topic. After a restart or a rebalance, the new owner of the partition starts from the last committed offset.

java
Properties props = new Properties();
props.put("bootstrap.servers", "kafka:9092");
props.put("group.id", "orders-projector");
props.put("enable.auto.commit", "false");                 // commit explicitly
props.put("partition.assignment.strategy", "org.apache.kafka.clients.consumer.CooperativeStickyAssignor");
props.put("max.poll.records", "200");
props.put("max.poll.interval.ms", "300000");

With enable.auto.commit=true, the client commits offsets on a timer after poll, whether or not your code finished the records. A crash can lose work that was committed but not processed. For correctness, commit after processing.

Commit after processing

java
try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props, new StringDeserializer(), new StringDeserializer())) {
    consumer.subscribe(List.of("orders"), new CommitOnRevoke(consumer, processed));

    while (running) {
        ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(1));
        for (ConsumerRecord<String, String> record : records) {
            handle(record);
            processed.put(new TopicPartition(record.topic(), record.partition()),
                          new OffsetAndMetadata(record.offset() + 1));
        }
        consumer.commitSync(processed);
        processed.clear();
    }
}

The committed offset is the offset of the next record to read, so you add one. Commit after the loop for a batch, or after each record when duplicates are expensive. A synchronous commit blocks and retries, and the asynchronous one is faster and needs a final synchronous commit at shutdown.

The rebalance listener

java
class CommitOnRevoke implements ConsumerRebalanceListener {
    private final KafkaConsumer<?, ?> consumer;
    private final Map<TopicPartition, OffsetAndMetadata> processed;

    CommitOnRevoke(KafkaConsumer<?, ?> consumer, Map<TopicPartition, OffsetAndMetadata> processed) {
        this.consumer = consumer; this.processed = processed;
    }

    @Override public void onPartitionsRevoked(Collection<TopicPartition> partitions) {
        consumer.commitSync(processed);           // flush what is done before losing the partitions
        processed.keySet().removeAll(partitions);
    }

    @Override public void onPartitionsAssigned(Collection<TopicPartition> partitions) { }
}

onPartitionsRevoked runs before the partitions go to someone else. Committing there means the new owner starts after your last finished record, not earlier. Keep this callback fast, since it blocks the rebalance. Spring Kafka offers the same through ConsumerAwareRebalanceListener and its acknowledgment modes.

Eager versus cooperative

The classic (eager) protocol revokes every partition from every consumer, then reassigns, so the whole group stops during each rebalance. The CooperativeStickyAssignor revokes only the partitions that move, and others keep consuming. It reduces pauses in large groups and during rolling deployments. All members must use it, and a switch from eager needs two rolling restarts. Static membership (group.instance.id) avoids rebalances on quick restarts.

Why unplanned rebalances happen

CauseFix
Processing a batch takes longer than max.poll.interval.msSmaller max.poll.records, faster processing, or hand work to another thread and pause partitions
Heartbeats stop: GC pause, networkTune session.timeout.ms, and check GC, see reading GC logs
Frequent restarts and deploysStatic membership, cooperative assignor
More consumers than partitionsExtra consumers sit idle, partitions set the maximum parallelism

The log line Member ... sending LeaveGroup request ... consumer poll timeout has expired is the poll-interval problem.

Duplicates are normal

Even with careful commits, a record can be delivered twice: the process crashed after handling and before commit, or the rebalance moved the partition. Design consumers to tolerate it with idempotent handling, shown in duplicate and out-of-order messages. If you need stronger guarantees across read and write, see exactly once.

Settings that control when a rebalance happens

Most unplanned rebalances come from three timeouts, and understanding their relationship removes the guesswork. A consumer is removed from the group when it misses heartbeats for session.timeout.ms, or when it goes max.poll.interval.ms without calling poll().

SettingDefaultWhat it controls
heartbeat.interval.ms3000How often the background thread signals liveness
session.timeout.ms45000Silence after which the broker evicts the member
max.poll.interval.ms300000Maximum time between poll() calls
max.poll.records500Records returned by one poll()

The common failure is processing 500 records that take longer than five minutes in total. The thread is alive and the heartbeat continues, and the consumer is still evicted for not polling. The log says consumer poll timeout has expired. Lower max.poll.records so a batch finishes comfortably inside the interval, or move slow work to another thread pool and pause the partitions meanwhile.

Static membership avoids rebalances on restart

A rolling deploy normally causes one rebalance per restarted instance, since each leave and join changes the membership. Giving each instance a stable identity lets the broker hold its partitions during a short absence:

java
props.put(ConsumerConfig.GROUP_INSTANCE_ID_CONFIG, "enricher-" + podOrdinal);
props.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, 60000);
props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG,
        CooperativeStickyAssignor.class.getName());

The restarted instance with the same group.instance.id gets its partitions back with no rebalance, if it returns within the session timeout. The id must be unique per live instance, and two instances using the same id cause one to be fenced with an error. A stateful set in Kubernetes gives each pod a stable ordinal, which suits this. A deployment with random pod names does not, unless the id comes from somewhere stable.

The newer protocol

Kafka 4.0 made the new consumer group protocol from KIP-848 generally available. The coordinator on the broker computes the assignment, and a rebalance no longer stops every member at once. Enable it on the client with group.protocol=consumer, on a cluster that supports it. Client-side assignors and ConsumerRebalanceListener still exist for the classic protocol, and a few settings, session.timeout.ms among them, move to the broker under the new one. Test the switch on a non-production group first, since the behaviour of custom assignors differs.

Look at the group while it rebalances

The command line tool shows what the broker believes, which settles arguments about which consumer holds which partition:

Shell
kafka-consumer-groups.sh --bootstrap-server kafka:9092 --describe --group enricher
kafka-consumer-groups.sh --bootstrap-server kafka:9092 --describe --group enricher --state

The first prints each partition, its current offset, the log end offset, the lag and the owning member. The second prints the group state: Stable, PreparingRebalance or CompletingRebalance. A group that stays in a rebalance state for minutes has a member that is not responding to the join, usually a consumer blocked in a listener callback. Take a thread dump of the instances and look for the poll loop.

Common symptoms

SymptomLikely cause
Repeated rebalances every few minutesProcessing exceeds max.poll.interval.ms
A rebalance on every deployNo static membership, eager assignor
Duplicate processing after a rebalanceOffsets committed after the work, which is expected
Lost recordsOffsets committed before the work finished
CommitFailedExceptionCommit after the partition moved, handle and continue
One consumer idle, others overloadedMore consumers than partitions, or a skewed key

The last row has nothing to do with rebalancing and is often reported as such. A group cannot use more consumers than there are partitions, and a key with most of the traffic keeps one partition hot. Count partitions before adding instances.

Where this does not apply

Consumers that use assign instead of subscribe manage partitions themselves and have no rebalances. Kafka Streams handles offsets itself. The reference is the Kafka consumer documentation, and the [ConsumerRebalanceListener Javadoc](https://kafka.apache.org/40/javadoc/org/apache/kafka/clients/consumer/ConsumerRebalanceListener.html) describes the callbacks.

Share

Keep reading

Get new posts by email

We send one email when a new post is published. No spam.

Unsubscribe in one click. No spam, ever.