Skip to content
javahacks.dev
Microservices

How to Configure Kafka Consumers for Back Pressure

Handle slow processing in Kafka consumers: tune max.poll.records and max.poll.interval.ms, pause and resume partitions and bound the work queue.

9 min read
Summarize in AI
Flat illustration of a producer, a topic with partitions and consumers, with the Apache Kafka, Spring Boot, OpenJDK and Redis logos.

In a push system, a slow consumer is a problem for the sender, which must buffer or drop. Kafka reverses the arrangement: consumers pull, and the log keeps the data until retention expires. A consumer that cannot keep up just falls behind, and the lag grows. That is back pressure in its purest form, and it works only if the consumer also survives being slow. The usual failure is not data loss but a rebalance loop: processing a batch takes too long, the broker decides the consumer is dead, reassigns its partitions, the batch is processed again by someone else and the cycle repeats. This post explains the settings behind that behaviour and the patterns that keep a slow consumer healthy: batch sizing, pausing, bounded queues and parallelism. The failure and recovery of consumers is also covered in handling consumer rebalances and offsets and writing a reliable Kafka consumer.

The poll loop and its two clocks

A consumer is a loop. It calls poll, processes the returned records and calls poll again. Two independent clocks watch it.

The heartbeat runs on a background thread. It sends a heartbeat every heartbeat.interval.ms (default 3 seconds), and if the broker hears nothing for session.timeout.ms (default 45 seconds in recent clients) it declares the consumer dead. A consumer whose processing is slow still heartbeats, so the heartbeat alone does not catch a stuck loop.

The poll interval catches that. The consumer must call poll at least every max.poll.interval.ms (default 5 minutes). If it does not, the client leaves the group, and the next poll fails with a commit error:

Text
org.apache.kafka.clients.consumer.CommitFailedException: Offset commit cannot be completed since the consumer is not part of an active group for auto-commit or manual assignment. This is likely because the consumer was kicked out of the group because the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing.

That message is the signature of back pressure that was not handled. The records in the lost batch were not committed, so the new owner of the partitions reads them again: duplicates, and a rebalance that pauses the whole group.

Size one batch to fit the interval

The simplest control is how much work one poll can return. max.poll.records (default 500) caps the records per call. The time to process a batch is the number of records times the time per record, and it has to stay safely under max.poll.interval.ms.

Text
max.poll.records  x  worst-case time per record  <  max.poll.interval.ms  (with margin)

If a record can take up to 2 seconds in the worst case and the interval is 5 minutes, 500 records is 1000 seconds, far over. Either lower the batch to 50 records (100 seconds) or raise the interval. Lowering the batch is the safer lever, since a long interval delays detection of a really stuck consumer.

src/main/resources/application.propertiesproperties
spring.kafka.consumer.max-poll-records=50
spring.kafka.consumer.properties.max.poll.interval.ms=300000
spring.kafka.consumer.properties.fetch.max.bytes=5242880
spring.kafka.consumer.properties.max.partition.fetch.bytes=1048576

fetch.max.bytes and max.partition.fetch.bytes bound how much data the consumer pulls into memory at once, which matters for large records and many partitions: a consumer assigned 50 partitions can hold 50 times max.partition.fetch.bytes in flight. Memory is the other half of back pressure.

A plain consumer that handles slow work

The loop below processes each batch and commits after, so the offset only advances when the work is done. If processing throws, nothing is committed and the batch is retried, which is at-least-once delivery:

src/main/java/dev/javahacks/kafka/SlowConsumer.javajava
public class SlowConsumer implements Runnable {
    private final KafkaConsumer<String, String> consumer;
    private final OrderProcessor processor;
    private volatile boolean running = true;

    public SlowConsumer(Properties props, OrderProcessor processor) {
        this.consumer = new KafkaConsumer<>(props);
        this.processor = processor;
    }

    @Override
    public void run() {
        consumer.subscribe(List.of("order-events"));
        try {
            while (running) {
                ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(1));
                for (ConsumerRecord<String, String> record : records) {
                    processor.process(record);              // may take a while
                }
                if (!records.isEmpty()) {
                    consumer.commitSync();                  // only after the whole batch succeeded
                }
            }
        } finally {
            consumer.close();
        }
    }

    public void stop() {
        running = false;
        consumer.wakeup();                                  // interrupt a blocking poll
    }
}

The properties it relies on: enable.auto.commit=false (otherwise offsets advance on a timer, regardless of your processing), auto.offset.reset set deliberately and max.poll.records sized as above. This loop applies back pressure automatically: while process runs, the consumer is not polling, so it pulls nothing more. The log fills and the lag grows, exactly as designed.

Hand work to other threads: pause and resume

A single threaded loop is simple and limited by one thread's speed. To process records in parallel, many consumers hand work to a pool. Now there is a danger: poll returns quickly, because handing off is fast, so the loop races ahead and fills an unbounded queue until memory is exhausted. This is back pressure lost. The fix is to stop fetching while the queue is full, with pause and resume, which keep the consumer in the group (it still polls and heartbeats) but receive no records for the paused partitions:

src/main/java/dev/javahacks/kafka/PausingConsumer.javajava
public class PausingConsumer implements Runnable {
    private static final int HIGH_WATER = 200;
    private static final int LOW_WATER = 50;

    private final KafkaConsumer<String, String> consumer;
    private final ThreadPoolExecutor workers = new ThreadPoolExecutor(
            8, 8, 0, TimeUnit.SECONDS, new LinkedBlockingQueue<>());
    private boolean paused = false;

    // constructor omitted

    @Override
    public void run() {
        consumer.subscribe(List.of("order-events"));
        while (true) {
            ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
            for (ConsumerRecord<String, String> record : records) {
                workers.submit(() -> process(record));
            }

            int backlog = workers.getQueue().size();
            if (!paused && backlog >= HIGH_WATER) {
                consumer.pause(consumer.assignment());          // stop fetching, keep polling
                paused = true;
            } else if (paused && backlog <= LOW_WATER) {
                consumer.resume(consumer.paused());
                paused = false;
            }
        }
    }
}

The two thresholds form a hysteresis band so the consumer does not flip between states on every poll. While paused, poll returns an empty set after the timeout and still counts as a poll for max.poll.interval.ms. The queue is now effectively bounded around the high water mark plus one batch.

This version has a flaw worth knowing: committing offsets. The records are processed on other threads and finish out of order, so the loop cannot just commit the offset after poll. Committing the offset of a record whose predecessor is still running risks losing that predecessor on a crash. Track the highest contiguous completed offset per partition and commit that:

java
// per partition: commit only up to the first unfinished offset
long safeOffset = completed.get(partition).firstGapAfter(committedOffset);
consumer.commitSync(Map.of(partition, new OffsetAndMetadata(safeOffset)));

The bookkeeping is easy to get wrong, which is why most teams prefer one of the alternatives below instead of writing it.

Spring Kafka: the container does it for you

Spring for Apache Kafka wraps the loop and gives configuration knobs for the same ideas. A listener container runs one consumer thread per concurrency unit and delivers records, or batches, to your method:

src/main/java/dev/javahacks/kafka/OrderListener.javajava
@Component
public class OrderListener {

    private final OrderProcessor processor;

    public OrderListener(OrderProcessor processor) {
        this.processor = processor;
    }

    @KafkaListener(id = "orders-consumer", topics = "order-events", groupId = "billing", concurrency = "4")
    public void on(ConsumerRecord<String, OrderEvent> record, Acknowledgment ack) {
        processor.process(record.value());
        ack.acknowledge();                                    // manual ack after the work succeeds
    }
}
properties
spring.kafka.listener.ack-mode=manual
spring.kafka.listener.concurrency=4
spring.kafka.consumer.enable-auto-commit=false
spring.kafka.consumer.max-poll-records=50

concurrency=4 starts four consumer threads, each with its own KafkaConsumer, sharing the group, so partitions are divided among them. Parallelism is bounded by the partition count: with 12 partitions, 12 consumers can work and a thirteenth is idle. For a batch listener, set spring.kafka.listener.type=batch and accept a List<ConsumerRecord>, so a database write can be done in one statement for the whole batch, which is often the biggest throughput win of all.

To pause the container under a downstream outage, for example when a database is down, the listener container can be paused and resumed programmatically, which is the managed form of pause above:

java
@Component
public class ConsumerControl {
    private final KafkaListenerEndpointRegistry registry;

    public ConsumerControl(KafkaListenerEndpointRegistry registry) {
        this.registry = registry;
    }

    public void pauseOrders() {
        registry.getListenerContainer("orders-consumer").pause();       // stops polling new records
    }

    public void resumeOrders() {
        registry.getListenerContainer("orders-consumer").resume();
    }
}

Hook this to a circuit breaker, so that when calls to the database or to a payment provider start failing, the consumer stops pulling work it cannot do, and resumes when the dependency recovers. That is better than letting every record fail, retry and land in a dead letter topic, which is described in dead letter queues with Kafka and Spring.

Choose the right control

SymptomCauseFix
CommitFailedException, frequent rebalancesBatch takes longer than max.poll.interval.msLower max.poll.records, speed up processing, then raise the interval
Lag grows steadily, CPU lowProcessing is I/O bound and single threadedIncrease concurrency, use a batch listener, process in parallel
Lag grows, consumers at capacityNot enough consumersAdd instances, up to the partition count
Lag grows, partition count is the limitMore parallelism needed than partitionsIncrease partitions (keys remap), or parallelise within a partition
Out of memory in the consumerUnbounded internal queue or huge fetchesPause and resume, lower fetch.max.bytes, bound the queue
A downstream outage floods the dead letter topicFailing fast on every recordPause the container with a circuit breaker

Parallelism within a partition

Kafka guarantees order within a partition, and a pool that processes records out of order gives it up. If order matters per key but not across keys, process in parallel by key: route each key to the same worker so a key's records stay in sequence, with a hash of the key selecting one of N single threaded executors. Throughput scales with the number of distinct keys. If order does not matter at all, a plain pool is fine, and the commit bookkeeping above applies. If you reach for within-partition parallelism often, the partition count is probably too low, a sizing question taken up in Kafka partitions and keys.

Test the slow path

The behaviour that matters is what happens when processing is slower than arrival. A test with a Testcontainers broker produces a burst and a deliberately slow handler, then asserts that the consumer stays in the group and no record is processed twice:

java
@Test
void aSlowHandlerDoesNotCauseARebalanceLoop() throws Exception {
    produce(500);                                                    // a burst
    slowHandler.delayPerRecord(Duration.ofMillis(50));               // 25 s total for the burst

    await().atMost(Duration.ofSeconds(60)).until(() -> handler.processed().size() == 500);

    assertThat(handler.processed()).doesNotHaveDuplicates();
    assertThat(rebalanceListener.revocations()).isLessThanOrEqualTo(1);       // only the initial assignment
}

Running the same test with max.poll.interval.ms=10000 and the batch of 500 shows the failure the settings prevent, which is a good way to convince yourself the numbers matter.

When this does not apply

If your consumer's work is trivial, such as copying records to another store, the defaults are fine. If you need rate control against an external API, a limiter in the handler is the right tool, covered in implementing a rate limiter in Java. Reactive consumers with Reactor Kafka have back pressure expressed in the stream, with limitRate, and a different set of settings. The primary references are the Kafka consumer configuration documentation and the Spring for Apache Kafka reference on listener containers.

Taggedhow to
Share

Keep reading

Get new posts by email

We send one email when a new post is published. No spam.

Unsubscribe in one click. No spam, ever.