Kafka Producer Configuration for Durability in Java Services
A complete guide to Kafka producer durability in Java: acks, replication, idempotence, retries, transactions, the outbox, testing and monitoring.

A producer calls send, no exception is thrown, the application carries on and a week later a customer reports that an order never reached the shipping system. Nothing logged an error. This is the usual shape of message loss in Kafka systems: not a dramatic failure, but a default that favoured speed, a callback that nobody checked, or a database commit that succeeded while the message that was supposed to follow it did not. Kafka can be run so that an acknowledged message survives the loss of a broker, and Java producers can be configured and written so that failures are loud. It takes settings on the topic, settings on the producer and discipline in the code, and all three have to agree. This guide builds the full picture in order: how a write travels, what each setting guarantees, how to configure a plain Java producer and a Spring Boot one, how transactions and the outbox pattern keep a database and a topic consistent, how to test and monitor durability and what each failure scenario looks like. It assumes Kafka 3.x or 4.x clients, JDK 25 and Spring Boot 4 with Spring for Apache Kafka.
The one sentence to keep: a message is durable only after the leader and enough followers have written it, and the producer has been told so and has checked. Every section below is a part of that sentence.
How a write travels through Kafka
A topic is split into partitions. Each partition has one leader broker and zero or more followers that copy the leader's log. A producer sends records for a partition to its leader. The leader appends them to its log, followers fetch and append, and the leader tracks which followers are caught up: the in-sync replicas, ISR. A record is committed once every replica in the ISR has it, and only committed records are visible to consumers. The producer's acks setting decides when the leader answers the producer.
producer --> leader (append) --> followers fetch and append
| |
+---- ISR = {leader, f1, f2} ----+
|
acks=all: reply when every ISR member has the recordIf the leader fails, the controller elects a new leader from the ISR. A follower outside the ISR may lack recent records, so promoting it would lose data; whether that is allowed is a topic setting we meet shortly. The durability question reduces to: what must be true when the producer is told "success" so that a leader crash cannot erase the record?
The topic: replication and min.insync.replicas
Durability starts where the topic is created. Two settings matter most:
| Setting | Meaning | Durable value |
|---|---|---|
replication.factor | How many copies of each partition | 3 |
min.insync.replicas | The minimum ISR size for acks=all to succeed | 2 |
unclean.leader.election.enable | Allow an out-of-sync replica to become leader | false |
With a replication factor of 3 and min.insync.replicas=2, an acks=all write succeeds only when at least two replicas, the leader and one follower, have it. If one broker dies, two replicas remain, writes continue and no acknowledged record is lost. If two brokers are down, the ISR has one member, min.insync.replicas cannot be met and the leader rejects writes with NotEnoughReplicasException. That is the correct behaviour: the cluster refuses a write it cannot protect, trading availability for durability. Set min.insync.replicas equal to the replication factor and a single broker failure blocks writes; set it to 1 and acks=all degrades to the leader's word alone.
Create the topic with these settings explicitly, in code, so they are versioned with the application:
@Configuration
public class TopicConfig {
@Bean
NewTopic orderEvents() {
return TopicBuilder.name("order-events")
.partitions(12)
.replicas(3)
.config(TopicConfig.MIN_IN_SYNC_REPLICAS_CONFIG, "2")
.config(TopicConfig.UNCLEAN_LEADER_ELECTION_ENABLE_CONFIG, "false")
.config(TopicConfig.RETENTION_MS_CONFIG, String.valueOf(Duration.ofDays(7).toMillis()))
.build();
}
}Spring Boot's KafkaAdmin creates the topic at startup if it does not exist, and it does not change an existing topic, so a topic created earlier with defaults keeps them. Check what is in force on a real cluster with the command line tools:
kafka-topics.sh --bootstrap-server broker:9092 --describe --topic order-events
kafka-configs.sh --bootstrap-server broker:9092 --entity-type topics --entity-name order-events --describeTopic: order-events TopicId: 5k2... PartitionCount: 12 ReplicationFactor: 3 Configs: min.insync.replicas=2,unclean.leader.election.enable=false
Topic: order-events Partition: 0 Leader: 1 Replicas: 1,2,3 Isr: 1,2,3Isr: 1,2,3 is healthy. A partition showing Isr: 1 is under-replicated and one broker failure away from being unavailable. The cluster-wide defaults, set on brokers as default.replication.factor=3 and min.insync.replicas=2, protect topics that someone creates without thinking; many managed offerings already set them.
The producer: acks
acks is the producer's choice about whose word to take:
acks | The producer waits for | Loss risk |
|---|---|---|
0 | Nothing; fire and forget | Any network or broker problem loses data, with no signal |
1 | The leader's append | The leader crashes before followers copy: the acknowledged record is lost |
all (or -1) | The leader plus every in-sync replica (at least min.insync.replicas) | None for acknowledged records, if the topic is set up as above |
acks=all is the default since Kafka 3.0 (with idempotence), and you should set it explicitly anyway, because a configuration that states its intent survives version changes. The cost is latency: the write waits for a follower round trip. On a healthy cluster this is milliseconds, and it is the price of the guarantee.
Retries, timeouts and the delivery contract
Temporary failures are normal: a leader election, a network blip, a broker restart. The producer retries them for you, within a bounded time. Older advice tuned retries and retry.backoff.ms separately; the modern model is a delivery timeout: the total time from the call to send until the producer gives up.
| Setting | Default | Role |
|---|---|---|
delivery.timeout.ms | 120000 | Upper bound for send to success or failure, covering batching, retries and waiting |
request.timeout.ms | 30000 | How long to wait for a single broker response |
retries | 2147483647 | Effectively unlimited; the delivery timeout is what stops it |
retry.backoff.ms | 100 | Pause between retries |
linger.ms | 5 (Kafka 4.x) or 0 | How long to wait to fill a batch |
max.block.ms | 60000 | How long send itself may block when the buffer is full or metadata is missing |
The relationship is a rule: delivery.timeout.ms must be at least linger.ms + request.timeout.ms, or the producer rejects the configuration at startup. A message that cannot be delivered within delivery.timeout.ms fails with a TimeoutException, delivered to your callback, and that is the point at which your code decides: log and alert, retry from the beginning or write it to a fallback store.
Idempotence: retries without duplicates
Retries create a problem. If the broker appended a record and the acknowledgement was lost in the network, the producer retries, and the broker appends a second copy. Before idempotence, at least once delivery meant duplicates under failure. Setting enable.idempotence=true fixes it at the producer: the broker assigns the producer an ID, the producer numbers records per partition with a sequence and the broker discards a record whose sequence it has already seen. Retries become safe.
Idempotence is on by default since Kafka 3.0, and it constrains other settings: acks must be all, retries greater than zero and max.in.flight.requests.per.connection at most 5. Within those limits ordering per partition is preserved even across retries, because the sequence numbers let the broker reject out of order batches. Without idempotence, more than one in-flight request plus retries can reorder records: a failed batch is retried after a later batch succeeds. If your design relies on per key ordering, idempotence is not optional.
A plain Java producer, configured for durability
Put the pieces together in the Kafka client without a framework. Everything is explicit, which makes it a good reference:
package dev.javahacks.kafka;
import org.apache.kafka.clients.producer.*;
import org.apache.kafka.common.errors.*;
import org.apache.kafka.common.serialization.StringSerializer;
import java.util.Properties;
import java.util.concurrent.TimeUnit;
public class DurableProducer implements AutoCloseable {
private final KafkaProducer<String, String> producer;
public DurableProducer(String bootstrapServers) {
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.ACKS_CONFIG, "all");
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true");
props.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, "5");
props.put(ProducerConfig.DELIVERY_TIMEOUT_MS_CONFIG, "60000");
props.put(ProducerConfig.REQUEST_TIMEOUT_MS_CONFIG, "15000");
props.put(ProducerConfig.LINGER_MS_CONFIG, "10");
props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "lz4");
props.put(ProducerConfig.CLIENT_ID_CONFIG, "orders-service");
this.producer = new KafkaProducer<>(props);
}
public void sendAsync(String topic, String key, String value, FailureHandler onFailure) {
producer.send(new ProducerRecord<>(topic, key, value), (metadata, exception) -> {
if (exception != null) {
onFailure.handle(topic, key, value, exception); // the one place a loss becomes visible
}
});
}
public RecordMetadata sendAndWait(String topic, String key, String value) throws Exception {
return producer.send(new ProducerRecord<>(topic, key, value)).get(30, TimeUnit.SECONDS);
}
@Override
public void close() {
producer.flush(); // push out whatever is still buffered
producer.close();
}
@FunctionalInterface
public interface FailureHandler {
void handle(String topic, String key, String value, Exception cause);
}
}Read the choices. acks, enable.idempotence and max.in.flight.requests.per.connection state the contract; relying on defaults hides it. delivery.timeout.ms at 60 seconds bounds how long a message can sit retrying, which should match how long your caller can reasonably wait. The compression type lz4 costs little CPU and reduces network and disk, and compression applies per batch, so it works better with a little linger.ms. client.id shows up in broker logs and metrics, which makes the producer identifiable when you debug. The close method flushes first: close() without flush() waits for in-flight batches up to a default timeout, but flushing explicitly makes the intent clear, and a service shutting down without it loses whatever was buffered.
send is asynchronous: check the result
The method send returns immediately after putting the record in an in-memory buffer. The actual write happens on a background thread, later. Whether it worked is known only through the returned Future or the callback. Code that calls producer.send(record) and moves on has told nobody about a failure:
producer.send(new ProducerRecord<>("order-events", orderId, json)); // the result is thrown awayIf the broker is down, the delivery timeout expires 60 seconds later and the failure lands in a future nobody is reading. The record is gone. Two disciplined patterns:
Fire with a callback, as sendAsync above, for throughput. The callback runs on the producer's I/O thread, so keep it short: log, count a metric, hand the failure to a fallback. Never block or call send from inside the callback.
Wait for the future, as sendAndWait, when the caller needs to know before proceeding, for instance before acknowledging an HTTP request. It limits throughput to one round trip per call from that thread, so use several threads, or collect futures and wait for them together.
Errors are of two kinds, and the callback receives them as the exception:
| Exception | Meaning | Action |
|---|---|---|
TimeoutException | Delivery timeout reached | Alert; resend from a fallback store |
NotEnoughReplicasException, NotEnoughReplicasAfterAppendException | ISR smaller than min.insync.replicas | Retriable by the producer; if it persists, the cluster is degraded |
RecordTooLargeException | Record exceeds max.request.size or the broker's limit | Not retriable; fix the message size, see below |
SerializationException | The serializer failed | Not retriable; a bug in your data |
AuthorizationException, InvalidTopicException | Configuration or ACL problem | Not retriable; fix the setup |
ProducerFencedException | A newer producer with the same transactional.id took over | Close this producer |
Retriable errors are retried by the client until the delivery timeout; you see them only when time runs out. Non retriable ones come back immediately, and each needs a different human response, which is why a generic "log and ignore" handler is the beginning of a lost-message story.
A failure handler worth having writes the failed record to somewhere durable you control, such as a table, a local spill file or a dead letter topic on a different path, and raises an alert:
FailureHandler spill = (topic, key, value, cause) -> {
log.error("kafka send failed topic={} key={} cause={}", topic, key, cause.toString());
failedMessages.save(new FailedMessage(topic, key, value, cause.getClass().getSimpleName(), Instant.now()));
meterRegistry.counter("kafka.producer.failed", "topic", topic).increment();
};A scheduled job can retry from the table. That turns an invisible loss into a visible queue.
Batching and throughput: what durability costs
Durability settings have a cost, and the right knobs claw back throughput without weakening the guarantee. Kafka producers batch records per partition:
| Setting | Effect | Tuning |
|---|---|---|
linger.ms | Wait this long for more records before sending a batch | 5 to 20 ms for throughput; 0 for minimum latency |
batch.size | Maximum bytes in a batch | 32 to 128 KB when records are small and traffic is heavy |
compression.type | Compress batches | lz4 or zstd; zstd compresses better for text and JSON |
buffer.memory | Total memory for unsent records | Raise if send blocks (max.block.ms) under bursts |
max.in.flight.requests.per.connection | Concurrent unacknowledged requests | Keep at 5 or fewer with idempotence |
A small linger.ms is nearly free in latency terms and can multiply throughput, because more records share each request and each compressed batch. None of these affect the durability guarantee, which is why they are the right place to tune. What does weaken it, and should not be used to buy speed, is acks below all, min.insync.replicas of 1, or ignoring results.
The same thing in Spring Boot
Spring Boot with Spring for Apache Kafka sets up the producer from properties:
spring:
kafka:
bootstrap-servers: ${KAFKA_BOOTSTRAP:localhost:9092}
producer:
acks: all
compression-type: lz4
key-serializer: org.apache.kafka.common.serialization.StringSerializer
value-serializer: org.springframework.kafka.support.serializer.JsonSerializer
properties:
enable.idempotence: true
max.in.flight.requests.per.connection: 5
delivery.timeout.ms: 60000
request.timeout.ms: 15000
linger.ms: 10
client.id: orders-serviceAnd the code that sends has to honour the asynchronous contract. KafkaTemplate.send returns a CompletableFuture<SendResult>:
@Component
public class OrderEventPublisher {
private static final Logger log = LoggerFactory.getLogger(OrderEventPublisher.class);
private final KafkaTemplate<String, OrderEvent> kafka;
private final FailedMessageStore failed;
private final MeterRegistry metrics;
public OrderEventPublisher(KafkaTemplate<String, OrderEvent> kafka, FailedMessageStore failed, MeterRegistry metrics) {
this.kafka = kafka;
this.failed = failed;
this.metrics = metrics;
}
public void publish(OrderEvent event) {
kafka.send("order-events", event.orderId(), event).whenComplete((result, error) -> {
if (error != null) {
log.error("could not publish order event {}", event.orderId(), error);
failed.save("order-events", event.orderId(), event, error);
metrics.counter("orders.events.failed").increment();
} else {
log.debug("published {} to partition {} offset {}",
event.orderId(), result.getRecordMetadata().partition(), result.getRecordMetadata().offset());
}
});
}
public SendResult<String, OrderEvent> publishAndWait(OrderEvent event) throws Exception {
return kafka.send("order-events", event.orderId(), event).get(30, TimeUnit.SECONDS);
}
}whenComplete is the callback, and the handler stores the failed message, logs with the exception and increments a metric. Do not call kafka.send(...) and discard the future. A Spring Boot application can also set a ProducerListener bean that logs every failure globally, as a safety net for places that forgot:
@Bean
ProducerListener<String, OrderEvent> producerListener() {
return new ProducerListener<>() {
@Override
public void onError(ProducerRecord<String, OrderEvent> record, RecordMetadata metadata, Exception exception) {
log.error("producer error for topic {} key {}", record.topic(), record.key(), exception);
}
};
}Large messages
A record larger than the limit fails immediately and permanently. The error message is the search query:
org.apache.kafka.common.errors.RecordTooLargeException: The message is 1048611 bytes when serialized which is larger than 1048576, which is the value of the max.request.size configuration.Three limits must agree: the producer's max.request.size (default 1 MB), the broker's message.max.bytes and the topic's max.message.bytes, plus the consumer's fetch.max.bytes. Raising them all works, and large records hurt: they raise memory use on every client and slow replication. A better design stores big payloads (documents, images) in object storage and sends a reference in the event, the claim check pattern. A message that is a path and some metadata is a few hundred bytes.
Transactions: atomic writes across topics
Idempotence protects a single partition session. A transaction makes writes to several partitions, and the consumer offsets of a consume-process-produce loop, atomic: all become visible together or none do. Enable it by giving the producer a stable transactional.id:
public void publishBoth(String orderId, String paid, String shipment) {
producer.initTransactions(); // once per producer, fences any older instance
try {
producer.beginTransaction();
producer.send(new ProducerRecord<>("payments", orderId, paid));
producer.send(new ProducerRecord<>("shipments", orderId, shipment));
producer.commitTransaction();
} catch (ProducerFencedException | OutOfOrderSequenceException | AuthorizationException e) {
producer.close(); // fatal: another instance owns this transactional.id
throw e;
} catch (KafkaException e) {
producer.abortTransaction(); // recoverable: roll back and retry the unit of work
throw e;
}
}Consumers see only committed transactions when they set isolation.level=read_committed; with the default read_uncommitted they also see records from aborted transactions. In Spring, set spring.kafka.producer.transaction-id-prefix=orders-tx- and the template becomes transactional, with KafkaTransactionManager used by @Transactional. The exactly-once semantics of the consume-transform-produce loop, and why they hold only inside Kafka, are the subject of Kafka exactly-once processing with transactions, and the general limits of the idea are in exactly-once delivery and the two generals.
Transactions have costs: commit markers, higher latency, a fenced-producer failure mode and the requirement that each producer instance use a unique, stable transactional.id. Use them where atomicity across topics or the read-process-write loop is the requirement, not as a default.
The dual write problem and the outbox
The most common durability failure has nothing to do with Kafka's settings. A service saves an order to its database and publishes an event:
@Transactional
public void placeOrder(Order order) {
orders.save(order); // commit happens when the method returns
kafka.send("order-events", ...); // happens before the commit, not atomically with it
}Two systems, two writes, no shared transaction. If the send succeeds and the commit then fails, the event announces an order that does not exist. If the commit succeeds and the process dies before the send, the order exists and no event ever goes out. Reordering the lines only swaps one failure for the other. Retrying cannot close the gap, since the crash that loses the send also loses the memory of needing to send it.
The reliable fix is the transactional outbox. Write the event into an outbox table in the same database transaction as the business change, so both commit or neither does, and have a separate relay publish rows from the table to Kafka:
@Service
public class OrderService {
private final OrderRepository orders;
private final OutboxRepository outbox;
private final ObjectMapper json;
// constructor omitted
@Transactional
public void placeOrder(Order order) throws JsonProcessingException {
orders.save(order);
outbox.save(new OutboxEvent(UUID.randomUUID(), "order-events", order.getId().toString(),
json.writeValueAsString(OrderEvent.placed(order)), Instant.now()));
}
}@Component
public class OutboxRelay {
private final OutboxRepository outbox;
private final KafkaTemplate<String, String> kafka;
// constructor omitted
@Scheduled(fixedDelay = 500)
public void publishPending() {
for (OutboxEvent event : outbox.findTop100ByPublishedAtIsNullOrderByCreatedAt()) {
try {
kafka.send(event.topic(), event.key(), event.payload()).get(10, TimeUnit.SECONDS);
outbox.markPublished(event.id(), Instant.now());
} catch (Exception e) {
log.warn("outbox publish failed for {}, will retry", event.id(), e);
return; // keep order: stop at the first failure
}
}
}
}The relay gives at least once delivery: a crash between the send and markPublished publishes the event again on restart. That is acceptable when consumers are idempotent, using the event id, see handling duplicate messages with an idempotent consumer. Running more than one relay needs a locking scheme so rows are not published concurrently (a SELECT ... FOR UPDATE SKIP LOCKED, covered in pessimistic locking in JPA). For high volume, replace the polling relay with change data capture reading the database log, described in the outbox pattern with Debezium. Either way the database becomes the durable record of intent, and Kafka a reliable delivery channel from it.
Test durability
You can test the producer's configuration and behaviour with a real broker in a container. These tests are quick to write with Testcontainers and catch the mistakes that matter: a misconfigured producer, a topic created with the wrong settings and a callback that is never invoked.
@Testcontainers
class DurableProducerIT {
@Container
static KafkaContainer kafka = new KafkaContainer(DockerImageName.parse("apache/kafka:3.9.0"));
@Test
void acknowledgedRecordsAreReadableAndOrderedPerKey() throws Exception {
try (var producer = new DurableProducer(kafka.getBootstrapServers())) {
for (int i = 0; i < 100; i++) {
producer.sendAndWait("order-events", "order-42", "event-" + i);
}
}
List<String> values = readAll("order-events", kafka.getBootstrapServers());
assertThat(values).hasSize(100);
assertThat(values).isSortedAccordingTo(Comparator.comparingInt(v -> Integer.parseInt(v.substring(6))));
}
@Test
void anUnreachableBrokerSurfacesAsAFailureNotASilentLoss() {
var failures = new CopyOnWriteArrayList<Exception>();
try (var producer = new DurableProducer("localhost:1")) { // nothing listens here
producer.sendAsync("order-events", "k", "v", (t, k, v, e) -> failures.add(e));
}
await().atMost(Duration.ofSeconds(90)).until(() -> !failures.isEmpty());
assertThat(failures.get(0)).isInstanceOf(TimeoutException.class);
}
}The second test is the valuable one: it proves that a dead broker produces a callback with an exception, within the delivery timeout, and so your failure handler actually gets called. For a smaller timeout in tests, make delivery.timeout.ms a constructor parameter. To test the min.insync.replicas behaviour you need a multi broker cluster, which Testcontainers can start with a Kafka cluster module or a Compose file, then stop two brokers and assert that sends fail with NotEnoughReplicasException and recover when a broker returns. That is more setup, and worth doing once, as a staging test. Testing Kafka consumers with Testcontainers covers the consuming side.
Reading offsets and metadata back
A producer that wants proof of durability can ask for it. The RecordMetadata returned by a successful send names the partition and offset the record was written at, and with acks=all that offset is committed. Logging them at debug level, as the Spring publisher does, gives support engineers a way to answer "was this event written": search the log for the order id and read the partition and offset, then confirm with the console consumer that the record is at that position.
kafka-console-consumer.sh --bootstrap-server broker:9092 --topic order-events \
--partition 3 --offset 118402 --max-messages 1 --property print.key=true --property print.timestamp=trueCreateTime:1793672042331 order-42 {"type":"ORDER_PLACED","orderId":"order-42"}The timestamp, key and value come back, which closes the loop between the application's belief and the broker's log. When a downstream team says the event never arrived, this check settles in one minute whether the problem is on the producing side or the consuming side, and it replaces a meeting. Make it part of the runbook for the service, with the exact commands and the broker addresses for each environment.
Monitor the things that precede loss
Durability fails slowly and then suddenly. Watch producer and broker signals:
| Metric | Where | What it tells you |
|---|---|---|
record-error-rate, record-error-total | Producer | Records that failed after retries |
record-retry-rate | Producer | Trouble reaching a leader; rising means instability |
request-latency-avg, request-latency-max | Producer | Slow brokers or network |
buffer-available-bytes, bufferpool-wait-ratio | Producer | Buffer pressure; send about to block |
UnderReplicatedPartitions | Broker | Followers behind; one failure from trouble |
IsrShrinksPerSec, IsrExpandsPerSec | Broker | Replicas falling out of and back into sync |
OfflinePartitionsCount | Controller | Partitions with no leader; writes failing |
Micrometer exposes the client's metrics with one line:
@Bean
MeterBinder kafkaProducerMetrics(ProducerFactory<String, OrderEvent> factory) {
return registry -> new KafkaClientMetrics(factory.createProducer()).bindTo(registry);
}Spring Boot also binds the producer metrics to the registry automatically in recent versions when Actuator and Micrometer are present, tagged by client.id. Alert on record-error-total increasing, on retries sustained above a baseline and on any under-replicated partitions for more than a few minutes. Consumer lag is the other half; see monitoring Kafka consumer lag from Java.
Inside the producer: where a record waits
Knowing what happens between send and the network explains most of the settings. A KafkaProducer is a small pipeline. When you call send, the calling thread serialises the key and value, picks a partition (from the key's hash, or round robin when the key is null) and appends the record to a record accumulator, an in-memory structure holding one queue of batches per partition. The call returns at that point; the record has not left the process. A separate sender thread drains the accumulator: it collects ready batches, groups them by destination broker and sends one request per broker. A batch is ready when it is full (batch.size), when it has waited linger.ms or when the producer is flushing or closing.
Three consequences follow. First, a record sits in memory for at least linger.ms and until a response arrives, so a crash during that window loses it, which is the gap the outbox closes. Second, if the sender cannot keep up, for instance because the broker is slow, the accumulator fills to buffer.memory, and send blocks for up to max.block.ms and then throws TimeoutException: Failed to allocate memory within the configured max blocking time. That is backpressure reaching your request thread, and it is a signal to read, not to silence by raising the buffer. Third, a batch that failed with a retriable error goes back to the head of its partition's queue, which is how ordering is preserved under idempotence.
The partitioner matters for durability in one respect: it fixes which leader a record goes to. With a key, all records for that key go to one partition and keep their order; with no key, records are spread, and ordering across them is not defined. If an event stream for one order must stay ordered, key it by order id, as Kafka partitions and keys explains.
Serialization is a failure point too
A serializer that throws on one bad object stops the record before it reaches the accumulator, as a SerializationException raised synchronously from send, unlike network errors which arrive in the callback. With Spring's JsonSerializer, a type that Jackson cannot write, a cyclic object graph or a missing module for java.time, fails here:
org.apache.kafka.common.errors.SerializationException: Can't convert value of class dev.javahacks.OrderEvent to class org.apache.kafka.common.serialization.StringSerializer specified in value.serializerThis particular message means the serializer configured does not match the type sent, a configuration mistake that tests catch immediately. Handle synchronous exceptions around send as well as the asynchronous ones, or an unexpected payload takes down the thread that was publishing. For events that will be consumed by other teams, a schema registry with Avro or Protobuf adds a compatibility check at produce time, so an incompatible change fails in the producer instead of poisoning consumers. The approach is described in versioning events with Avro and a schema registry. Durability is also about being able to read the data later, and a record that no consumer can deserialise is lost in every way that matters.
Operating durable producers through broker restarts
The settings pay off during routine operations. Rolling a Kafka cluster, upgrading brokers or patching hosts, is done one broker at a time. With replication factor 3 and min.insync.replicas=2, taking one broker down leaves two replicas, writes continue and producers see brief, retriable errors for the partitions whose leader moved: NOT_LEADER_OR_FOLLOWER, handled inside the client by refreshing metadata and retrying, which idempotence makes safe. The operator's job is to wait between restarts until the restarted broker has rejoined every ISR, which means no under-replicated partitions, before touching the next one. If that wait is skipped, two replicas of a partition are briefly absent, the ISR drops below the minimum and producers receive NotEnoughReplicasException; with acks=all and a sensible delivery timeout the writes are retried until the broker returns and nothing is lost, but a timeout that is too short turns the rolling restart into failed sends.
This is the strongest argument for stating the delivery timeout deliberately. A restart of a broker typically takes tens of seconds to a few minutes for the ISR to recover. A delivery.timeout.ms of 30 seconds fails sends during a normal maintenance window, while 120 seconds rides through it. Set it from the worst maintenance event you accept, and compare it with how long the caller can wait. If the caller cannot wait that long, the answer is the outbox, so the request returns after the database commit and the relay absorbs the broker's slow minutes.
Disk, retention and what "durable" means over time
Durability in Kafka is bounded by retention. A record is kept for retention.ms (seven days by default) or until a partition exceeds retention.bytes, then it is deleted whether or not any consumer has read it. A consumer that is down for longer than the retention loses the records it did not read, and no producer setting prevents that. Size the retention to the longest outage you intend to survive, plus the time to catch up, and monitor consumer lag against it. Topics that hold the source of truth, for example a changelog that rebuilds state, use log compaction (cleanup.policy=compact), which keeps the latest record per key indefinitely instead of deleting by age. Disk is the other bound: a full broker disk stops writes, so alert on usage well before it fills, and remember replication multiplies the storage, three copies of everything.
Notes for Kafka 4
Kafka 4.0 removed ZooKeeper, so clusters run in KRaft mode, and raised the minimum supported client and broker Java versions. For producers the visible changes are small: the default linger.ms became 5, so batches form even with the default configuration, and the idempotent, acks=all configuration is the baseline. Clients from the 3.x line still work with a 4.x cluster for the supported compatibility range, and the durable settings in this guide are unchanged. Read the upgrade notes for your exact versions before changing a production cluster, and test the producer against the new broker version with the Testcontainers image set to match.
A worked incident: reading the producer log
A concrete case shows how the settings and the code combine when something goes wrong. A deployment restarts one of three brokers while the orders service is publishing. The service logs, in order:
WARN [Producer clientId=orders-service] Got error produce response with correlation id 8814 on topic-partition order-events-3, retrying (2147483646 attempts left). Error: NOT_LEADER_OR_FOLLOWER
INFO [Producer clientId=orders-service] Resetting the last seen epoch of partition order-events-3 to 41 since the associated topicId changed
WARN [Producer clientId=orders-service] Got error produce response with correlation id 8920 on topic-partition order-events-7, retrying (2147483645 attempts left). Error: NOT_ENOUGH_REPLICAS
INFO orders.events.published partition=3 offset=118402Read it line by line. NOT_LEADER_OR_FOLLOWER on partition 3 is the leader moving off the restarting broker: the client refreshed its metadata and retried, and nothing needs doing. The "attempts left" number is huge because retries are bounded by time, not count. NOT_ENOUGH_REPLICAS on partition 7 is the more serious sign: for a moment that partition had fewer in-sync replicas than min.insync.replicas, which happens when a second replica is slow or restarting while the first is down. The producer kept retrying, the replicas caught up and the final line shows a successful publish with an offset. No callback fired with an error, no message was lost, and the only cost was latency on those two partitions for a short time.
Now change one thing: set delivery.timeout.ms to 5000 and repeat the restart. The same warnings appear, but after five seconds the callback receives TimeoutException: Expiring 14 record(s) for order-events-7: 5003 ms has passed since batch creation. Fourteen records are now the application's problem, and the failure handler decides what happens to them. That contrast is the point of this guide in one example: the infrastructure behaves identically, and what changes between "the restart was invisible" and "we lost fourteen orders" is the timeout and the handler.
Failure scenarios, and what each setting does
| Scenario | With the durable settings | With weak settings |
|---|---|---|
| A follower broker restarts | ISR shrinks to 2, writes continue, follower catches up | Same |
| The leader crashes mid-write | The producer's request fails or times out, retries to the new leader, idempotence prevents a duplicate | acks=1: an acknowledged record may be lost |
| Two of three brokers are down | Writes fail with NotEnoughReplicasException; no data lost, availability reduced | min.insync.replicas=1: writes continue on one replica and are lost if it dies |
| A network partition isolates the producer | TimeoutException after the delivery timeout, callback fires | acks=0: no signal at all |
| The application crashes with records in the buffer | Buffered records are lost; records acknowledged before the crash are safe | Same, which is why the outbox exists |
| The application is shut down cleanly | flush and close send what remains | close without flush can drop the tail |
| The broker disk fills | The broker stops accepting writes; producers time out | Same, monitor disk |
| A duplicate is sent because the ack was lost | Idempotence discards it | Without idempotence: two copies in the topic |
The one scenario the producer settings cannot protect against is a crash with unsent records in memory, which is the reason the outbox pattern exists and the reason to prefer it when a message must follow a database change.
Three configuration profiles
Different workloads sit at different points:
| Setting | Maximum durability | Balanced (the default recommendation) | Throughput first, loss tolerable |
|---|---|---|---|
acks | all | all | 1 |
enable.idempotence | true | true | false |
Topic replication / min.insync.replicas | 3 / 2 | 3 / 2 | 3 / 1 |
delivery.timeout.ms | 120000 | 60000 | 30000 |
linger.ms | 0 to 5 | 10 | 50 |
compression.type | lz4 | lz4 | zstd |
| Transactions | when atomicity needed | rarely | no |
| Result handling | Wait or callback with spill | Callback with spill | Callback and metric |
Use the first or second profile for business events: orders, payments, state changes. The third suits telemetry, clickstream and logs where occasional loss is acceptable, and even there acks=all is rarely the bottleneck.
Common mistakes
Ignoring the future. The commonest, covered above.
Relying on defaults across versions. The defaults changed in Kafka 3.0; code that worked unchanged for years may be running with settings its authors never saw. State the durability settings in your configuration.
Topic settings forgotten. acks=all with min.insync.replicas=1 on a topic created by auto creation is not durable. Disable auto topic creation in production (auto.create.topics.enable=false) and create topics explicitly.
Message keys that skew. A key with one dominant value sends most traffic to one partition, which is a throughput and availability issue more than durability, treated in Kafka partitions and keys.
Blocking inside the callback. The callback runs on the producer's network thread; blocking it stalls all sends.
Creating a producer per message. A KafkaProducer is thread safe and expensive to build. Create one per application, or one per transactional.id when using transactions, and share it.
Unbounded retries without a timeout. Older configurations set retries=Integer.MAX_VALUE with max.block.ms unlimited, and a broker outage made request threads hang forever. Use the delivery timeout and a bounded wait.
Treating success as end to end. A durable producer gets the record safely into Kafka. Whether a consumer processes it exactly once is a separate design problem, see writing a reliable Kafka consumer and handling dead letters.
A review checklist
- Topic: replication factor 3,
min.insync.replicas=2, unclean election off, created explicitly. - Producer:
acks=all,enable.idempotence=true, in-flight at most 5, delivery timeout stated. - Code: every send has a callback or a waited future; failures are logged, counted and stored.
- Shutdown:
flushthenclose, and the application waits for in-flight work first. - Consistency: a database change that must produce an event uses an outbox or a transaction, not two independent writes.
- Consumers:
isolation.level=read_committedif transactions are used; idempotent processing. - Tests: a broker-down test proving the callback fires; a staging test with a broker killed.
- Monitoring: producer error and retry rates, under-replicated partitions, alerts wired and tried.
A last check before production
Before a producer goes live, send a message with the broker's leader killed and watch the result. The send should either succeed after a short retry or fail with an exception your handler catches, with no third outcome. Run the same experiment with the broker count reduced below min.insync.replicas and confirm the failure is loud. A durability setting that has never been seen to fail safely is a belief, not a property, and this ten minute test replaces the belief with evidence.
When this does not apply
If a message can be lost without anyone caring, such as a debug trace or a sampled metric, the cheaper settings are the right ones, and over-engineering durability wastes latency and money. If you cannot control the topic, because another team owns the cluster and its defaults, the producer settings still help but the guarantee is the weaker of the two sides; ask for the topic settings in writing. Kafka is not the only way to move events reliably between services: a database queue or a broker with built in acknowledgements, as in using RabbitMQ acks, retries and dead letters, is simpler if you do not need Kafka's log semantics. For the primary sources, read the Apache Kafka documentation on the producer configs, the section on design and replication and the Spring for Apache Kafka reference.


