Kafka Exactly Once Java Processing With Transactions
Use Kafka exactly once semantics in Java: the idempotent producer, transactions for consume transform produce, isolation level and what it cannot cover.

"Exactly once" is easy to misread. Kafka guarantees that a consume-transform-produce loop, entirely inside Kafka, has the effect of running once. It does not make your email sender or your database exactly once.
Level 1: the idempotent producer
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true); // default since Kafka 3.0
props.put(ProducerConfig.ACKS_CONFIG, "all");Each producer gets an id and numbers its records per partition. If a retry sends a record the broker already has, the broker discards it. This removes duplicates caused by network retries, within one producer session and one partition. It does not stop duplicates when the application sends the same message twice on purpose or after a restart.
Level 2: transactions
A transaction groups sends to many topics and partitions, and the offsets the consumer read, so they all take effect or none do.
Properties producerProps = new Properties();
producerProps.put("bootstrap.servers", "kafka:9092");
producerProps.put("transactional.id", "enricher-1"); // stable per instance
try (KafkaProducer<String, String> producer = new KafkaProducer<>(producerProps, new StringSerializer(), new StringSerializer());
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(consumerProps, new StringDeserializer(), new StringDeserializer())) {
producer.initTransactions();
consumer.subscribe(List.of("orders"));
while (running) {
ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(1));
if (records.isEmpty()) continue;
producer.beginTransaction();
try {
Map<TopicPartition, OffsetAndMetadata> offsets = new HashMap<>();
for (ConsumerRecord<String, String> r : records) {
producer.send(new ProducerRecord<>("orders-enriched", r.key(), enrich(r.value())));
offsets.put(new TopicPartition(r.topic(), r.partition()), new OffsetAndMetadata(r.offset() + 1));
}
producer.sendOffsetsToTransaction(offsets, consumer.groupMetadata());
producer.commitTransaction();
} catch (ProducerFencedException | InvalidProducerEpochException e) {
throw e; // another instance took over: stop
} catch (KafkaException e) {
producer.abortTransaction(); // output discarded, offsets not advanced
}
}
}The consumer has enable.auto.commit=false and isolation.level=read_committed. sendOffsetsToTransaction is the piece that ties the input position to the output: the offsets are committed with the transaction. If the process dies mid-way, the transaction is aborted, and the next owner re-reads from the old offset and redoes the work. The output of the failed attempt is invisible to read_committed readers, so downstream sees each result once.
Fencing
transactional.id must be unique per producer instance and stable across restarts of that instance. When a new instance starts with the same id, the broker bumps an epoch and fences the old one, so a zombie cannot commit. Wrong ids (the same one on two live instances) lead to constant fencing errors. In Kafka Streams, set processing.guarantee=exactly_once_v2 and it handles all of this.
The consumer side
consumerProps.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");With the default read_uncommitted, consumers see records from open and aborted transactions. With read_committed they see only committed ones, and they wait on the last stable offset, so a long-running transaction delays everyone downstream.
Where exactly once stops
Writing to a database inside the loop breaks the guarantee: if the process crashes after the database commit but before the Kafka commit, the record is processed again and the row is written twice. Options:
- Make the database write idempotent (an upsert keyed by the record's identity), the approach in idempotent consumers.
- Store the consumed offset in the same database transaction as the result and seek on startup.
- Use the outbox pattern to publish events from the database.
Calls to HTTP services need idempotency keys, as in idempotency keys.
Cost
Transactions add a round trip for begin and commit, and a coordinator write. Batch several records per transaction (hundreds), not one. Latency for read_committed readers includes transaction duration. Measure on your cluster before adopting it everywhere, and use at-least-once with idempotent handling where it suffices. The offset logic is related to rebalance handling.
Broker settings the transaction depends on
Transactions use an internal topic, __transaction_state, and its defaults assume a cluster of at least three brokers. A single-broker development setup needs the replication factor lowered, or the first initTransactions() call blocks until it times out:
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1Production keeps the defaults, which are 3 and 2. Two related limits matter to applications. transaction.timeout.ms on the producer defaults to 60 seconds, and the broker rejects a value above transaction.max.timeout.ms, which defaults to 15 minutes. A processing loop that holds a transaction open longer than the timeout is aborted by the coordinator, and the next commitTransaction() fails with ProducerFencedException or InvalidProducerEpochException. Keep each transaction to one poll batch, and the timeout never comes into play.
With Spring Kafka
Spring Boot turns on transactions with one property, which makes the producer factory transactional and creates a KafkaTransactionManager:
spring.kafka.producer.transaction-id-prefix=enricher-
spring.kafka.consumer.isolation-level=read_committed
spring.kafka.consumer.enable-auto-commit=falseA listener method then runs inside a transaction, and the offsets are sent with it:
@KafkaListener(topics = "orders", groupId = "enricher")
@Transactional
public void handle(ConsumerRecord<String, String> record) {
kafkaTemplate.send("orders-enriched", record.key(), enrich(record.value()));
}The prefix is combined with a suffix per consumer and partition so ids stay stable across restarts. Do not set the same prefix on two different applications that share a cluster, because their producers would fence each other.
Test it against a real broker
Mocks of Producer accept anything, so they cannot show that a transaction aborts correctly. Start a broker in a container and force a failure halfway through the batch:
@Testcontainers
class ExactlyOnceTest {
@Container
static KafkaContainer kafka = new KafkaContainer(DockerImageName.parse("apache/kafka:3.9.0"));
@Test
void abortedBatchIsInvisibleToReadCommittedConsumers() throws Exception {
// process 10 records, throw on the 7th, then read orders-enriched with read_committed
assertThat(readAll("orders-enriched")).isEmpty();
}
}The assertion that matters is the empty read. A consumer with the default read_uncommitted isolation sees the 6 records written before the failure, and that is the bug the test is protecting against.
What each failure does
| Failure | Result with transactions |
|---|---|
Crash before commitTransaction() | Transaction aborted, input re-read, output hidden |
Crash after commitTransaction() | Both input offset and output are final, nothing repeats |
| Network error on a send | Idempotent producer retries, no duplicate |
| Old instance wakes after a rebalance | Fenced by the epoch and cannot commit |
| Exception thrown by your own code | abortTransaction(), then the same batch is polled again |
Downstream consumer uses read_uncommitted | Sees aborted records, guarantee lost for that reader |
The last row is a property of the reader and not of the writer. A guarantee that depends on every consumer setting is easy to lose with one new service, so add the isolation level to a shared consumer configuration instead of repeating it.
When at-least-once is the better trade
A transaction costs throughput and adds latency for downstream readers, since read_committed consumers wait until the transaction commits. A pipeline where every step is idempotent, such as an upsert keyed by order id, gets the same observable result from plain at-least-once delivery at lower cost. Choose transactions when the output is a stream of events that are not idempotent by nature, such as a running count, and choose idempotent consumers everywhere else.
Monitor what the transaction layer reports
Transactions fail in ways that retries hide, so watch them. The producer exposes metrics for the number of aborted transactions and the time spent in commit, and the consumer exposes lag for read_committed readers. A rise in aborts after a deploy points at a bug in the processing code, and a rise in commit time points at the broker or the network. Alert on the abort rate, and on the lag of the last stable offset, which grows when a transaction stays open.
Log every ProducerFencedException with the instance id. It means two instances claimed the same transactional id, and a deployment that overlaps old and new pods can cause it for a few seconds, which is harmless, or continuously, which is a misconfiguration.
Where this does not apply
If your topology ends at a database, plan idempotency there. The reference is the Kafka documentation on transactions, and the Confluent explanation of exactly-once gives the design background.


