Two Generals and Why Exactly Once Delivery Is Hard
Why exactly once delivery is hard: the Two Generals problem, at-least-once and at-most-once semantics, and how Kafka and idempotence give effectively once.

Message systems advertise delivery guarantees. "Exactly once" sounds like the right one to want, and it is the one that cannot be had in the pure sense. Understanding why explains how to design around it.
The Two Generals problem
Two armies, separated by a valley, must attack at the same time. They communicate by messengers who may be captured. General A sends "attack at dawn". A cannot attack unless B acknowledges. B sends an acknowledgement, but now B cannot be sure it arrived, so B cannot attack unless A confirms receipt of the acknowledgement. Each confirmation needs a confirmation. No finite exchange gives certainty.
Applied to services: a producer sends a message, the broker stores it, and the acknowledgement is lost. The producer sees a timeout. It cannot tell whether the message was stored or lost, and has two choices.
The two real guarantees
- At-most-once: do not retry. If the message was lost, it stays lost.
- At-least-once: retry until acknowledged. If the acknowledgement was lost, the message is delivered twice.
// at-least-once: retry on timeout
for (int attempt = 0; attempt < 3; attempt++) {
try {
broker.send(message);
break;
} catch (TimeoutException e) {
// maybe sent, maybe not; send again
}
}Nothing in this loop can remove the possibility of a duplicate.
Effectively once
What systems offer is at-least-once delivery plus a way to make duplicates harmless:
- Idempotent operations. Setting a status to
PAIDtwice has the same result as once. - Deduplication. Store processed message ids and skip repeats.
- Transactions. Process and record the offset atomically.
@Transactional
public void handle(PaymentEvent e) {
if (processed.existsById(e.id())) return; // dedupe
ledger.credit(e.account(), e.amount());
processed.save(new Processed(e.id())); // same transaction
}Because the effect and the record of it commit together, a crash and redelivery cannot apply the credit twice. See idempotent consumers.
What Kafka offers
Kafka has an idempotent producer, which assigns sequence numbers so that a retried send does not create a duplicate in the log, and transactions, which write to several partitions and commit consumer offsets atomically. With isolation.level=read_committed, a Kafka-to-Kafka pipeline can process records exactly once. The details are in Kafka exactly-once processing.
The guarantee ends at the boundary of Kafka. If a consumer calls a payment API or writes to a database, a crash after the call and before the offset commit repeats the call. The external call needs an idempotency key, as in idempotent REST endpoints, or the database write must go with the offset in the same transaction.
Design rules
- Assume every message can arrive twice and out of order.
- Give every command or event a unique id.
- Make handlers idempotent by construction (upserts, state transitions).
- Use idempotency keys when calling external services.
- Treat "exactly once" claims with suspicion and ask what they cover.
Related ideas
The same impossibility shapes consensus protocols, which use majorities and retries to agree. For the consistency angle, see CAP and what it means for services and eventual consistency.
An idempotent consumer in thirty lines
The practical answer to the impossibility is to make processing safe to repeat. A table of handled message ids with a unique constraint does it, and the check and the effect share one transaction:
@Service
class PaymentHandler {
private final JdbcTemplate jdbc;
PaymentHandler(JdbcTemplate jdbc) {
this.jdbc = jdbc;
}
@Transactional
public void handle(String messageId, Payment payment) {
int inserted = jdbc.update(
"insert into processed_message (id) values (?) on conflict do nothing", messageId);
if (inserted == 0) {
return; // seen before, skip
}
jdbc.update("update account set balance = balance + ? where id = ?",
payment.amount(), payment.accountId());
}
}The insert and the balance update commit together or not at all. A redelivered message hits the conflict and returns, so the second delivery changes nothing. The sentence "exactly once" is then true of the effect on the account, though not of the delivery. The syntax shown is PostgreSQL's, and MySQL uses insert ignore for the same purpose.
How long to keep the ids
A table of processed ids grows without bound, and the retention period is a design decision with a real consequence. Delivery duplicates arrive within the retry window of the system that sends them, which is minutes for most brokers and days for a dead-letter replay. Keep ids for longer than the longest replay anyone can trigger. A seven day window covers normal retries and most replays. A delete job that removes older rows keeps the table small, and a partitioned table by day makes the delete a cheap drop of a partition.
The producer's half: the outbox
The consumer is only half of the pair. A service that writes to its database and then publishes a message has the Two Generals problem inside one process: the commit can succeed and the publish fail, or the reverse. The outbox pattern writes the message to a table in the same transaction as the data, and a separate relay publishes rows from that table:
| Step | Failure | Result |
|---|---|---|
| Transaction writes data and outbox row | Crash before commit | Neither exists, consistent |
| Relay reads the row and publishes | Crash after publish, before marking sent | Published again on restart |
| Consumer receives the duplicate | Idempotent handler | No second effect |
The relay publishes at least once, and the consumer filters duplicates. Together they give an effect that happens once, with no distributed transaction. The cost is a polling or log-tailing component, and a delay of a second or so between the commit and the message.
Retries need a limit and a shape
At-least-once delivery depends on the sender retrying, and retries that are too eager make an outage worse. Use exponential backoff with jitter, so thousands of clients do not retry in step, and a cap on attempts after which the message goes to a dead-letter queue for a person to look at. A delay of 100 ms doubling to a cap of 30 seconds, with up to half of each delay randomised, is a sound default. The idempotency check makes the retries safe, and the cap makes them bounded.
Where the guarantee breaks anyway
Idempotency covers effects inside systems that you control. A call to an email provider or a payment gateway repeated after a timeout can send two emails or charge twice, unless the remote system supports an idempotency key. Send the message id as that key, and treat a remote system without one as the weak point to design around, with a lookup of the previous attempt before retrying.
When this does not apply
In-process queues in one JVM with a shared transaction can offer exactly-once behaviour, since there is no unreliable link. References: the Two Generals problem overview and Confluent's exactly-once semantics explanation.


