Every few months someone on the team asks whether the queue can guarantee exactly-once delivery. The honest answer is no, and chasing that guarantee usually produces a system that is slower, more fragile, and still occasionally double-processes. The useful move is to understand why the guarantee cannot exist over a network, then design for the thing you can actually build.

The three delivery semantics #

Messaging systems offer one of three contracts between producer and consumer.

At-most-once: the message is delivered zero or one times. The producer sends and forgets, or the broker drops on failure rather than retrying. You never see duplicates, but you can silently lose messages. Fine for metrics samples or fire-and-forget telemetry, wrong for anything involving money or state changes.

At-least-once: the message is delivered one or more times. The system retries until it gets an acknowledgment, so nothing is lost, but a lost or delayed ack means the same message arrives again. This is the default and the workhorse of nearly every serious queue.

Exactly-once: the message is processed once and only once, no loss and no duplicates. This is what everyone wants and what marketing pages love to promise.

Why true exactly-once delivery is impossible #

The blocker is the two generals problem. Imagine a producer sending a message and waiting for an acknowledgment. The message arrives, the consumer processes it, and the ack travels back. If that ack is lost in the network, the producer has no way to distinguish "consumer never got it" from "consumer got it but the ack vanished." Its only safe options are to retry, which risks a duplicate, or to give up, which risks a loss.

No amount of extra handshaking fixes this. Every acknowledgment needs its own acknowledgment, and you get an infinite regress. Over an unreliable channel, no protocol can guarantee that both sides agree a message was delivered exactly once. This is not an engineering gap that a better broker will close. It is a proven property of distributed communication.

So any real system faces a choice: retry and risk duplicates, or do not retry and risk loss. Since silent loss is usually worse, mature systems pick at-least-once and then deal with the duplicates.

What exactly-once really means in practice #

When a vendor says exactly-once, they mean exactly-once processing, not delivery. The message may physically arrive several times, but the observable effect on your system happens once. Duplicates still hit the wire; the consumer recognizes and discards them. The guarantee lives at the processing boundary, not the transport.

Reframed that way, the problem becomes tractable. You cannot stop duplicates from arriving, but you can make a second arrival a no-op. That is idempotency, and it is the real answer.

Idempotent consumers #

An operation is idempotent when applying it twice has the same effect as applying it once. Some operations are naturally idempotent: setting a user's status to active, or overwriting a record with an absolute value. Others are not: incrementing a balance, appending to a ledger, sending an email.

For the non-idempotent cases, you attach an idempotency key to each message. The producer generates a stable unique ID (an order ID, a UUID chosen at creation and preserved across retries) and the consumer records which keys it has already handled in a dedup store. Before acting, it checks the store. Seen the key, skip the work. New key, do the work and record the key.

The subtlety is atomicity. Recording the key and performing the side effect must commit together, or you reopen the same gap you were closing. If you process first and record second, a crash between them leaves the key unrecorded and the next delivery reprocesses. The clean version keeps the dedup record and the business write in one transaction.

sql.sql

CREATE TABLE processed_messages (
    idempotency_key TEXT PRIMARY KEY,
    processed_at    TIMESTAMPTZ NOT NULL DEFAULT now()
);

python.python

def handle_payment(conn, msg):
    key = msg["idempotency_key"]
    with conn:  # single transaction: dedup check + side effect commit together
        cur = conn.cursor()
        cur.execute(
            "INSERT INTO processed_messages (idempotency_key) "
            "VALUES (%s) ON CONFLICT DO NOTHING",
            (key,),
        )
        if cur.rowcount == 0:
            return  # duplicate: key already present, do nothing

        cur.execute(
            "UPDATE accounts SET balance = balance - %s WHERE id = %s",
            (msg["amount"], msg["account_id"]),
        )
    ack(msg)

The ON CONFLICT DO NOTHING plus rowcount check makes the insert the gate. If the key is already there, rowcount is zero and the balance update never runs. Because both statements share one transaction, a crash rolls back the whole thing and redelivery retries cleanly. The processed-id table is the dedup store; the primary key does the deduplication for you.

The transactional outbox pattern #

Idempotency handles the consumer side. The producer side has a matching hazard: you update your database and then publish an event, and a crash between the two steps loses the event or, if reordered, publishes without persisting. The outbox pattern collapses those into one atomic write. You insert the event into an outbox table in the same transaction as your business update, then a separate relay process reads the outbox and publishes to the broker, marking rows as sent. The relay retries freely, which means the broker gets at-least-once, which is exactly why the consumer needs to be idempotent. The two patterns are complementary halves of the same design.

Kafka's exactly-once semantics #

Kafka is the system most often cited as delivering exactly-once, and it genuinely does within its own boundary. Two mechanisms combine. The idempotent producer tags each batch with a producer ID and sequence number so the broker discards duplicate writes from retries. Transactions let you write to multiple partitions and commit consumer offsets atomically, so a consume-transform-produce loop either fully commits or fully aborts.

The limit is the phrase "within Kafka's boundary." The guarantee holds when your inputs and outputs are both Kafka topics and your processing state lives in Kafka. The moment your consumer writes to an external database, calls a payment API, or sends an email, you have stepped outside the transactional fence, and Kafka can no longer make it atomic. For those side effects you are back to idempotency keys and a dedup store. Kafka exactly-once is real and useful for stream processing pipelines; it is not a blanket exemption from thinking about duplicates.

The call we'd make #

Design for at-least-once and make every consumer idempotent. It is the one combination that loses nothing and tolerates the duplicates the network will inevitably produce. Assign a stable idempotency key at the edge, keep a processed-id table, and commit the dedup record in the same transaction as the side effect. Add the outbox pattern on the producer side so publishes never diverge from your database. Reach for Kafka transactions only when the whole pipeline stays inside Kafka, and treat any external call as a place where you own the dedup. For the wider context on how these events flow between services, see our guide to event-driven architecture. Stop hunting for a delivery guarantee that cannot exist and build the processing guarantee that can.

The three delivery semantics#

Why true exactly-once delivery is impossible#

What exactly-once really means in practice#

Idempotent consumers#

The transactional outbox pattern#

Kafka's exactly-once semantics#

#Idempotency

Exactly-Once Delivery: Myth, Reality, and What to Do Instead