A practical tour of the message queue patterns that keep distributed backends decoupled, resilient, and able to survive traffic spikes.
Message queues are the connective tissue of most non-trivial backends. Once you split a monolith into services, or you need to absorb bursty load without falling over, a queue sits in the middle and buys you time, isolation, and the freedom to scale producers and consumers separately. The patterns below show up over and over, and knowing them by name saves a lot of whiteboard arguments. They are the building blocks of any serious event-driven architecture.
The work queue is the workhorse. A producer drops tasks onto a queue and a pool of workers pulls from it, each message going to exactly one worker. Add workers and throughput goes up; remove them and the queue simply grows until capacity returns.
Problem it solves: distributing expensive work (image resizing, PDF generation, sending email) so a slow task never blocks the request path.
When to use it: any time work can happen after the response is returned, and you want horizontal scaling for free.
The one gotcha is fairness. Prefetch limits matter, otherwise one worker grabs a hundred messages while others sit idle.
Where a work queue delivers each message once, pub/sub delivers each message to every interested subscriber. One "order placed" event feeds billing, search indexing, and the email service without the producer knowing any of them exist.
Problem it solves: decoupling one event from the many independent reactions it triggers.
When to use it: when adding a new consumer should not require touching the producer. If you want the full mental model, see pub/sub explained.
Sometimes you need an answer back. You send a message, include a correlation ID and the name of a reply queue, then wait for a response tagged with that same ID.
correlation_id = uuid4().hex
channel.basic_publish(
exchange="",
routing_key="rpc_requests",
properties=BasicProperties(
reply_to="rpc_replies",
correlation_id=correlation_id,
),
body=json.dumps({"op": "price", "sku": "A-100"}),
)
# consumer side matches the id back to the waiting caller
def on_reply(ch, method, props, body):
if props.correlation_id == correlation_id:
futures[correlation_id].set_result(json.loads(body))
Problem it solves: synchronous-feeling calls across an async transport, with loose coupling and a buffer in the middle.
When to use it: rarely. It helps when you already run a broker and want one transport, but plain HTTP is usually simpler for true request/response.
A routing key lets subscribers filter. Bind a queue to orders.eu.* and you only get European order events; bind another to orders.# and you get everything under orders. The broker does the filtering so consumers stay lean.
Problem it solves: slicing a firehose into meaningful streams without every consumer reading every message.
When to use it: when subscribers care about a subset of a busy topic, split by region, tenant, or event type.
Not all work is equal. A priority queue lets urgent messages jump ahead of the backlog, so a password-reset email is not stuck behind ten thousand marketing sends.
Problem it solves: latency guarantees for a small class of important messages sharing infrastructure with bulk work.
When to use it: sparingly. Priorities help under moderate load, but under sustained overload low-priority messages can starve forever. Often two separate queues with dedicated workers are clearer than one queue with flags.
You want an action to happen later: retry in 30 seconds, send a reminder in 24 hours, cancel an unpaid order after 15 minutes. A delayed message stays invisible until its scheduled time.
Problem it solves: time-based workflows without a cron job polling a database.
When to use it: reminders, timeouts, and the backoff delays described below. Some brokers support this natively; others fake it with a per-message TTL that routes to the real queue on expiry.
Some messages can never succeed. A malformed payload or a permanently missing record will fail every retry and, left alone, block the queue behind it. A dead-letter queue (DLQ) is the holding pen where these poison messages land after exhausting their attempts.
Problem it solves: getting stuck messages out of the way while preserving them for inspection instead of silently dropping them.
When to use it: always. A DLQ with an alert on its depth is one of the highest-leverage things you can add. We cover the operational details in dead letter queues.
Transient failures, a database blip or a rate limit, deserve another try, but retrying instantly in a tight loop just amplifies the problem. Exponential backoff spaces attempts out, and jitter stops a thundering herd of clients all retrying in lockstep.
def next_delay(attempt, base=1, cap=60):
exp = min(cap, base * (2 ** attempt))
return random.uniform(0, exp) # full jitter
if attempt < MAX_RETRIES:
schedule_redelivery(msg, delay=next_delay(attempt))
else:
send_to_dlq(msg)
The retry count travels with the message in a header. Once it crosses the ceiling, the message goes to the DLQ instead of looping forever.
Here is the reality most brokers hand you: at-least-once delivery. A message can arrive twice if an ack is lost or a worker crashes mid-processing. Exactly-once delivery is mostly a marketing phrase; what you can actually build is exactly-once effects by making your consumers idempotent.
Problem it solves: duplicate deliveries that would otherwise double-charge a card or send two emails.
When to use it: everywhere at-least-once delivery is in play, which is nearly always. The usual approach is a dedup key. Record each processed message ID, and skip anything you have seen.
def handle(msg):
if seen.add_if_absent(msg.id): # atomic insert, returns False if exists
process(msg)
seen.confirm(msg.id)
ack(msg)
Producers can outrun consumers. Without a limit, an unbounded queue swallows memory until the broker dies and takes everything with it. Backpressure pushes back: bounded queue depth, prefetch limits, or slowing producers when the backlog grows.
Problem it solves: protecting the system from overload instead of pretending capacity is infinite.
When to use it: design it in from the start. Decide early what happens when the queue fills, whether you block producers, shed load, or reject with a clear error.
Start every service with a work queue, a DLQ, and idempotent consumers. That trio handles most real traffic and failure modes with very little code. Reach for pub/sub the moment a second consumer wants the same event, add routing keys when one topic gets busy, and layer retries with jittered backoff on top. Treat priority and delayed messages as targeted tools, not defaults, and never ship a queue without an alert on its depth. Assume at-least-once, build for duplicates, and cap your queues. Do that, and the queue becomes the calm buffer it should be rather than the thing that pages you at 3 a.m.
