Skip to main content
Message Queue Patterns Every Backend Engineer Should Know

Message Queue Patterns Every Backend Engineer Should Know

KU
· Founder & Chief Editor
• 8 min read•Updated •23 views

A practical tour of the message queue patterns that keep distributed backends decoupled, resilient, and able to survive traffic spikes.

Key takeaways

  • Most backends need only three message queue patterns by default, a work queue with competing consumers, a dead-letter queue, and idempotent consumers.
  • Add pub/sub when a second service wants the same event, routing keys when a topic gets busy, and jittered retries for transient failures.
  • Assume at-least-once delivery and cap queue depth.

The message queue patterns that matter most are work queues (competing consumers), publish/subscribe fan-out, dead-letter queues, retries with jittered backoff, idempotent consumers and backpressure. Start every service with a work queue, a DLQ and idempotent consumers; add pub/sub, routing, priorities and delays only when a specific need shows up.

Verified against

RabbitMQ 4.4 (4.4.0, released 2026-10-20), against the rabbitmq.com docs for dead lettering, priority queues, consumer prefetch and quorum queues, plus the official RabbitMQ tutorials. Checked October 2026.

Once you split a monolith into services or need to absorb bursty load, a queue sits in the middle and lets producers and consumers scale separately. These patterns are the building blocks of any serious event-driven architecture. We use RabbitMQ terms below because they map cleanly to the patterns; Kafka vs RabbitMQ covers when a log beats a broker.

Work queues: competing consumers

The work queue is the workhorse. A producer drops tasks onto a queue and a pool of workers pulls from it, each message going to exactly one worker. Add workers and throughput rises; remove them and the queue grows until capacity returns.

Problem it solves: distributing expensive work (image resizing, PDF generation, sending email) so a slow task never blocks the request path.

When to use it: any time work can happen after the response is returned, and you want horizontal scaling for free.

The one gotcha is fairness. Prefetch limits matter, otherwise one worker grabs a hundred messages while others sit idle. In RabbitMQ that is basic.qos with a small prefetch count, so each consumer holds only a few unacknowledged messages at a time.

Publish/subscribe: fan-out

Where a work queue delivers each message once, pub/sub delivers each message to every interested subscriber. One "order placed" event feeds billing, search indexing, and the email service without the producer knowing any of them exist.

Problem it solves: decoupling one event from the many independent reactions it triggers.

When to use it: when adding a new consumer should not require touching the producer. If you want the full mental model, see pub/sub explained.

Request/reply over messaging

Sometimes you need an answer back. You send a message, include a correlation ID and the name of a reply queue, then wait for a response tagged with that same ID.

Python
correlation_id = uuid4().hex
channel.basic_publish(
    exchange="",
    routing_key="rpc_requests",
    properties=BasicProperties(
        reply_to="rpc_replies",
        correlation_id=correlation_id,
    ),
    body=json.dumps({"op": "price", "sku": "A-100"}),
)

# consumer side matches the id back to the waiting caller
def on_reply(ch, method, props, body):
    if props.correlation_id == correlation_id:
        futures[correlation_id].set_result(json.loads(body))

Problem it solves: synchronous-feeling calls across an async transport, with loose coupling and a buffer in the middle.

When to use it: rarely. Plain HTTP is usually simpler for true request/response.

Routing and topic exchanges

A routing key lets subscribers filter. Bind a queue to orders.eu.* and you only get European order events; bind another to orders.# and you get everything under orders. The broker does the filtering so consumers stay lean.

Problem it solves: slicing a firehose into meaningful streams without every consumer reading every message.

When to use it: when subscribers care about a subset of a busy topic, split by region, tenant, or event type.

Priority queues

Not all work is equal. A priority queue lets urgent messages jump ahead of the backlog, so a password-reset email is not stuck behind ten thousand marketing sends.

Problem it solves: latency guarantees for a small class of important messages sharing infrastructure with bulk work.

When to use it: sparingly. Priorities help under moderate load, but under sustained overload low-priority messages can starve forever. RabbitMQ accepts x-max-priority values from 1 to 255, yet its docs strongly recommend using only 2 to 4 priorities, because every level costs CPU and memory. Often two separate queues with dedicated workers are clearer than one queue with flags.

Delayed and scheduled messages

You want an action to happen later: retry in 30 seconds, send a reminder in 24 hours, cancel an unpaid order after 15 minutes. A delayed message stays invisible until its scheduled time.

Problem it solves: time-based workflows without a cron job polling a database.

When to use it: reminders, timeouts, and the backoff delays described below. Some brokers support this natively; others fake it with a per-message TTL that routes to the real queue on expiry. RabbitMQ's dead-letter docs list TTL expiry as one of the events that dead-letters a message, which is exactly what that trick relies on.

Dead-letter queues for poison messages

Some messages can never succeed. A malformed payload or a permanently missing record will fail every retry and, left alone, block the queue behind it. A dead-letter queue (DLQ) is the holding pen where these poison messages land after exhausting their attempts.

Problem it solves: getting stuck messages out of the way while preserving them for inspection instead of silently dropping them.

When to use it: always. A DLQ with an alert on its depth is one of the cheapest, most valuable things you can add. In RabbitMQ a message is dead-lettered when a consumer rejects it with requeue=false, its TTL expires, the queue exceeds a length limit, or a quorum queue's delivery limit is hit. Since RabbitMQ 4.0, quorum queues enforce a default delivery limit of 20, so a poison message can no longer loop forever by default. We cover the operational details in dead letter queues.

Python
# declare_queues.py: route rejected or expired messages to a DLQ
channel.exchange_declare(exchange="orders.dlx", exchange_type="fanout")
channel.queue_declare(queue="orders.dlq", durable=True)
channel.queue_bind(queue="orders.dlq", exchange="orders.dlx")

channel.queue_declare(
    queue="orders",
    durable=True,
    arguments={
        "x-queue-type": "quorum",
        "x-dead-letter-exchange": "orders.dlx",
        "x-delivery-limit": 5,  # tighter than the default of 20
    },
)

Retry with backoff

Transient failures, a database blip or a rate limit, deserve another try, but retrying instantly in a tight loop just amplifies the problem. Exponential backoff spaces attempts out, and jitter stops a thundering herd of clients all retrying in lockstep.

Python
def next_delay(attempt, base=1, cap=60):
    exp = min(cap, base * (2 ** attempt))
    return random.uniform(0, exp)  # full jitter

if attempt < MAX_RETRIES:
    schedule_redelivery(msg, delay=next_delay(attempt))
else:
    send_to_dlq(msg)

The retry count travels with the message in a header. Once it crosses the ceiling, the message goes to the DLQ instead of looping forever.

Idempotent consumers

Most brokers hand you at-least-once delivery. A message can arrive twice if an ack is lost or a worker crashes mid-processing. Exactly-once delivery is mostly a marketing phrase; what you can actually build is exactly-once effects by making your consumers idempotent.

Problem it solves: duplicate deliveries that would otherwise double-charge a card or send two emails.

When to use it: everywhere at-least-once delivery is in play, which is nearly always. The usual approach is a dedup key. Record each processed message ID, and skip anything you have seen.

Python
def handle(msg):
    if seen.add_if_absent(msg.id):   # atomic insert, returns False if exists
        process(msg)
        seen.confirm(msg.id)
    ack(msg)

Backpressure

Producers can outrun consumers. Without a limit, an unbounded queue swallows memory until the broker dies and takes everything with it. Backpressure pushes back: bounded queue depth, prefetch limits, or slowing producers when the backlog grows.

Problem it solves: protecting the system from overload instead of pretending capacity is infinite.

When to use it: design it in from the start. Decide early what happens when the queue fills, whether you block producers, shed load, or reject with a clear error. In Kafka, the equivalent lever is consumer parallelism, which is bounded by partition count; Kafka partition strategies covers how to size it.

Questions people ask

What is the difference between a work queue and pub/sub?

A work queue delivers each message to exactly one consumer from a pool, so adding workers spreads the load. Pub/sub delivers a copy of each message to every subscriber, so one event can drive billing, search indexing and email independently. Use a work queue to distribute jobs and pub/sub to broadcast facts. Many systems combine them: fan out to one queue per service, with competing workers on each.

Do I need exactly-once delivery?

You need exactly-once effects, which you get from idempotent consumers, not from the broker. Assume at-least-once delivery: a lost ack or a crashed worker redelivers the message. Record a dedup key, such as the message ID or a business key, in the same transaction as the side effect, and skip anything already processed. That handles duplicates from retries, redeliveries and producer resends alike.

When should a message go to the dead-letter queue?

When retrying cannot succeed: a malformed payload, a schema mismatch, or a record that no longer exists. Transient errors like timeouts and rate limits deserve retries with backoff first. Cap attempts with a retry counter or a broker delivery limit (RabbitMQ quorum queues default to 20 since 4.0), then dead-letter the message and alert on DLQ depth so a human looks at it.

The call we'd make

Start every service with a work queue, a DLQ, and idempotent consumers. That trio handles most real traffic and failure modes with very little code. Reach for pub/sub the moment a second consumer wants the same event, add routing keys when one topic gets busy, and layer retries with jittered backoff on top. Treat priority and delayed messages as targeted tools, not defaults, and never ship a queue without an alert on its depth. Assume at-least-once, build for duplicates, and cap your queues.

Sources

Explore topics:DevOps
React
Share this post
Cite this article
Plain text

Kiril Urbonas (2026). Message Queue Patterns Every Backend Engineer Should Know. DevOpsNess. Updated 2026-10-31. https://www.devopsness.com/blog/message-queue-patterns

Markdown

[Message Queue Patterns Every Backend Engineer Should Know](https://www.devopsness.com/blog/message-queue-patterns) (Kiril Urbonas, DevOpsNess, updated 2026-10-31)

KU

Kiril Urbonas

Founder & Chief Editor

605 articles
View all articles by Kiril Urbonas

You might have missed

Evergreen posts worth revisiting.