A practical guide to publish-subscribe messaging, covering topics, fan-out, delivery guarantees, durable subscriptions, and the real systems that power it.
Publish/subscribe, usually shortened to pub/sub, is one of those patterns that quietly runs under most of the systems you use every day. When you get a push notification, when an analytics dashboard updates, when a payment triggers a receipt email and a fraud check and a ledger write all at once, there is a decent chance a pub/sub broker sat in the middle. It is worth understanding well, because it changes how you think about wiring services together.
The core idea is decoupling. A publisher emits an event to a named destination (a topic or channel) and then forgets about it. It does not know who is listening, how many listeners exist, or whether anyone is listening at all. On the other side, subscribers register interest in a topic and receive matching events. They do not know who produced the event or how many other subscribers exist.
Sitting between them is the broker. The broker owns the topics, tracks subscriptions, and routes each published message to every interested subscriber. Because neither side holds a reference to the other, you can add, remove, or restart producers and consumers independently. That independence is the whole point.
Contrast this with a direct call. If service A calls service B over HTTP, A needs B's address, B needs to be up, and A blocks or fails when B is slow. With pub/sub, A publishes and moves on. B processes when it can.
People conflate pub/sub with message queues, but the routing model differs in one crucial way.
A point-to-point queue delivers each message to exactly one consumer. If ten workers pull from the same queue, a given job goes to one of them. This is competing-consumer work distribution, ideal for spreading load.
Pub/sub does fan-out: each message goes to every subscriber. Ten subscribers means ten copies delivered.
Point-to-point queue Publish/subscribe
-------------------- -----------------
+--> Sub A (email)
Producer --> [ queue ] --> Worker |
(1 of N gets Publisher --> (topic) --+--> Sub B (analytics)
each message) (fan-out to |
all subscribers) +--> Sub C (audit log)
In practice systems blend both. A topic fans out to several subscriptions, and each subscription is itself a queue with competing consumers behind it. AWS models this explicitly with SNS (the topic, fan-out) feeding several SQS queues (each a load-balanced work pool). For a fuller tour of the queue side, see message queue patterns.
A topic (or channel, or subject) is the named routing key that publishers write to. A subscription is a subscriber's standing registration against a topic. The distinction matters because two subscriptions on the same topic each get their own independent copy of the stream, with their own acknowledgment state and backlog.
Delivery happens in one of two styles:
Push: the broker calls the subscriber, POSTing the message to an endpoint or invoking a handler. Low latency, but the subscriber must keep up or the broker throttles.
Pull: the subscriber asks the broker for messages when it is ready, then acknowledges them. This gives the consumer control over pacing and makes backpressure natural, at the cost of a polling loop.
Kafka is pull-based by design; consumers track their own offset. Google Pub/Sub offers both. MQTT and Redis lean toward push.
This is where good intentions meet distributed reality. Most brokers offer at-least-once delivery: a message is delivered one or more times, and duplicates are possible when an acknowledgment is lost and the broker retries. Your handlers should therefore be idempotent. Exactly-once exists in specific configurations (Kafka transactions, Pub/Sub's exactly-once subscriptions) but it constrains throughput and only holds within the broker's boundary, not across your side effects.
Ordering is the other trap. A global order across a whole topic is expensive and usually not offered. What you typically get is per-partition or per-key ordering. Kafka keeps order within a partition; Google Pub/Sub orders messages that share an ordering key. If your logic depends on sequence, route related events to the same key so they land in order, and design everything else to tolerate reordering.
A durable subscription survives the subscriber going offline. The broker retains messages (up to a retention window) and delivers the backlog when the subscriber returns. Kafka topics, Google Pub/Sub subscriptions, and NATS JetStream all persist this way.
An ephemeral subscription exists only while the subscriber is connected. Classic Redis pub/sub works like this: if nobody is subscribed the moment a message publishes, it is gone. No storage, no replay. That makes Redis pub/sub blazingly fast and perfectly fine for live presence indicators or cache invalidation, and completely wrong for anything you cannot afford to lose. Plain MQTT at QoS 0 has the same characteristic.
Google Pub/Sub: managed, durable, push or pull, ordering keys, strong for cloud-native event backbones.
Kafka topics: a partitioned, replayable log rather than a classic broker; consumers read by offset and can rewind. Best when you want durable history and high throughput.
Redis pub/sub: in-memory, ephemeral, sub-millisecond, no persistence. Redis Streams adds durability when you need it.
NATS: lightweight and fast, ephemeral by default with JetStream adding durability and replay.
MQTT: built for IoT and constrained networks, with QoS levels trading reliability for overhead.
SNS + SQS: the AWS fan-out pattern, where one topic feeds many durable queues.
The upside is real. Loose coupling lets teams ship producers and consumers independently. Scalability comes from fan-out plus competing consumers behind each subscription. Extensibility is the quiet win: adding a new consumer to an existing event stream needs zero changes to the publisher, so you bolt on a new audit log or search indexer without touching the code that emits the event.
The pitfalls are equally real. Pub/sub buys you eventual consistency, so a read right after a write may not reflect it yet, and your product and UI need to account for that. Debugging gets harder because there is no single call stack; a message crosses several independent consumers, and you need correlation IDs and good tracing to follow it. And message loss with ephemeral subscriptions bites teams who assumed the broker was storing things it never was. Always confirm the durability model before you rely on delivery.
Reach for pub/sub when one event needs to trigger several unrelated reactions, when producers and consumers scale or deploy independently, or when you want to add future consumers without disturbing today's code. It shines in event-driven backends, notification systems, and data pipelines. This pattern is a building block of event-driven architecture, so if you are already heading that direction, pub/sub is likely part of the answer.
Skip it when you need an immediate, synchronous answer, when strict global ordering is non-negotiable, or when a simple direct call or a single work queue does the job. Pub/sub adds a broker to operate and a mental model to maintain, and that cost is not always worth paying.
Default to a durable broker unless you have a specific reason not to. The convenience of ephemeral Redis pub/sub is tempting, but the day you lose a message you cannot explain is the day you wish you had picked persistence. Start with durable subscriptions, make every handler idempotent, attach an ordering key where sequence matters, and thread a correlation ID through every message from day one. Do that and pub/sub becomes the flexible, forgiving backbone it is supposed to be rather than a source of mysteries.
