Kafka Partition Strategies — Scaling Consumers Without Reshuffling Everything

Kafka Partition Strategies — Scaling Consumers Without Reshuffling Everything#

Kafka partition counts and keying decisions look like throwaway config — until you hit a hot partition, or try to add consumers and find out partitions limit your parallelism, or change the partition count and discover all your downstream ordering assumptions broke. The patterns below are what survived two years of running Kafka behind ~40 producer-consumer pairs.

The two things partitions actually control#

Strip away the marketing: partitions decide two things and only two:

1. Maximum consumer parallelism within a consumer group. A topic with 12 partitions can be processed by at most 12 consumers in the same group. Add a 13th consumer and it sits idle.
2. Ordering guarantees. Within a partition, Kafka guarantees order. Across partitions, no guarantee — messages with key A and key B can arrive at consumers in either relative order.

Every other consideration (throughput, durability, retention) flows from those two.

Sizing rule that actually works#

The advice "pick lots of partitions" is half right and half misleading. The actual rule we use:

partitions = max( peak_target_throughput_MB_per_sec / per_partition_throughput_MB_per_sec,
                  expected_max_consumers )

Per-partition throughput on modest hardware is ~5–10 MB/s for sustained writes. So if you expect peak 50 MB/s and might want 8 parallel consumers eventually, pick max(10, 8) = 10.

Common mistakes:

• Way too few. A topic with 3 partitions caps consumer parallelism at 3 forever. Hard to fix later without downtime.
• Way too many. 1000 partitions on a topic with 100 KB/s of traffic. Each partition has memory + file overhead on brokers; thousands of tiny partitions slow cluster recovery and bloat ZooKeeper/KRaft metadata.

We aim for "enough headroom for 2–3× current consumer count, sized for peak throughput, rounded to a small number you'll remember." Typically 6, 12, 24, or 48 partitions per topic — not 7 or 19.

Key choice = ordering policy#

The partition key determines which partition a message lands on. partition = hash(key) mod num_partitions. So:

• Same key → same partition → ordered relative to each other.
• Different keys → possibly different partitions → no order guarantee between them.

This is the most under-thought decision in Kafka design. What do you actually need ordered?

For an order-processing service: probably "events for the same order_id" must arrive in order. Key = order_id.

For a user-event tracker: probably "events for the same user_id" — key = user_id.

For application logs: usually no ordering needed across messages. Key = null → Kafka round-robins. Maximum parallelism.

The biggest mistake here: keying on something coarse (like "tenant_id" for a multi-tenant SaaS) when many tenants are small but one is huge. All the big tenant's traffic lands on one partition. Hot partition.

The hot partition problem#

You key by tenant_id. Tenant 42 produces 80% of the traffic. The partition holding tenant 42's messages is the bottleneck; the other partitions are mostly idle. You scaled to 12 partitions hoping for 12× throughput; you got ~1.2×.

Three real fixes:

1. Add randomness to the key. Key = tenant_id + ":" + random_bucket(0..N-1). Spreads tenant 42 across N partitions. Trade: messages for the same tenant aren't strictly ordered anymore, which may or may not matter.

2. Custom partitioner. Detect hot tenants and route them across more partitions. Most Kafka clients support a custom partitioner. More code; more control.

3. Per-tenant topics for huge tenants. A separate dedicated topic for tenant 42; consumers handle it specially. Hard to scale to many huge tenants; works for the "a few whales" pattern.

We've used (1) most often. It's a clear concession in ordering guarantees in exchange for actual parallelism.

Changing partition counts is painful#

Adding partitions to a topic is technically easy (kafka-topics --alter). Doing it without breaking things isn't.

When you add partitions, the partition assignment changes for new messages. If your consumers cache state keyed by partition (e.g., "consumer 5 is responsible for partition 5's state"), that state is stale. Worse, if you were keying by user_id and getting consistent partition assignment, the new partition count means user_id now maps to a different partition — so events for the same user could arrive at a different consumer than before.

What we do:

• Pick a partition count up front and stick with it. This is the cheapest answer.
• If we have to add partitions: plan a deployment that resets stateful consumers cleanly. Stop consumers, drain, change partition count, restart consumers, accept some reshuffle pain.
• Never add partitions on a live topic where consumers maintain per-partition state without coordinating.

The "just add more partitions later" mindset works for simple stateless consumers. Anything stateful, you pay later.

Consumer group sizing#

A consumer group should have:

• One consumer per partition for max parallelism (12 partitions, 12 consumers).
• Fewer if your consumers are fast enough to handle multiple partitions each.
• More than partitions = wasted instances. They sit idle.

For autoscaling consumer pools: scale based on consumer lag, not CPU. CPU on a consumer that's keeping up looks fine; consumer lag tells you the actual story.

We use KEDA (Kubernetes Event-Driven Autoscaling) with the Kafka lag scaler. Target lag of ~1000 messages per partition; scale up if exceeded for >2 min, scale down if under for >10 min. Works well.

Rebalance pain#

When a consumer joins or leaves the group, Kafka rebalances: every partition is reassigned. During rebalance:

• All consumers stop processing.
• The coordinator computes new assignment.
• Consumers receive their new partitions.

This pause can be milliseconds (small groups, small state) or seconds (large groups, large state). For latency-sensitive consumers, rebalance is the worst part of running Kafka.

Mitigations:

• session.timeout.ms: how long a consumer can be away before it's considered dead. Bigger = fewer rebalances on transient issues, longer detection of real failures. We use 30 seconds.
• max.poll.interval.ms: how long between polls. If a consumer takes too long processing one message, it misses the deadline and triggers a rebalance. Tune based on your actual processing time + safety margin.
• Cooperative rebalancing (vs eager). Cooperative only reassigns affected partitions instead of all of them; less disruption. Default in newer clients.

The "consumer keeps getting kicked out of the group" problem is almost always a max.poll.interval.ms issue — processing took longer than expected, the broker thought the consumer died, triggered a rebalance.

Specific patterns we run#

• Idempotent producers (enable.idempotence=true). Prevents duplicate messages on producer retry. Default in modern clients; verify.
• acks=all on producers. Wait for all in-sync replicas to confirm. Durability over latency for anything important.
• Manual commit on consumers (enable.auto.commit=false). Commit after successful processing, not on poll. Prevents data loss on consumer crash.
• Dead letter topic for messages that fail processing. After N retries, ship to a separate topic for human triage.
• Compaction for state-like topics (latest value per key wins; old values garbage-collected). Useful for change data capture, config distribution.

What we monitor#

• Consumer lag per partition. The single most important metric. Lag growing = consumers can't keep up. Per-partition lag surfaces hot partitions.
• Under-replicated partitions at the broker level. Indicates broker health issues.
• Rebalance rate. Healthy is "near zero." Frequent rebalances mean something's wrong.
• Message size distribution. Sudden growth in message sizes can quietly tip throughput over the partition's capacity.

What to read next#

• Job queues — Sidekiq, Celery, BullMQ patterns — the adjacent pattern when Kafka is overkill
• Pipeline observability — why CI failures don't trigger alerts — same observability discipline applied to CI
• Multi-provider LLM routing — failover, cost, load balancing — partition-like fan-out applied to LLM APIs

Kafka partitions are simple in concept and tricky in operation. The bulk of issues we've debugged trace back to one of three decisions: wrong partition count, wrong key, or unanticipated hot tenant. Make those right at design time and Kafka is one of the most predictable parts of the stack.