# Job Queues: Sidekiq, Celery, BullMQ Patterns That Hold Up

URL: https://www.devopsness.com/blog/job-queues-sidekiq-celery-bullmq-patterns
Author: Kiril Urbonas
Published: 2026-05-10
Updated: 2026-10-31
Categories: DevOps
Tags: Job Queues, Sidekiq, Celery, Bullmq, Background Jobs

Summary: Sidekiq, Celery and BullMQ look different but fail the same way. The patterns that work across all three, the defaults that bite, and the operational gotchas.

---
# Job Queues: Sidekiq, Celery, BullMQ Patterns That Hold Up

Pick the queue that matches your language: Sidekiq for Ruby, Celery for Python, BullMQ for Node. Then apply the same rules in all three: idempotent jobs, IDs instead of objects, bounded retries with exponential backoff and jitter, no retries on permanent errors, a reviewed dead set, and alerts on queue depth. The defaults differ, and each one bites differently.

> [!VERIFIED]
> Sidekiq 8.1.7, Celery 5.6.3, BullMQ 6.5.3, against the Sidekiq error-handling wiki, the Celery tasks user guide and the BullMQ retry docs. Checked October 2026.

## Why background jobs at all

Three needs recur. **Latency**: work the user shouldn't wait for, like a welcome email or a thumbnail. **Reliability**: work that must retry, like calling a flaky third-party API or processing a webhook. **Bursty load**: work that arrives in waves, which a queue smooths so workers process at their own pace.

All three tools share the same shape: a backend (Redis for Sidekiq and BullMQ; RabbitMQ or Redis for Celery), workers that pull and execute jobs, JSON-serialized job name plus arguments, retries with backoff, a dead or failed set, delayed jobs, and a web UI (Sidekiq Web, Flower, Bull Board). Switching languages means relearning syntax, not architecture. Cross-language workflows belong on a message bus instead; the trade-offs are in [Kafka vs RabbitMQ](/blog/kafka-vs-rabbitmq).

## The defaults that differ

| | Sidekiq 8.1 | Celery 5.6 | BullMQ 6.5 |
|---|---|---|---|
| Default retries | 25, about 20 days total | `max_retries=3`, only when you call `retry()` or set `autoretry_for` | None unless `attempts` > 1 |
| Default backoff | Polynomial, `retry_count ** 4 + 15` plus jitter | 180 s fixed `default_retry_delay` | None (immediate) unless `backoff` is set |
| Stop retrying | `retry: 0` or `retry: false` | Re-raise without `retry()` | Throw `UnrecoverableError` |
| Exhausted jobs | Dead set, capped at 10,000 jobs or 6 months | Task marked failed; no built-in DLQ | Failed set |
| Crash mid-job | OSS: job lost; Pro `super_fetch` recovers it | Acked before execution unless `acks_late=True` | Lock renewed while running; stalled jobs re-queued |

Those defaults are the source of most surprises. Open-source Sidekiq removes a job from Redis when it fetches it, so a hard crash mid-job loses it. Sidekiq's 25 retries over about 20 days mean a broken job can keep firing for weeks. Celery's early acknowledgement means a worker killed mid-task loses that task unless you opt into `acks_late`, and then the docs say plainly: "Make sure your tasks are idempotent." BullMQ does nothing on failure until you configure `attempts`.

## The patterns that earn their place

**Idempotency.** Every queue here is at-least-once in practice: retries, redeliveries after a crash and duplicate enqueues all happen. Either make the work naturally idempotent (`UPDATE ... WHERE state = 'pending'`) or pass an idempotency key and check it at the destination.

**Small, focused jobs.** "Send email, update database, call API" in one job means a retry of step three repeats steps one and two. Split them and chain.

**Pass IDs, not objects.** A job takes `user_id`, not a serialized user. A retry five minutes later should read current state, not a stale snapshot.

**Bounded retries, classified failures.** Transient errors (timeouts, 429s, 503s) get exponential backoff with jitter; permanent errors (validation failures, 404s) go straight to the dead set. Five to ten attempts is plenty for most jobs.

**Timeouts.** Every job gets a maximum runtime, or a hung HTTP call ties up a worker forever.

Here are those rules in each tool. Celery, with automatic retry on transient errors only:

```python
# app/tasks.py
from celery import shared_task

class TransientError(Exception): ...

@shared_task(
    bind=True,
    autoretry_for=(TransientError,),
    max_retries=6,
    retry_backoff=True,        # 1s, 2s, 4s ... capped by retry_backoff_max (600s)
    retry_jitter=True,
    acks_late=True,            # redeliver if the worker dies; requires idempotency
    soft_time_limit=60,
    time_limit=90,
)
def send_email(self, user_id: int, template: str) -> None:
    deliver(user_id, template)  # PermanentError is not listed, so it fails immediately
```

Sidekiq, with a bounded count and a hook when retries run out:

```ruby
# app/jobs/send_email_job.rb
class SendEmailJob
  include Sidekiq::Job
  sidekiq_options retry: 8, queue: "mailers"

  sidekiq_retries_exhausted do |job, ex|
    Sidekiq.logger.warn "Giving up on #{job['class']} #{job['jid']}: #{ex.message}"
  end

  def perform(user_id, template)
    Mailer.deliver(user_id, template)
  rescue Mailer::InvalidAddress
    # permanent: do not retry
  end
end
```

BullMQ, with exponential backoff, jitter and an explicit stop:

```typescript
// src/queue.ts
import { Queue, Worker, UnrecoverableError } from "bullmq";

const connection = { host: "localhost", port: 6379, maxRetriesPerRequest: null };

export const emails = new Queue("emails", {
  connection,
  defaultJobOptions: {
    attempts: 6,
    backoff: { type: "exponential", delay: 1000, jitter: 0.5 },
    removeOnComplete: 1000,
  },
});

new Worker("emails", async (job) => {
  const user = await findUser(job.data.userId);
  if (!user) throw new UnrecoverableError("user deleted");
  await deliver(user, job.data.template);
}, { connection, concurrency: 10 });
```

BullMQ's exponential backoff waits `2 ^ (attempts - 1) * delay` milliseconds, so the sixth attempt here waits about 32 seconds before jitter.

## Dead sets need an owner

A job that exhausts retries lands in Sidekiq's Dead set, BullMQ's failed set, or wherever your Celery failure handler puts it. That is a triage queue for humans, not a graveyard. Alert when it grows faster than a few jobs an hour, have someone classify each entry as replay, discard or investigate, and archive anything older than a week. The full discipline is in our [dead-letter queues](/blog/dead-letter-queues) guide.

## Scheduling and operating workers

**Cap delays.** Jobs scheduled months ahead sit in Redis and get forgotten. For "in six months", store the date in your database and run a daily job, or use a real scheduler; our notes on [systemd timers vs cron](/blog/systemd-timers-vs-cron-what-we-learned) cover that side.

**Jitter everything.** Retries without jitter synchronize after a downstream outage and hit it as a thundering herd. Celery enables `retry_jitter` by default; BullMQ's jitter defaults to 0.

**Separate fast and slow queues.** A ten-minute export should never block password-reset emails. Give long jobs their own queue and worker pool.

**Modest concurrency, scale out.** Ten to twenty-five threads or a concurrency of ten per process, then add processes. More threads mean more contention on connection pools.

**Graceful shutdown.** Workers must finish or requeue current jobs on SIGTERM. Set your orchestrator's grace period longer than your longest typical job.

**Watch six signals**: queue depth per queue, job latency p95, error rate per job class, retry rate, dead set size, and worker restarts.

## Errors you will hit, and the fixes

**`WorkerLostError: Worker exited prematurely: signal 9 (SIGKILL)`** (Celery): the kernel OOM killer or a container memory limit killed the worker. Lower `worker_concurrency`, set `worker_max_memory_per_child`, or raise the limit.

**`SoftTimeLimitExceeded`** (Celery): the task passed `soft_time_limit`. Catch it to clean up, then fix the slow call rather than raising the limit.

**`RedisClient::CannotConnectError`** (Sidekiq): Redis is unreachable or out of connections. Check `REDIS_URL`, and make sure the pool is at least your thread concurrency.

**`job stalled more than allowable limit`** (BullMQ): CPU-heavy work blocked the Node event loop, so the worker couldn't renew its lock. Move CPU work to a sandboxed processor or yield more often.

**BullMQ throws on worker start about `maxRetriesPerRequest`**: an ioredis client passed to a Worker must have `maxRetriesPerRequest: null`.

## Where this breaks

These patterns assume jobs that finish in seconds or minutes. Multi-hour workflows with branching and human approval steps want a workflow engine (Temporal, Step Functions) instead. Redis-backed queues also lose data if Redis loses data, so run Redis with persistence and failover, or pick Celery on RabbitMQ when losing a job is unacceptable. Exactly-once delivery isn't on offer from any of the three; idempotency is how you get its effect.

## Questions people ask

### How many times does Sidekiq retry a failed job by default?

Twenty-five times, with polynomial backoff of roughly `retry_count ** 4 + 15` seconds plus random jitter, which spreads the attempts over about 20 days. After that the job moves to the Dead set, which holds up to 10,000 jobs or six months. Set `sidekiq_options retry: 8` per job to shorten it, or `retry: 0` to skip retries.

### Does Celery retry failed tasks automatically?

No. A Celery task fails once unless you call `self.retry()` or list exceptions in `autoretry_for`. When retrying, `max_retries` defaults to 3 and `default_retry_delay` to 180 seconds. Set `retry_backoff=True` for exponential delays capped at 600 seconds by default; `retry_jitter` is on by default.

### Sidekiq vs Celery vs BullMQ: which should I use?

Use the one native to your stack: Sidekiq for Ruby, Celery for Python, BullMQ for Node and TypeScript. Their architectures are close enough that language fit and team familiarity matter more than features. If several languages need to share work, put a message broker between services instead of forcing one job library across them.

## The call we'd make

Use the queue native to your language and spend your effort on the patterns, not the tool. Every job idempotent, IDs not objects, five to ten retries with backoff and jitter, permanent errors sent straight to the dead set, a named owner for that dead set, and alerts on queue depth. Teams that struggle with job queues usually skip idempotency, keep Sidekiq's 20-day default, or never look at the dead set. Fix those three and most of the pain goes away.

## Sources

- [Sidekiq wiki: Error Handling (retries and the Dead set)](https://github.com/sidekiq/sidekiq/wiki/Error-Handling)
- [Celery user guide: Tasks (retries, acks_late, time limits)](https://docs.celeryq.dev/en/stable/userguide/tasks.html)
- [BullMQ docs: Retrying failing jobs](https://docs.bullmq.io/guide/retrying-failing-jobs)
- [BullMQ docs: Stop retrying jobs](https://docs.bullmq.io/patterns/stop-retrying-jobs)