Job Queues: Sidekiq, Celery, BullMQ Patterns That Hold Up

Pick the queue that matches your language: Sidekiq for Ruby, Celery for Python, BullMQ for Node. Then apply the same rules in all three: idempotent jobs, IDs instead of objects, bounded retries with exponential backoff and jitter, no retries on permanent errors, a reviewed dead set, and alerts on queue depth. The defaults differ, and each one bites differently.

Why background jobs at all

Three needs recur. Latency: work the user shouldn't wait for, like a welcome email or a thumbnail. Reliability: work that must retry, like calling a flaky third-party API or processing a webhook. Bursty load: work that arrives in waves, which a queue smooths so workers process at their own pace.

All three tools share the same shape: a backend (Redis for Sidekiq and BullMQ; RabbitMQ or Redis for Celery), workers that pull and execute jobs, JSON-serialized job name plus arguments, retries with backoff, a dead or failed set, delayed jobs, and a web UI (Sidekiq Web, Flower, Bull Board). Switching languages means relearning syntax, not architecture. Cross-language workflows belong on a message bus instead; the trade-offs are in Kafka vs RabbitMQ.

The defaults that differ

Those defaults are the source of most surprises. Open-source Sidekiq removes a job from Redis when it fetches it, so a hard crash mid-job loses it. Sidekiq's 25 retries over about 20 days mean a broken job can keep firing for weeks. Celery's early acknowledgement means a worker killed mid-task loses that task unless you opt into acks_late, and then the docs say plainly: "Make sure your tasks are idempotent." BullMQ does nothing on failure until you configure attempts.

The patterns that earn their place

Idempotency. Every queue here is at-least-once in practice: retries, redeliveries after a crash and duplicate enqueues all happen. Either make the work naturally idempotent (UPDATE ... WHERE state = 'pending') or pass an idempotency key and check it at the destination.

Small, focused jobs. "Send email, update database, call API" in one job means a retry of step three repeats steps one and two. Split them and chain.

Pass IDs, not objects. A job takes user_id, not a serialized user. A retry five minutes later should read current state, not a stale snapshot.

Bounded retries, classified failures. Transient errors (timeouts, 429s, 503s) get exponential backoff with jitter; permanent errors (validation failures, 404s) go straight to the dead set. Five to ten attempts is plenty for most jobs.

Timeouts. Every job gets a maximum runtime, or a hung HTTP call ties up a worker forever.

Here are those rules in each tool. Celery, with automatic retry on transient errors only:

from celery import shared_task

class TransientError(Exception): ...

@shared_task(
    bind=True,
    autoretry_for=(TransientError,),
    max_retries=6,
    retry_backoff=True,        # 1s, 2s, 4s ... capped by retry_backoff_max (600s)
    retry_jitter=True,
    acks_late=True,            # redeliver if the worker dies; requires idempotency
    soft_time_limit=60,
    time_limit=90,
)
def send_email(self, user_id: int, template: str) -> None:
    deliver(user_id, template)  # PermanentError is not listed, so it fails immediately

Sidekiq, with a bounded count and a hook when retries run out:

class SendEmailJob
  include Sidekiq::Job
  sidekiq_options retry: 8, queue: "mailers"

  sidekiq_retries_exhausted do |job, ex|
    Sidekiq.logger.warn "Giving up on #{job['class']} #{job['jid']}: #{ex.message}"
  end

  def perform(user_id, template)
    Mailer.deliver(user_id, template)
  rescue Mailer::InvalidAddress
    # permanent: do not retry
  end
end

BullMQ, with exponential backoff, jitter and an explicit stop:

import { Queue, Worker, UnrecoverableError } from "bullmq";

const connection = { host: "localhost", port: 6379, maxRetriesPerRequest: null };

export const emails = new Queue("emails", {
  connection,
  defaultJobOptions: {
    attempts: 6,
    backoff: { type: "exponential", delay: 1000, jitter: 0.5 },
    removeOnComplete: 1000,
  },
});

new Worker("emails", async (job) => {
  const user = await findUser(job.data.userId);
  if (!user) throw new UnrecoverableError("user deleted");
  await deliver(user, job.data.template);
}, { connection, concurrency: 10 });

BullMQ's exponential backoff waits 2 ^ (attempts - 1) * delay milliseconds, so the sixth attempt here waits about 32 seconds before jitter.

Dead sets need an owner

A job that exhausts retries lands in Sidekiq's Dead set, BullMQ's failed set, or wherever your Celery failure handler puts it. That is a triage queue for humans, not a graveyard. Alert when it grows faster than a few jobs an hour, have someone classify each entry as replay, discard or investigate, and archive anything older than a week. The full discipline is in our dead-letter queues guide.

Scheduling and operating workers

Cap delays. Jobs scheduled months ahead sit in Redis and get forgotten. For "in six months", store the date in your database and run a daily job, or use a real scheduler; our notes on systemd timers vs cron cover that side.

Jitter everything. Retries without jitter synchronize after a downstream outage and hit it as a thundering herd. Celery enables retry_jitter by default; BullMQ's jitter defaults to 0.

Separate fast and slow queues. A ten-minute export should never block password-reset emails. Give long jobs their own queue and worker pool.

Modest concurrency, scale out. Ten to twenty-five threads or a concurrency of ten per process, then add processes. More threads mean more contention on connection pools.

Graceful shutdown. Workers must finish or requeue current jobs on SIGTERM. Set your orchestrator's grace period longer than your longest typical job.

Watch six signals: queue depth per queue, job latency p95, error rate per job class, retry rate, dead set size, and worker restarts.

Errors you will hit, and the fixes

WorkerLostError: Worker exited prematurely: signal 9 (SIGKILL) (Celery): the kernel OOM killer or a container memory limit killed the worker. Lower worker_concurrency, set worker_max_memory_per_child, or raise the limit.

SoftTimeLimitExceeded (Celery): the task passed soft_time_limit. Catch it to clean up, then fix the slow call rather than raising the limit.

RedisClient::CannotConnectError (Sidekiq): Redis is unreachable or out of connections. Check REDIS_URL, and make sure the pool is at least your thread concurrency.

job stalled more than allowable limit (BullMQ): CPU-heavy work blocked the Node event loop, so the worker couldn't renew its lock. Move CPU work to a sandboxed processor or yield more often.

BullMQ throws on worker start about maxRetriesPerRequest: an ioredis client passed to a Worker must have maxRetriesPerRequest: null.

Where this breaks

These patterns assume jobs that finish in seconds or minutes. Multi-hour workflows with branching and human approval steps want a workflow engine (Temporal, Step Functions) instead. Redis-backed queues also lose data if Redis loses data, so run Redis with persistence and failover, or pick Celery on RabbitMQ when losing a job is unacceptable. Exactly-once delivery isn't on offer from any of the three; idempotency is how you get its effect.

Questions people ask

How many times does Sidekiq retry a failed job by default?

Twenty-five times, with polynomial backoff of roughly retry_count ** 4 + 15 seconds plus random jitter, which spreads the attempts over about 20 days. After that the job moves to the Dead set, which holds up to 10,000 jobs or six months. Set sidekiq_options retry: 8 per job to shorten it, or retry: 0 to skip retries.

Does Celery retry failed tasks automatically?

No. A Celery task fails once unless you call self.retry() or list exceptions in autoretry_for. When retrying, max_retries defaults to 3 and default_retry_delay to 180 seconds. Set retry_backoff=True for exponential delays capped at 600 seconds by default; retry_jitter is on by default.

Sidekiq vs Celery vs BullMQ: which should I use?

Use the one native to your stack: Sidekiq for Ruby, Celery for Python, BullMQ for Node and TypeScript. Their architectures are close enough that language fit and team familiarity matter more than features. If several languages need to share work, put a message broker between services instead of forcing one job library across them.

The call we'd make

Use the queue native to your language and spend your effort on the patterns, not the tool. Every job idempotent, IDs not objects, five to ten retries with backoff and jitter, permanent errors sent straight to the dead set, a named owner for that dead set, and alerts on queue depth. Teams that struggle with job queues usually skip idempotency, keep Sidekiq's 20-day default, or never look at the dead set. Fix those three and most of the pain goes away.

Sources

• Sidekiq wiki: Error Handling (retries and the Dead set)
• Celery user guide: Tasks (retries, acks_late, time limits)
• BullMQ docs: Retrying failing jobs
• BullMQ docs: Stop retrying jobs