background tasks

the pattern: a producer schedules a unit of work (now, or at time T); a worker pool consumes the work and runs it; results land somewhere observable; failures retry or terminate cleanly. the variation between systems is where the queue lives and what the consumer-side semantics are.

the lineage

• celery (python, 2009) — broker (redis or rabbitmq) + worker processes + result backend. dead-simple decorators; weak typing; "the worker dies and your task disappears" is a regular operational problem. one of the first widely-adopted patterns.
• sidekiq (ruby, 2012) — redis-backed; uses redis lists as the queue. fast, single-binary worker. proof that redis-as-broker can carry serious load.
• arq (python, 2017) — asyncio-native; redis streams instead of lists. introduced the at-least-once-via-PEL model to python.
• pydocket (python, 2024) — same redis-streams substrate as arq, but typed-task-centric, with interval-rescheduled "perpetual" tasks and atomic Lua scripts for the queue transitions. used by self-hosted prefect.

what's unchanged across the lineage:

• "a task" is a named function + JSON-serializable params
• "the queue" is durable and survives worker restarts
• consumer groups (or list pops) load-balance work across N workers
• task completion is signaled by ack, not by exit code
• failed deliveries stay claimed until orphan-recovery reassigns them

why a remembering queue beats a forgetting one

a list pop (BLPOP-style) hands work over and forgets — consumer death means a lost task. a log with per-consumer pending state (redis streams' PEL, SQS visibility timeouts, kafka consumer offsets) keeps an unacked delivery visible to recovery, preserves order for inspection, and load-balances N workers without manual sharding. the cost is a bigger protocol surface and an append-only log that needs trimming. the redis mechanics (PEL, XAUTOCLAIM, XTRIM) live in storage/redis/streams.

at-least-once is the contract

delivery and ack are separate steps, and anything can happen between them:

• worker dies → entry stays pending → another worker claims it eventually → re-execution
• worker hangs → same path, just delayed
• worker succeeds but crashes before ack → re-execution

so a task body has to be idempotent on the keys it touches. "set state to Late" is idempotent for free; "send an email" needs an idempotency key the task itself enforces. at-most-once is not on the menu — the moment you have orphan recovery, duplicate delivery becomes possible.

perpetuals

a perpetual task reschedules itself after each success. the naive form — the task body reschedules itself at the end — can silently die if the reschedule fails after the ack. the robust form moves rescheduling into the worker and orders it before the ack: if reschedule fails, the entry stays pending and orphan recovery retries the whole pair. reschedule-then-ack trades at-most-once-reschedule for "a perpetual never silently dies." periodic maintenance loops (schedulers, lease cleanup, lateness sweeps) are all natural perpetuals.

scheduling for future execution

an append-only queue has no "deliver at time T" — appends are now. so future work needs a parking structure and a promotion pass: a sorted set keyed by due-time, payload parked alongside, and a periodic atomic script that moves due entries into the live queue. multiple workers running the promotion race harmlessly when each pass is atomic and clears what it promoted. the concrete redis shape (ZSET + parked hash + Lua promote script) is in storage/redis/streams.

a swappable broker keeps the abstraction honest

a task system is the natural place to support "memory mode" for tests and small deployments: the consumer speaks one protocol and a URL scheme picks a real broker or an in-process compatible engine. the constraint worth enforcing is that features land for both backends simultaneously — no "memory-only" milestones — because that forces the abstraction to stay at the protocol, not leak either implementation. measured example: docket runs the same Lua scripts byte-for-byte against redis:// or an in-process engine (burner-redis) via memory://, selected per-deploy.

what a task queue is not

• not a flow engine — no DAGs, no inter-task dependencies, no fan-out; tasks are independent
• not an RPC system — you can't await a return value; storing results for later inspection is an explicit, separate operation
• not a cron — perpetuals reschedule relative to completion; "every monday at 09:00" is a layer on top

if you need those, the task queue is the substrate you'd build them on, not the answer.

sources

• docket — zig port of pydocket (2026)
• burner-redis — in-process redis-compat engine
• pydocket — chris guidry; the redis streams + PEL model and the three Lua scripts
• prefect-server's loop services (late_runs, scheduler, lease_cleanup, automations) — the perpetuals that motivated the design