leases and expiry

a resource whose release depends on a live actor (a queue message, a concurrency slot, a lock, a transitional state awaiting confirmation) is eventually held by something that is dead. the holder dying is not an edge case; it is the failure the bookkeeping exists to survive.

ownership therefore needs a deadline the owner does not control. there are two ways a holding ends:

release on completion is the fast path. it is never the whole answer, because it runs only when the holder survives to run it.

the symptom

nothing errors. a queue stops moving, a slot never frees, a run sits in a transitional state forever. nothing logs, because from the system's point of view somebody still owns the thing and is presumably working on it. Prefect's design issue for this (#17415) describes orphaned slots that "become unavailable to other tasks, gradually depleting the pool", with no indication of which tasks hold them.

three server-owned deadlines

• redis stream messages (pydocket worker.py): a consumer that dies after XREADGROUP but before XACK leaves the message in its pending entries list forever. XAUTOCLAIM with min_idle_time reclaims it after redelivery_timeout (default 5 min). without this, "at least once" is silently "at most once".
• concurrency slots (Prefect): a slot is a lease with a TTL (initial_deployment_lease_duration, default 300 s), renewed by the holder at 0.75 of the duration, and a Repossessor service (loop 15 s) decrements slots for expired leases. v1 tag limits go through leases with a 100-year TTL, so they are never repossessed: the bug class is fixed only where a real TTL was chosen.
• transitional states (Prefect CANCELLING): a run whose infrastructure died can never confirm its own cancellation. the server schedules a keyed deadline on entering the state (cancelling_timeout_seconds, default 300) and on expiry writes Cancelled itself. the worker becomes a stateless "kill this pid" executor that does not propose state transitions (#21911).

the general rule: every non-terminal state that waits on an external actor to confirm completion needs a deadline owned by the server.

lease design details

• Key the deadline to the resource (cancelling-timeout:<run_id>) so re-entering the state is idempotent, and sweep for pre-existing holders on startup — otherwise a restart strands everything already stuck.
• Guard the reclaim against races: Prefect only enqueues cleanup if the row's locked state id still equals the id its own timeout task wrote, so a late worker update or a spoofed message cannot trigger teardown.
• Renew at a fraction of the TTL (0.75) with bounded retries, and decide explicitly what a renewal failure means — continue and risk over-allocation, or abort. Prefect's flow engine chooses abort: "slots are no longer reserved. Terminating execution to prevent over-allocation."
• Reclaim is not the same as crash detection. Prefect's Foreman marks workers offline on stale heartbeats but never marks their runs CRASHED; that is a separate heartbeat+automation path. Don't assume one implies the other.
• Hold a slot until started work drains. Persist the admission ticket before work begins, bind release to the execution owner, and let cancellation keep the slot until already-started requests finish; a remote request can outlive the client's timeout. A cancel-before-create tombstone stops a delayed admission from resurrecting cancelled work. A missed heartbeat is suspicion, not proof the work stopped: reclaim after an enforced execution lifetime or an explicit drain.

slot counts and owners

Expiry adds a second releaser, and a counter of slots in use stays correct only if every release is tied to one owner and happens once.

• A slot belongs to its holder. A counter that any exit from an active state decrements is anonymous: a run that never took a slot (created directly in an active state) gives one back on exit, and the limit admits one run too many while the real holder still runs. Record which holder took the slot and release it in the same transaction as the holder's own state change.
• Release once, decided by the revoke. "Read the lease, decrement its limits, revoke it" lets two overlapping releases of one lease (a client retrying after a timeout, or the holder racing the reclaimer) both decrement, and the second frees another holder's slot. Make removing the lease the claim: only the release whose revoke actually removed it decrements.
• The lease and the count share durability. Leases held in process memory over a count stored in the database strand every held slot on restart: the reclaimer has no lease to expire, the holder's release finds none and frees nothing, and the limit stays full until someone edits the count by hand. Keep both in the same store, or rebuild one from the other at startup.

All three pass any test where one client acts alone on a healthy server; a simulation that interleaves releases and restarts finds each in a few steps (deterministic simulation testing).

sources

• Prefect: #17415 (lease design), #18492 (repossessor), #18567 (deployment limits on leases), #21860
  • #22238 (CANCELLING timeout), #9895 / #16027 / #17563 (the bugs)
• pydocket worker.py: get_redeliveries()
• observed 2026-08-10 operating stream's gate: a flow run whose process was killed by a worker restart sat in CANCELLING holding the deployment's only concurrency slot, so the next gate sat in SCHEDULED behind a ghost. The server would have healed it in 300 s; the concurrency limit added to prevent double-runs had briefly become a deadlock instead.
• prefect-server — dst/findings/008-slot-given-back-by-a-run-that-never-took-one, 004-restart-strands-lease-slots, 022-lease-released-twice-frees-another-slot, and the holder-owned slot in commit b03e631, 2026-10-09; the same release race and restart in Prefect 3.8.2: #23384, #23389
• pensieve, docs/OPERATIONS.md "Shared indexing admission" (SQLite Durable Object tickets, drain-before-release, bounded recovery), 2026-09; Cloudflare Workers limits.