a health check is a promise to the supervisor

a health check is the one sentence a process gets to say to whatever supervises it — a kubelet, fly's machine runner, docker's restart policy, a load balancer. the supervisor can only do two things with the answer: keep sending traffic, or stop and replace the process. so the check is a contract: 200 means "replacing me would not help"; anything else means "replace me". a check that answers 200 for any other reason has opted the process out of the only automated recovery it has.

the default check everyone ships — "the http listener answered" — proves exactly one thing: the accept loop is scheduled. it says nothing about the thread that does the work. an ingester whose upstream subscription died, a poller whose loop is wedged, a relay that is connected and delivering a quarter of the firehose — all of them answer 200, and all of them are replaced only when a human notices. this is masked-failure at the narrowest possible point: the check is the defense that hides the failure.

what "healthy" has to mean

the question the check must answer is not "am i running" but "did i recently do the thing i exist to do". that has three parts, and each one is a place the naive version goes wrong.

a heartbeat of work, not of life. record a monotonic timestamp when a unit of useful work completes — an event ingested, a poll iteration that succeeded. a loop that raised and swallowed the exception must not bump it. if the heartbeat fires on every iteration regardless of outcome it measures the scheduler, and the scheduler is never the thing that broke.

staleness, judged at read time. health is now - last_work < threshold. the threshold is a property of the workload: a firehose that is never quiet for a minute can use 120s; a poller whose iteration can block on a long agent run needs tens of intervals or it flaps. pick it from how the service actually behaves, and make it a runtime knob (see below).

deliberate idleness is not a fault. paused, draining, in startup grace — these are states the operator chose, and replacing the process would undo the choice. report them in the body, keep returning 200.

the supervisor decides what a failure does

this is the part that is easy to get wrong because every platform is different, and the check does nothing by itself.

• kubernetes liveness: failureThreshold × periodSeconds of failures → the container is killed and restarted. readiness: failures → removed from endpoints, not restarted. they are different probes for a reason.
• fly.io [[http_service.checks]]: a failing check makes the proxy stop routing to the machine. it does not restart it. a machine is restarted only when its process exits, per [[restart]]. a 503 from /health on fly is readiness, not liveness — the recovery has to be a watchdog in the process that exits non-zero.
• docker restart: unless-stopped: restarts on exit only. a HEALTHCHECK marks the container unhealthy; nothing acts on that unless an orchestrator is watching.

so the shape that always works is two cooperating pieces: a truthful endpoint (the observability surface — dashboards, humans, proxies read it) and a watchdog that turns the same verdict into an exit where the platform needs one. the verdict logic is a pure function; both callers use it; the test covers the function.

exit without losing state

a watchdog exit is a crash you scheduled. make it as cheap as the crash you were already going to have: state that matters is written atomically on each tick or by a periodic saver, and the watchdog routes through the same save-then-exit path as SIGTERM rather than a second one. the bound on loss should be a number you can state ("at most 30s of graph updates").

the backstop that caused an outage

the zlay relay had, for a week in august 2026, a process that was alive, answering 200, and 75% deaf — workers connected to upstream hosts and delivering nothing. the first fix was a per-worker sweep: tear down any worker that had gone silent. it tore down 4283 healthy workers in an hour and took the relay to zero. silence per worker is the normal case on a long tail of mostly-idle hosts; silence process-wide is not.

two rules fell out of the revert:

• judge liveness at the widest scope you can. the process-wide ingest rate is a signal the check can act on; a single worker's rate is noise. the restart is coarse, so the condition that triggers it should be too.
• a backstop needs a runtime kill switch before it needs to exist. during the incident the only lever was an image rollback because the timeout was compile-time. the standing play for a wedge is capture-before-restart — forensics first — and a liveness probe that can't be paused from an admin endpoint destroys the evidence.

and one ordering constraint: the alert should page a human before the probe kills the pod. set the liveness window longer than the alert window, so a 3am restart is never the first anyone hears of it.

case studies (2026-08-22)

one audit across five services found the same gap in each. what each one shipped:

two of these had the watchdog before and lost it: coral's python bridge self-terminated on a dead subscription and the safeguard was deleted with the rewrite; typeahead's ingester exits after 10 minutes of silence (see masked-failure). a rewrite that keeps the endpoint and drops the exit path keeps the appearance of the check and none of its function.

sources

• zlay — docs/deployment.md, and the relay repo's docs/incidents/wedge-capture-20260818T220432Z/ANALYSIS.md
• coral — docs/04-architecture.md health section
• bot — src/bot/core/watchdog.py, AGENTS.md
• stream — docs/gotchas.md (StreamProcessRestarted), deploy/prometheus/rules.yml
• fly.io health checks — "your Machines won't automatically restart or stop due to failing their health checks"; configuration reference — [[restart]] applies when "a machine unexpectedly exits"