reliability features hide failures

in june 2026 the ingester behind typeahead (an actor-search service over atproto) died quietly and stayed dead for 13 days. nobody operating it noticed; a user noticed, because a stats chart flatlined. search kept working the whole time — the snapshot architecture that insulates queries from ingest trouble also kept the outage invisible. per the retro: errors were console-only, no alert existed, and the design that protects users "is why the wedge went unnoticed."

pub-search (search over atproto publications) hit the same mechanism from the other side. for months it ingested through a lossy feed that delivered ~30% of events. when a verified ingester replaced it and delivered 100%, a 13-second-per-record write path — present since january — finally mattered, and a six-hour incident cascade followed. from that retro: "the cutover didn't break anything — it removed the anesthesia."

the general statement both incidents instantiate: a reliability feature moves failure from loud and small to silent and large. a retry hides a flaky dependency until the dependency is fully down; a lossy feed hid a 13s write path until delivery hit 100%; a snapshot hid a dead ingester for 13 days. richard cook's how complex systems fail (1998) treats this as the baseline condition: complex systems "contain changing mixtures of failures latent within them" and "run in degraded mode" — the defenses working is exactly what lets the latent failures accumulate.

what both repos shipped after

• a watchdog: typeahead's ingester now exits after 10 minutes of silence and lets the platform restart it — "crashing fast converts any future wedge from weeks of data loss to none."
• heartbeats with absence alarms: a periodic "alive and did work" signal, alerted on its absence — the one signal a dead process can't emit. (phi, the bluesky agent in the sibling bot repo, is watched the same way: an alert fires when its telemetry goes quiet, since a quiet dashboard is what its outage looks like.)
• staleness as a metric: where one layer's health depends on another running recently (the overlay depends on snapshot builds — see serving-from-snapshots), measure the age and alarm on it.

chaos engineering is the same reasoning run proactively: if a defense hides a failure, the way to know the defense works is to cause the failure deliberately.

one line from the pub-search retro worth keeping verbatim, since it's a usable diagnostic rule: "absence of telemetry from a live system is itself a signal — of hanging, not idleness."

sources

• typeahead — docs/retros/2026-06-12-ingester-wedge-and-recovery.md
• pub-search — docs/retro-2026-06-10-cutover-cascade.md
• how complex systems fail — richard cook, 1998