operations

running and shipping software: what breaks after deploy, how to find out, how to size what you rent, and why the deploy path itself is the least-tested code you own.

design patterns live in systems/. the split is roughly "how to build it" against "what it does to you at 2am".

a recurring theme across these notes: the failures that stay open longest do not produce errors. they produce silence, a green dashboard, or a confidently wrong belief.

topics

observability and failure

• masked-failure — a reliability feature moves failure from loud and small to silent and large; watchdogs, heartbeats, staleness
• health-checks — 200 means "replacing me would not help"; heartbeat of work, verdict at read time, supervisor decides what a failure does (fly checks route, they do not restart), kill switch first
• observability-of-absence — absent is not zero, a disabled invariant is an open outage, and the signals you must add deliberately because nothing emits them

diagnosis

• throughput-bottlenecks — bound from outside, profile the real workload, read upstream's source; one variable per deploy
• bounded-scans — a bound limits work performed, not work read; push the skip check to the cheapest tier, and guarantee the cursor moves

configuration and policy

• reconcilers-own-config — if a reconciler owns a value, a hand edit is temporary; change the source, not the reconciled object
• policy-enforced-by-accident — a rule enforced by a dependency's implementation detail has no owner; enforce at a layer you own, and test the policy rather than the mechanism

what you rent

• deploy-path — untested deploy branches are bug reservoirs; use the repository's recipe, verify the launcher ran your command
• instance-sizing — measure the working set at a fresh start; start small because disk rescale is one-way; read the whole bill
• home-infra/ — the home-box cluster: cost optimization, public cost declaration, tailscale