if a reconciler owns a value, a hand edit is temporary

in july 2026, a watchdog stopped the pub-search snapshot builder after 45 minutes, on more than one day. the operator correction looked complete: fly machine update -e BUILDER_TIMEOUT_SECS=5400. the operator read the value back and saw 5400. the incident was recorded as fixed. the next backend deploy set the value back to 2700 without a message. a deploy script wrote the full environment of the builder machine from constant values. the script then asserted that the value was 2700. the same failure caused an alert on the next day. the operator now also had an incorrect belief: "i fixed that."

the general shape: when an automation reconciles a configuration value toward a source of truth, the live system stops being a valid place to change that value. examples of such automation are deploy scripts, gitops, terraform, and kubernetes operators. a hand edit to the live system is not incorrect. it is temporary. the only open question is when the automation will remove it. this failure is worse than drift, because the operator verified the change. verification at the wrong layer produces confident, incorrect beliefs.

three practices follow:

• change the source that the reconciler reads. do not change the object that the reconciler writes. if you must edit the live system in an emergency, change the source in the same work session. the live edit is a cache write, not the true write.
• find the reconciler before you record a fix as durable. the check is cheap: search the repository and the CI configuration for the value that you changed. the constant 2700 appeared twice in pub-search: in the --env flag and in the assertion. one search result was sufficient to prevent the second alert.
• assertions enforce the source, correct or not. the deploy script asserted the value to stop drift. it stopped the fix, because the source was wrong. this is still better than silent drift. after the source was corrected, the same assertion now fails any deploy that would set the old value again. the assertion was not the defect. the edit to the reconciled object was the defect.

the positive form: a reconciler is an offer of durability. accept the offer. put the change where the reconciler reads. the automation then applies the change everywhere, permanently. related: policy-enforced-by-accident is the opposite case. that note describes a rule with no owner. this note describes a rule with an owner that you appointed and then forgot.

sources

• pub-search docs/builder-offbox-plan.md; commit 71b4e1e (the fix, applied in the reconciler source); commit d3348bf (the reconciler deleted when the builder moved off fly — the other valid end state: remove the owner)