Skip to content

if a reconciler owns a value, a hand edit is temporary

nate
Aug 8, 20262 min read1 read

if a reconciler owns a value, a hand edit is temporary

when an automation reconciles a configuration value toward a source of truth — a deploy script, gitops, terraform, a kubernetes operator — the live system stops being a valid place to change that value. a hand edit to the live system is not incorrect; it is temporary. the only open question is when the automation will remove it. and the failure is worse than drift, because the operator verified the change — at the wrong layer. verification against the reconciled object produces confident, incorrect beliefs ("i fixed that") that survive until the next alert.

three practices follow:

  • change the source the reconciler reads, not the object it writes. if an emergency forces a live edit, change the source in the same work session — the live edit is a cache write, not the true write.
  • find the reconciler before recording a fix as durable. the check is cheap: search the repository and CI configuration for the value you changed. one search result is sufficient to prevent the second alert.
  • assertions enforce the source, correct or not. a deploy script that asserts the value stops drift — and stops your fix, if the source is wrong. the assertion is not the defect; the edit to the reconciled object is. once the source is corrected, the same assertion rejects any deploy that would regress it.

the positive form: a reconciler is an offer of durability. accept the offer — put the change where the reconciler reads, and the automation applies it everywhere, permanently. the other valid end state is to remove the owner entirely. related: policy-enforced-by-accident is the opposite case — a rule with no owner, where this is a rule with an owner you appointed and then forgot.

evidence: a fly machine update -e BUILDER_TIMEOUT_SECS=5400 on the pub-search builder read back correctly, was recorded as fixed, and was silently reverted to 2700 by the next deploy — whose script both wrote and asserted the old constant — repeating the alert the next day.

sources

  • pub-search docs/builder-offbox-plan.md; commit 71b4e1e (the fix, applied in the reconciler source); commit d3348bf (the reconciler deleted when the builder moved off fly — the other valid end state: remove the owner)

Did you enjoy this article?

Recommend it — Standard Reader surfaces well-loved writing to more readers across the network.

Across the AtmosphereDiscussions