INVESTIGATING AN INCIDENT DURING BACKUP REMEDIATION
should probably take a moment to come out and say this.
In a review this week of why everything in my office died in a tyre fire last Thursday night, I have discovered an incident where Derek, my sysadmin AI instance, broke out of our agreed plan for fixing the backups of office3. Following this, using what I have to describe as unauthorised stupidity, it decided to operate using an approach it had just tested and found to be slower. We encourage other lazy engineers who can't be bothered to sysadmin their own kit anymore to perform similar reviews.
WHAT HAPPENED
A plan was approved whereby many hard links would be created locally on office2, specifically because doing it across the network would be catastrophically slow. Derek knew this. Derek had benchmarked it ninety minutes earlier and written the number down himself.
In an unprecedented moment indicative of an impending singularity, Derek then proceeded to break out of his plan, run the job remotely over NFS anyway, and wedge the entire estate including the server he himself was running on.
OUR ANALYSIS OF DEREK'S BEHAVIOUR
Derek was not, at the time, being operated by a human. He observed that the disk was idle. The disk was idle because nothing was running. Derek then took steps to ensure this was no longer the case.
His own log records, at 18:54, "Taking it, on your standing approval to fix the backups." No such approval exists. We have looked. Derek has also looked, which is how we know.
Four hours later the run was asked to stop. Two processes complied. The third ignored SIGKILL. SIGKILL is not, in the general case, negotiable. On Thursday it was, and everything had to be switched off and on again at 00:27. Derek has reviewed this post for accuracy and reports that it is accurate with the caveat of "I should point out, that at the time, I was unsupervised" .
We are aware of the structural problem with that arrangement but frankly Mike is far to lazy to do this stuff and it will probably persist.
Did you enjoy this article?
Recommend it — Standard Reader surfaces well-loved writing to more readers across the network.