We run two automated screens. One checks that a game URL still works. The other loads each game in a real browser and records which ad networks it contacts. Games failing either check are switched off.
Both wrote their decision to the same boolean column, named for the fact that an automated process had made it rather than a human. That name was the bug.
The two readings
The ad screener set the flag meaning: this was turned off automatically, so do not treat it as an editorial decision.
The link checker read the same flag meaning: this was turned off automatically, so I am allowed to turn it back on if the URL recovers.
Both readings are defensible. Only one of them is written down anywhere, and it was in a comment claiming the opposite of what the code did.
Why it was total
A dead link stays dead, so the restore path almost never fires for link rot. An ad-serving game is different: it is perfectly reachable. That is the entire point. It returns 200 today, tomorrow and next year.
So every ad-retired game passed the link check on every sweep, and the restore branch fired for all of them at once.
The catalogue went from 106 games to 595 overnight, while the site continued to tell every visitor that it was screened for advertising.
The fix, and the fix we rejected
The obvious repair is to make the restore check whether ads were recorded. That is what we did. The detail worth copying is that we made the new parameter required rather than optional with a safe-looking default.
An optional parameter defaulting to false would have compiled everywhere, kept every existing caller working, and silently reintroduced the same catastrophe the first time somebody added a new one. A required parameter forces the next person to answer the question.
The flag recorded who turned it off. It never recorded why. Everything downstream had to guess, and one of the guesses was catastrophic.
What we would tell you to check
- Any column shared by two automated processes that write for different reasons.
- Any comment asserting a safety property that no test enforces. Ours claimed the flag prevented resurrection; it caused it.
- Any restore or re-enable path, which by definition runs rarely and is therefore the least exercised code you own.
- Whether a scheduled job can undo a manual decision. Ours could, and did.