Testing assertions by reintroducing bugs

Legibility is the link in my strategy game's simulation that measures how well the state can see its own country, and in one version it hit its 1.0 cap by year 3 of a twenty-year game. I fixed it and wrote two assertions to keep it fixed. Then I put the bug back, and both assertions passed. They counted line-years sitting exactly at the cap, and with the bug in that was 13%. The bug showed up in a different quantity: the median line-year sat at 0.89 and 46% of them were above 0.90, against a median of 0.42 and none above 0.90 with the fix in.

I now count a check as evidence against a bug only after I've put that bug back and watched the check fail. The saturation check now measures the share of line-years above 0.90 for every link and fails at 10%. I set that from the measurement: across three budget mixes the reintroduced bug puts 30% of line-years above 0.90, and the fixed simulation puts none there.

The system is the simulation core of that game, one Python file, game/sim.py, checked by seven check scripts that ./verify.sh runs as one command. What follows is the runner I built from that lesson, game/mutants.py, which reintroduces twelve bugs the project actually shipped and reports which ones no check catches, the four checks that passed with their bug in, and the one time the runner put a mutant into a commit. It claims nothing about generated-mutant tools beyond the one paragraph that compares them.

The build log records coverage at 8 of 12 on the first run and 10 of 12 after two mutations were retargeted. Of the last two, one was closed by resetting a threshold from measurement and the other is a redundant clamp the runner marks as an expected miss, so the commit that closed them, bfadefa, stands at 11 caught. I reran the runner on a copy of that commit and on a copy of the current working tree, with the baseline checks passing on both, and each catches 11 of 12. The one it misses is the clamp.

Put the Bug Back

game/mutants.py holds a list of twelve entries. Each one is a name, a line copied exactly from game/sim.py, and the line that brings an old bug back. The runner writes the mutated file, runs game.checks, records whether any check failed, and restores the original before the next entry. If the anchor line isn't in sim.py any more, it reports that mutation as stale and counts it as a miss rather than skipping it without saying so.

The twelve are regressions the project actually had: supply response removed, arbitrage overshooting, welfare ignoring price, extraction made free, an override that costs nothing, a census that doesn't correct belief, no local production floor, a treasury that can go negative, legibility saturating, instrument costs that don't scale with the map, a railway that doesn't extend its network, and population that never grows. Most are one line. Supply response, for example, is the elasticity of 0.35 for grain and 0.55 for everything else set to zero.

# Reduced from game/mutants.py: two of the twelve entries and the loop.# (The source's stamp has a dash where this one has a comma.)MUTANTS = [    ("supply response removed",     'elast = 0.35 if g == "grain" else 0.55',     'elast = 0.0 if g == "grain" else 0.0'),    ("population never grows",     "            p.pop = max(0.5, p.pop * (1.0 + rate))",     "            p.pop = max(0.5, p.pop)"),]for name, old, new in MUTANTS:    if old not in src:                      # the anchor moved: report it as stale        missed.append(name + " [stale]")        continue    stamp = f"# MUTANT: {name}, mutation run in progress, do not commit\n"    open(SIM, "w").write(stamp + src.replace(old, new, 1))    caught, why = run_suite()               # runs game.checks, True if anything failed    if not caught:        missed.append(name)    open(SIM, "w").write(src)

Two of the first run's four misses turned out not to be bugs any more. The arbitrage mutation first broke the stale-spread re-check, and that moved the median year-over-year price volatility from 26% to 20% and the share of swings over 150% from 3.6% to 3.8%. No check could tell that apart from a healthy run, because the thing actually holding the market together was the damping on the size of each move. The entry now removes the damping: min(equalise * 0.6, lo.stocks[g] * 0.30) becomes lo.stocks[g] * 0.30.

The other was instrument costs. Unscaling them moved the treasury against revenue only from 1.9x to 2.0x, because the links had been tightened and spending was now limited by the binding link long before it was limited by money. Its entry keeps a comment saying so, and the entry stays in the list. On the current tree it's caught, and the first two checks to fail are "no link spends most of a run near its cap" and "money stays scarce", which fails when the mean treasury reaches 3x mean revenue. A mutation that stops biting gets a note in the file rather than a deletion, so the record that the bug was real survives.

You could ask why I didn't use mutation testing, which already exists as a tool. PIT for Java and mutmut for Python generate mutants by the hundred, flipping operators and deleting statements, so why write twelve by hand? A generated mutant measures whether the suite notices an arbitrary edit, and some arbitrary edits change nothing a check could observe. Each surviving mutant then has to be read by a person to tell a hole from a no-op. The arbitrage re-check above is that problem on a small scale: a real bug once, and by the time the runner existed, an edit that moved nothing. The twelve here are edits this codebase has already made once, so a miss means a bug the project has demonstrably written can come back with every check green.

Four Checks That Passed With the Bug In

Across the check scripts, four checks passed the first time their bug was put back. The build log lists them as "the pattern most worth keeping: a test that counted an action rather than its effect, a test that compared fields the broken state happened to share, a tolerance wide enough to be comfortable, and a dominance test whose condition no option could meet."

The controls script, game.inputs, plays a whole game through the real event handlers. I reintroduced six handler regressions one at a time and five were caught. The sixth broke the mouse handler for the crisis dialog. The test clicked a crisis button and counted the click. With the handler broken the crisis stayed up, the next pass answered it by keyboard, and the counter still read one answer by click. The check now requires the crisis to be gone.

Loading a save had nine ways to kill a game in progress, and the worst was a save missing one province, which loaded silently as a partial world. I added validation and a check that a bad save is refused, then removed the validation, and the check passed. It compared the year and the treasury before and after the load. A save of the same turn with one province cut out has the same year and the same treasury. Now it checks that the game object after a refused load is the same object as before. If the bad save had gone through, it would be a new one.

game.readme_check re-measures the numbers the README states. The README said the railway takes price dispersion from 3.98x to 1.69x. On its first run the checker's playthrough loop counted answering a crisis as a turn, so every run finished short, and it measured 3.40x to 1.82x. The tolerances I'd picked were wide enough to pass that. I fixed the loop and tightened the bands to ±0.20 and ±0.12. The short runs sit 0.58 and 0.13 away from the claim, so both numbers now fall outside their band.

game.deadcontent forks each crisis from identical states and flags a choice that wins on every axis, and one that loses on every axis. As first written, "best on every axis" was a condition no choice could meet, because every option costs money. Cash came out of the axes. The losing flag was blind the other way: population wasn't an axis, so quarantine, whose whole benefit is keeping people alive, was called dominated. Subjects went in. Then I validated both flags by putting each bug back. Removing the commission's cost in clerical consent makes inaction the option nobody should take, and removing inaction's mortality makes it quarantine.

The build log now carries what those fixes came to as three rules. Assert the state the action was supposed to change. Compare identity, not fields, when the question is whether something got replaced. Set a threshold by measuring both the working and the broken state, and re-derive it when the constants under it move.

The Runner Corrupted a Commit

The runner rewrites game/sim.py in place. During one run a git add -A fired and captured the mutated file, and a separate checks run against the same rewritten file reported seven failures that weren't real. I scanned all 31 commits for the twelve mutations' replacement text, and exactly one carried one: fd3212d, the commit that added mutants.py itself. It has unit_cost * 0.25 where the working line is unit_cost * (held / 8.0) ** 0.85, which is the instrument-cost mutant. The docstring above it still says a census of fourteen counties costs more than a census of eight, while the body returns a constant.

I didn't rewrite history. The commit was one back from HEAD and already superseded, HEAD verified green, and the repo had no remote. The log records it instead: don't bisect across fd3212d, and don't treat it as a known-good point.

I added four guards. The runner writes .mutants.lock when it starts and refuses to start if one exists, and verify.sh refuses to run while it exists. The mutated game/sim.py carries the # MUTANT: stamp on line 1, which Python ignores and anyone who opens the file mid-run sees (fd3212d was committed without one). The pre-commit hook is the gate, reduced here to its two tests:

if [ -f .mutants.lock ]; then exit 1; fiif git diff --cached -U0 -- game/sim.py | grep -q '^+.*# MUTANT'; then exit 1; fi

The fourth guard is about the other half of the run. The runner reads the check scripts as well as rewriting the simulation, and once one of them was edited mid-run, so one mutation was judged by a different suite than another and the report blended two experiments. It now hashes game/checks.py, game/deadcontent.py and game/adversarial.py at the start and exits with code 3, without reporting, if the hash has changed by the end.

Figure 2.17.3 drops a commit anywhere along a run. With the guards in, the lock refuses it, and the stamp would trip the gate if the lock were somehow gone. The hash isn't drawn.

The Last Two Misses

The log's 10 of 12 names the two misses: "census does not correct belief" and "treasury may go negative". The census mutation was missed because belief has two paths to correction. Disabling the census correction moved a census-heavy state's final belief error only from 7.9% to 12.4%, because funding the census raises the clerks, the clerks raise the register, and the register unlocks vital registration, which corrects belief anyway. The check's 0.60 ratio threshold couldn't separate 0.26 from 0.40, so it was reset to 0.34 from that measurement, and that closed it.

Commit bfadefa records the result as "Coverage 8/12 -> 12/12 (the one miss is an explicitly marked redundant clamp)", and a rerun of that commit has the census mutant failing "funding the census sharpens the state's belief". The 0.34 didn't last. A later fix to the army moved the working and broken ratios to 0.226 and 0.304, both under it, and the threshold went to 0.27. On the current tree the census mutant is caught again, and the first two checks to fail are "a completed census files a new count and preserves earlier evidence" and "the state holds more than two kinds of number for the same fact".

The treasury mutation removes the clamp in self.treasury = max(0.0, self.treasury - spent). Appropriations are already scaled down to the treasury before anything is spent, so spent <= treasury holds without the clamp and removing it can't produce a negative balance. No check can fail on it, and a check that did would be a second copy of the scaling. The clamp stays as defence in depth, and the mutation stays in the list marked as an expected miss. The runner's tally separates misses into expected ones and holes, and exits non-zero only on a hole.

The Playbook

Keep a list of the bugs you actually shipped, as code that reintroduces each one. An anchor string and its replacement is enough, and an anchor that stops matching gets reported as stale.

Run every check against every mutant.

When a check passes with the bug in, fix the check. Assert the effect rather than the action, compare against an expectation the broken state can't share, and set the tolerance from the gap between the working and the broken measurement.

Guard the runner, because it writes broken code on purpose: a lock that other tools check, a stamp in the mutated file, a pre-commit gate that refuses either, and a hash of the check scripts so a report is from one suite.

Report coverage as k of n with the uncovered ones named, and say for each one whether it's a hole or an expected miss.

Limits

These are twelve bugs from one project. The counts won't carry to other codebases. Coverage here means some check failed with the mutant in, not that the right check failed for the right reason, and the log doesn't record which checks caught each of the first run's eight. The commit scan found one corrupted commit among 31. Whether any run left something the scan's twelve replacement strings wouldn't match is not established beyond the log. The 11 of 12 holds at bfadefa and on the current working tree, but the checks that catch each mutant differ between the two.

If your suite has a check written for a specific bug, put that bug back and run the suite. It costs one string replacement and one test run.