Counting the assumptions a test shares with its code

Before I cite a test result, I count the conventions the code keys off and check how many of them the test's data happened to satisfy. The code here is Cairn, a tool I built to catch confabulation in long-horizon LLM fiction writing. It walks a project's files and indexes the ones that carry canon frontmatter. From the index it writes a compact canon file that the model reads each session. A gate then checks the model's new claims against the canon as it stood when the claim was generated. Over the 70 projects it was written for, only 25 kept the walker's .md rule and only 3 of 69 repos kept the commit-subject rule the gate anchored on, and both projects the dogfoods had run on sat in the corner where four of its rules held at once.

I had validated it on two real projects. On Project P, a full dogfood with 66 entities, the resident context went from about 16,633 tokens to about 3,254. Project I's summary-only projection dropped from about 14,588 to about 1,598.

Then I ran a read-only sweep over all 70 projects in the corpus it was built for. The walker crashed on none of them. The sweep counted 1,898 content files; 747 of those ended in .md, and the walker read only .md. 1,117 files, 59% of the corpus, were extensionless prose with names like notepad and outline. 45 of the 70 projects contained real writing and zero .md files. On those the tool indexed nothing and reported nothing wrong.

Both dogfoods had run on that corner. Projects P and I were among the 25 that use .md, and among the 3 repos out of 69 whose commit subjects start Session N, which the gate's anchor keyed off.

The corpus is one author's, 70 fiction projects with 69 of them git repos, all under one writing directory, and I make no claim about any other. Every corpus-wide count here comes from two read-only harnesses I ran over it. The per-project numbers come from dogfood runs on three of its projects. All of it is recorded in the design document, and nothing was re-run for the essay.

The Walker Read Only .md, Which Hid 45 of 70 Projects

The .md gap was already in the record. The first dogfood had found that Project P has 11 extensionless tracked elaboration files, and I had logged it as a known gap, "Needs a configurable include-glob (deferred, logged)". The walker hadn't changed by a line when the breadth sweep moved the entry from a cosmetic deferral to the primary adoption blocker. A continuity substrate can't be adopted on the 45 projects where it is invisible, 64% of the corpus it serves.

The question that opened the audit is in the design doc: "You can't count on everything being .md. Who told you that?" The record's answer is one or two example projects. The .md filter had been keyed off what those projects did, and it held in 25 of the 70.

The walker before the fix, from the first commit, trimmed to the branch that matters:

function walk(root: string, out: string[]): void {  // ...  for (const name of entries) {    if (SKIP.has(name)) continue;    const full = join(root, name);    const st = statSync(full);    if (st.isDirectory()) walk(full, out);    else if (name.endsWith(".md")) out.push(full);  }}export function walkMarkdown(root: string): string[] {

After the fix the default set is written out where a caller can read it, and an empty string in the set stands for "no extension". Binary files are sniffed out by a NUL byte in the first 8 KiB. A [walk] table in the project config can add extensions or deny basenames:

const DEFAULT_EXT = new Set([".md", ".markdown", ".txt", ".text", ""]);export function walkContent(  root: string,  opts: WalkOpts = {}): string[] {  const inc = new Set([...DEFAULT_EXT, ...(opts.include ?? [])]);  const out: string[] = [];  walk(    root, out, inc,    opts.exclude ?? new Set(),    opts.maxBytes ?? 1_000_000  );  return out;}

I renamed walkMarkdown to walkContent, since the old name had the assumption in it. A second .md filter, in the committed corpus the linter checks unknown names against, got the same set in the same commit. Re-running the breadth sweep with the generalized walker took the projects it was invisible on from 45 to 0, still with 0 crashes.

The Assumption Audit Tests Every Convention Against the Whole Corpus

The audit's rule, as I wrote it into the design doc, is that every assumption gets tested against the actual 70-project corpus and its git history, and none gets reasoned about. The first step is a list of every convention the code keys off, which is a grep for regexes, extension filters, path layouts and tag formats. In Cairn that turned up the walker's extension filter, the gate's commit-subject regex, the linter's proper-noun regex, the migration bootstrap's directory layout, the parser's UTF-8 assumption and the assumption that a project is a git repo.

Next I count how many of the 70 real projects satisfy each one, or sample when a count is too costly, as with the 400-file UTF-8 check. The regex row carries what it drops instead of a count. Then I rank them by what the code does when the convention is absent. A crash gets noticed. A null that flows on into the next stage doesn't, and I rank those above the loud ones. The last question is which test corpora satisfied the conventions at once. In Cairn the answer was Project P and Project I, the two projects the validations ran on.

The walker's .md convention held in 25 of the 70 projects. Git repos were 69 of 70 and 400 of 400 sampled files were UTF-8. Neither needed a fix. The migration bootstrap expected an elaborations/ directory and a map file, and read only .md under the directory. 37 projects have the directory and 36 have the file. On a project with the layout and extensionless elaborations it migrated nothing and said nothing. The rewrite walks all content through walkContent minus a structural denylist, and still reads the map file for summaries. The linter's proper-noun regex was [A-Z][a-z]+, which never matches an accented letter, so every accented name in a period novel reached the confabulation linter either not at all or cut off at the accent. The regex is now Unicode-aware, \p{Lu}[\p{Ll}\p{M}]+ with explicit non-letter boundaries, and I verified it on the real repo where accented names had been invisible before the fix.

The design doc's sentence on the two validated projects is that they satisfy the walker, anchor, bootstrap and frontmatter conventions simultaneously, the maximally-Cairn-shaped corner of the corpus and not a representative sample. The commit-subject convention the gate's temporal anchor keyed off held in 3 of 69 git repos. And the convention that authors write YAML frontmatter and canon tags held in 0 unmodified repos. The only two that have it are the two I migrated.

Silent degradation to null with no crash is the defect class Cairn exists to catch in a model's memory, and the audit found it in four places in Cairn's own code. The walker skipped 1,117 extensionless files. The temporal lookup returned null for every file on 66 of 69 repos, the regex dropped or truncated every accented name, and the bootstrap migrated nothing from an elaborations/ directory of extensionless files. None of the four threw.

The Temporal Anchor Resolved No Canon on 66 of 69 Repos

The .md filter made 45 projects invisible, and the commit-subject convention still ranked above it. The gate checks a model's new claim against canon as it stood in the session that produced the claim. The reason comes from the six confabulations the project was designed from. Two of the three Class C cases, where the model distorts a fact that is correctly in canon, only read as contradictions against canon that was settled later. As of the session that produced them the fact was still undecided, which makes them Class A, an invented absent specific, and a gate fed the latest canon labels them Class C. The anchor, before the fix, took the session number when the subject had one and otherwise the first word of the subject:

function sessionTag(subject: string): string {  const m = subject.match(/^Session (\d+)/);  return m ? `S${m[1]}` : subject.split(/\s+/)[0] ?? "";}

That regex was the only way provenance() and bodyAsOfSession() could locate a point in a file's history. On the 3 repos whose subjects start Session N it worked. The other 66 of 69 use the subject format of my writing tooling, usually with no session number, so provenance came back as the first word of a commit subject, and bodyAsOfSession() found no commit to stop at and returned null. When HEAD carried no session tag the CLI set genSession to MAX_SAFE_INTEGER, which on a session repo means the latest session. On the other 66 it meant that no canon resolved for any file: the span collector skipped every file, the fallback threw, the audit labeled every claim NO-CANON-AS-OF, and the judge never ran. The design document had warned that a gate fed the latest canon mislabels Class A as Class C. On 66 of 69 repos, about 96% of them, the gate had no canon to label with at all, and nothing threw.

The fix I wrote is a general mechanism and the regex is unchanged. sessionTag now returns null when the convention is absent, and provenance takes the session tag when it is present and the commit author-date when it isn't, which every git repo has:

const tag = (r: { date: string; subject: string }) =>  sessionTag(r.subject) ?? r.date;return {  firstSession: tag(rows[rows.length - 1]!),  lastSession: tag(rows[0]!)};

And the as-of point became a commit-ish: bodyAsOf(repo, relPath, at), where at is either { commit } or { session }. The session path stays, so the Project P proof still stands. The CLI gained --as-of-commit, and the MAX_SAFE_INTEGER fallback was replaced by an anchor that is a real commit: the diff base when the audit has one, else HEAD. A new regression test runs the temporal path on a repo with no session subjects.

The post-fix probe then corrected my picture of the corner. The session subjects mark the wrap of a session, and most commits inside a session don't carry one. The probe sampled one file each from eight repos without the convention and two with it; even the session-convention repos mostly yielded dates per file. The date fallback is the path most files take even on those 3, and per-file provenance now resolves to a real point on all 69.

Auditing the Fix Found Two More Defects, Then Project G Found Three

I sent the read-only probe across the 70 projects a second time and looked at what the new walker admits. It admitted 1,865 files, and 114 of them were over 200 KB. One was a 2.8 MB raw export of an LLM session with no extension. The old walker had never admitted it, since the name had no .md. The new walker needed a size cap, which the old one never had. I set the default at 1 MB, configurable as [walk] maxBytes (a 250,000-word novel as plain text is about 1.5 MB). That took the admitted count from 1,865 to 1,841, with 24 files over 1 MB out.

The probe also showed that [walk] exclude denied files and didn't deny directories, because the basename check ran after the recursion branch. It now runs before it, and on the one project where I set an exclude by hand for a raw-export directory, 53 fewer files were admitted. (The 53 comes from that one project's hand-set exclude, not from a default.)

Then I ran the whole pipeline on Project G, which the record puts off the corner on every axis. It has no Session N subjects, and 128 of its 144 content files are extensionless. Its canon lives in a canon/<topic> directory with a parallel elaborations/elaboration-<topic> tree, and its dead material sits under a superseded/ subtree. The pre-fix tool would have seen 15 of its files and resolved no canon for the gate. The fixed bootstrap migrated 139 files, additively (+1,112 lines, -0). The resident surface went from about 9,925 tokens to about 3,445, a 66% cut measured summary-only. And bodyAsOf({ commit }) resolved 188 KB of a body that had returned null on this repo under the old anchor.

Project G also surfaced three defects on first contact that the corner had never shown. The first was a join import the bootstrap rewrite had dropped, which the first invocation caught. The second was the parallel layout, which produces the same entity name under canon/ and under elaborations/. The bootstrap collapsed both to one name, so buildIndex threw on a duplicate live entity. I added a two-pass qualifier that suffixes identical live names with their parent directory, and left a collision between a live file and a superseded one alone. Only duplicate live names need the qualifier; a live file and its superseded predecessor keep one name.

The third was a content error. The bootstrap had been migrating files under superseded/ as status: live, which resurrects dead canon as current. Status is now path-driven: any superseded/ component up a file's path makes it superseded whatever its frontmatter says.

The audit over G's 139-file diff also wouldn't finish. collectSpans re-walked the repo and re-ran git show inside resolveBody, which runs once per claim. The cost was claims times files times a git call, and I killed the run at 15 s of CPU. runAudit now builds the spans map once before the worker pool and threads it through to resolveBody. Each claim's span lookup is now O(1). I re-ran the same workload, 143 claims over 140 diff hunks. Anchored at a bare commit SHA, it finished in 8.9 s real and 4 s user.

What the Fixes Do and Do Not Prove

I wrote a correction into the record against the earlier claims. Every "proven on real data" result had been measured on the corner. Each one was real there and unproven-to-silently-broken outside it, and none could be cited as corpus-general. The fixes remove the silent-failure mechanism. The record says they do not make the corner-only validations corpus-general, which takes a re-run on non-corner projects, and Project G was the first.

The three surface cuts stand with their conditions attached: Project P's 80% on the full dogfood with 66 entities, and Project I's 90% as a summary-only projection. Project G's 66% is the smallest of the three. It is summary-only like Project I's and off-corner on every axis: no Session N subjects and 128 of 144 files extensionless. After G the surface cut and recall are no longer corner-only. Audit throughput at scale is still measured on one project.

I keep "the mechanism fires" and "the writer benefits" as separate claims with separate evidence. The reproductions show the gate firing correctly on P, and the commit-anchored temporal path resolving real canon end to end on G, which supports the first one. Whether a model mid-session heeds the signal instead of confabulating anyway is a behavioral question, and the only evidence for it is real writing sessions over time, of which there are none yet.

Limits

One author's corpus, 70 projects, all in one structural family. The prevalence counts say nothing about anyone else's corpus. The 45 of 70 and the 66 of 69 are counts of projects where a convention is absent, and I haven't measured downstream harm on any of them beyond Project G. The audit was run once, by me, over code I wrote, and the list of assumptions came from my grep. An assumption I didn't think to grep for isn't in the table. The 33-file gap between the two walkers is unexplained in the record.

The frontmatter assumption is that writers will annotate canon in a frontmatter and tag convention that, as of the audit, 0 unmodified repos use. Two of the four fixes, the walker and the bootstrap, lower its cost without touching it, and until a writer adopts it the count stays at 0 unmodified repos.

The audit found four silent failures in Cairn's own code with one count per convention over the 70 projects: 25 keep the walker's .md, 37 the bootstrap's elaborations/ directory, and 3 of 69 repos the anchor's Session N subjects.

Appendix: The Six Confabulations and the Ceiling

The design started from a paper trace of six real confabulations in one project's session summary. Class A is an invented absent specific, a value for a fact the canon leaves undecided. In Class B the model asserts a false predicate between two entities that both resolve in the index. Class C is distortion of live canon: the fact is in canon and correct, and the model overstates it or invents a mechanic that contradicts it. On the traced six, the structure alone addressed 2 and the first linter netted 1 more. I wrote that 3 of 6 into the doc as the ceiling the confabulation claim must not exceed before the gate existed.

Class C is why the gate exists and why the temporal anchor is the worst row in the audit. The model's input was correct. The check runs after generation. It takes the claim, the span of canon it cites, and a judge asked whether the span entails or contradicts the claim as of its generating session.

I closed that loop first on the three Class C cases, using condensed canonical spans and a judge model from a different family than the one that wrote the prose. There was no answer key. All 3 came back CONTRADICTS with the correct cited spans, at about 0.97 confidence. That is n=3 with every label the same. The run also surfaced a contract bug in my judge prompt. The prompt didn't pin the JSON schema. The judge's evidence came back under its own key names and the coercion silently dropped it, until I made the coercion judge-agnostic.

The same three claims then ran against the actual committed bodies, resolved through the index and the temporal fetch as of their session and sent to the real judge: about 23,500 tokens of layered canon that held both the confabulation and its later correction. The judge cited the correction instead of the confabulation sitting beside it. It was still 3 of 3 with every label the same. The floor with a cheap judge is unmeasured.