Calibrating an LLM Judge

The judge in my Mandarin flashcard pipeline is a model call, an LLM judge. It reads a generated card against the card's contract: the tested word and its sense, the distractor the card must rule out, the vocabulary the card may use. It returns ok or not ok, a naturalness score from 1 to 5, five boolean checks and a list of findings. The first bank was judged by Haiku, and 755 cards passed. I re-screened all 755 through a Sonnet judge with a tightened rubric. 263 flipped to reject, which is 34.8%. I then pulled the Haiku judge's own verdicts on those 263 cards. It had scored naturalness 5 out of 5 on 108 of them and 4 on 153, and on 136 it had passed the card with an empty findings list. The test that separates the two judges is four cards with known defects, run blind through each candidate: the Haiku judge had passed all four when it gated the bank, and the Sonnet judge failed all four.

The Haiku judge couldn't see whole classes of defect. It had passed all 263 cards the Sonnet judge later rejected, and scored 261 of them 4 or 5. When it rejected a draft for something else, its findings went back to the generator as the retry's feedback, with those classes missing. Rewriting the rubric to quote the four defects the closing audit had found still left it passing two of the four. So I built a calibration set of four cards with known, named defects and ran it blind through each candidate judge. The four had gone into the bank on the Haiku judge's pass, and each of the Sonnet judge's findings on them quotes the defective fragment.

The scope is one app and one pipeline: Haiku drafts, Haiku judges, Haiku retries on the Haiku judge's feedback, and Sonnet drafts the escalation stage. It includes three audits of the same bank at three different bars, and the procedure that came out of them. The cards are A1 Mandarin.

The re-screen also cost coverage. Before it, 461 of the 492 focusable words had at least one clean card (93.7%). After it, 373 did (75.8%). I imported the 263 flips anyway.

Haiku-Drafted Cards Flipped at 57%, Sonnet-Drafted at 12%

The bank was built in three stages, and every clean card came from one of them. Pass1 is Haiku drafting a card from its contract and Haiku judging it. A card that fails goes to retry, where Haiku redrafts it with the rejection's findings as feedback and Haiku judges again. A card that fails retry goes to escalation, where Sonnet drafts with the retry verdict's findings as feedback and Haiku judges once more. Haiku was the judge at all three stages, so escalation moved only the drafting to Sonnet. The 755 clean cards were 222 from pass1, 156 from retry and 377 from escalation.

Cut the 263 flips by stage. Pass1 cards flipped at 126 of 222 (57%). Retry cards flipped at 91 of 156 (58%). Escalation cards flipped at 46 of 377 (12%). Put the two Haiku-drafted stages together and 217 of 378 flipped (57%), against 46 of 377 (12%) for the Sonnet-drafted stage. By card kind, recognition cards flipped at 248 of 691 (36%) and cloze cards at 15 of 64 (23%), as I counted them at the time. (The live bank now gives 249 of 698 and 14 of 57 for the same cut, because the later regeneration rounds rewrote the kind column on the rows they replaced.)

The Sonnet findings named four characteristic classes of defect. Each repeats across cards rather than turning up at random. The first is a post-verbal 在-phrase, as in 我拿了面包在商场, where the location 在商场 hangs after the verb. The second is a bare-noun location complement, as in 住在北京的路. The third is a context that never rules the distractor out. The fourth is a distractor "why" text that is empty or says something false.

Card anki-1-40:2 tests 唱 (sing) against the distractor 商场 (shopping mall), which sounds like it. Its contract lists the vocabulary the card may use, and 歌 (song) isn't on the list. Haiku's pass1 draft used 歌, and the mechanical gate rejected it for a hanzi outside the contract vocabulary. The retry draft used 歌 again, and the gate rejected it again. Sonnet's escalation draft stayed inside the vocabulary by writing 他的爱好是唱, his hobby is sing, with the object dropped. The Haiku judge passed that card with naturalness 5, every check true and no findings. This is its verdict as recorded in the card's stage history in the bank (one line in the database, wrapped here):

{"stage":"escalation","mech":[], "judge":{"ok":true,"naturalness":5,  "checks":{"grammar_ok":true,"discriminates":true,   "distractors_realize_edges":true,"vocab_within_contract":true,   "readings_correct":true},  "findings":[]}}

The Retry Stage Flipped at 58%, the Same as the First Draft

Retry cards flipped at 58% and pass1 cards at 57%. The retry stage exists to fix the first draft's findings. The judge's rejections carried findings in the classes it could see: a word outside the contract's vocabulary, a context that didn't force the tested sense, a weak distractor. Haiku redrafted around those. The second verdict came from the same judge. It had waved a defect in a class it couldn't see through on the draft, and the redraft, written around the findings it could see, carried that defect into a second pass. Of the 156 retry cards that passed, 76 had been redrafted from a pass1 draft the mechanical gate rejected, with no judge call, 78 from one the Haiku judge rejected, and 2 from a first draft that came back as unparseable JSON and never reached either.

Card anki-1-100:1 tests 饭店 (restaurant) against the distractor 饭 (rice, meal). The pass1 draft was rejected with naturalness 3 and three findings. 饭 appeared in a distractor option but wasn't in the allowed vocabulary. The sentence didn't force the learner to tell 饭店 from 饭. The distractor's "why" text was weak. The retry draft fixed the vocabulary and the discrimination and got naturalness 4 with two P2 findings. One of them began "The sentence is grammatically sound and natural, but slightly awkward in combining two independent clauses". The second clause was 我们帮忙他. 帮忙 is a verb-object compound and separable, so it can't take 他 after the whole verb, and the sentence needs 我们帮他. The judge passed the card with that finding attached at P2.

Three Audits at Three Bars Gave 20%, 34.8% and 65%

Three audits were run on this bank, and they gave 20%, 34.8% and 65%. The three aren't estimates of one number. Each had its own auditor, its own bar and its own sample: every 37th clean card by id, all 755 clean cards, and 20 random cards from the 492 survivors.

The first was the closing audit of the build task. A different model from the judge reviewed every 37th clean card by id, 20 cards in all. It disagreed with the Haiku judge on 4 of the 20 (20%), all in the accepting direction, and on none of the discrimination, containment or distractor checks. Of the 263 cards the re-screen later flipped, 119 were in the discrimination class and 84 in the distractor-realization class, the two checks on which this audit had recorded no disagreement. The four cards it disagreed on are the four in the calibration set.

The second was the re-screen, with Sonnet judging all 755 clean cards against the v2 rubric. The rubric sets naturalness at a native-speaker bar. 263 cards flipped (34.8%). A flip is a disagreement between two judges on one card, where the second judge rejected what the first had passed, and by itself it doesn't say whether a card is defective.

The third was a blind cross-model audit after the re-screen. Opus, a different model from both judges, reviewed 20 random cards from the 492 that had survived. It used the auditor prompt in scripts/kc-rescreen.mjs, which checks naturalness, discrimination, distractor realization, vocabulary and readings. It disagreed on 13 of 20 (65%), against an acceptance bar of 20% or below. Split by who drafted the card, 10 of the 11 Haiku-drafted cards drew a disagreement and 3 of the 9 Sonnet-drafted cards did. 7 of the 10 Haiku disagreements were P1 defects. One P1 was card anki-1-479:2, whose sentence 我爱好中文 was meant to force the reading 中文 (the Chinese language) over the distractor 中国 (China). The auditor's finding was that 我爱好中国 reads fine too, so the context rules nothing out, and the card's own "why" text, which claims it does, is false.

The 65% is the number that moved me to regenerate every Haiku-drafted card, and it rests on 20 cards. The 34.8% counts disagreements between two judges across 755 cards. In the figure above, each dot's area is drawn to its n.

Rewriting the Rubric Did Not Change the Judge

The task that produced the re-screen started from the four audit cards. Its first move was to tighten the rubric text and keep the judge. The judge prompt's rubric went from v1 to v2, and v2 names each defect with its fragment quoted. Rule 1a says a separable verb must not take a direct object after the whole verb, and quotes 帮忙他 as wrong with 帮他 or 帮他的忙 as the correction. Under rule 1b a predicate must fit the subject's semantic type, and the rule quotes that a hobby cannot 很好听. Rule 1c says a transitive verb must carry its object where A1 usage expects one, and quotes that 爱好是唱 needs 歌. Rule 1d allows one sentence, one clause structure and one question. A unit test pins the native-speaker bar, the four pattern names and their example fragments as strings, so an edit that drops one fails the test suite.

The v2 rubric then went to the Haiku judge, the same model that had gated the bank. I ran it three separate times over the exported batches of 50 cards. Each run came back templated or off the rubric: the same naturalness score given to all 50 cards in a batch, cards rejected for criteria the rubric doesn't contain, such as missing final punctuation and the standard separators in pinyin, and ok-false verdicts with no finding attached. Across the three runs the Haiku judge passed two of the four cards whose defects the rubric quoted verbatim. I moved the judge to Sonnet. Its first run over the same batches failed all four cited cards with findings quoting the defective fragments, and that run became the re-screen.

The figure shows the verdicts that put each of the four cards in the bank, under the v1 rubric, beside the Sonnet verdicts from the re-screen. The Haiku judge had passed all four with naturalness 4, 4, 5 and 4, the grammar check true on each, and 2, 1, 0 and 1 findings. Three of the four were Haiku retry drafts and one was Sonnet's escalation draft.

The objection I'd raise myself is that a rubric is a prompt, prompts move judge behavior a long way, and a better-written Haiku prompt might have caught the four. The v2 rubric was the strongest form of that prompt I could write: it quoted each card's defective fragment with the correction beside it, and the Haiku judge still passed two of the four under it.

Calibrate the Judge on Labeled Defects

First, collect a small set of cards with known, named defects. Here it was the four cards the closing audit flagged, each with its defect class written down. Second, run every candidate judge over them blind, with the four mixed into ordinary batches. A judge that passes a labeled defect is disqualified for that class. Third, set the judge stage's minimum model at the weakest model that passes the set, which here is Sonnet. A model below that line can still draft under the stronger judge. Here I stopped drafting with Haiku as well, since 217 of its 378 surviving drafts had failed the Sonnet judge.

Fourth, keep the labeled cards and their contracts as fixtures, and rerun them through the script on every judge-model change. Fifth, report flips as flips, and estimate prevalence from a different model's audit of a random sample, with n stated beside it.

The script is scripts/kc-rescreen-quota.mjs. Its export mode writes the judge prompt for every clean row not yet judged at the current prompt version into batch files of 50. The judging runs outside the script. The import mode reads the verdict files, validates each verdict's shape and refuses duplicates. If any row's verdict is missing it writes nothing, because a partial import would stall the ordered walk at the first hole. It writes through one function, rescreenBank. That function may change the clean flag, the judge result, the judge model, the prompt version and the stage history. It may not touch the card text, the contract, the tested word, the mechanical-gate result or the created-at stamp.

After the walk it reads the four cited cards' clean flags straight from the database and writes them into the manifest. This is the manifest the Sonnet run wrote, with the 263 flipped ids, the billing-rail line and the spend field cut:

{ "generator": "scripts/kc-rescreen-quota.mjs --import", "promptVersion": "cardgen-v2", "judgeModel": "claude-sonnet-5 (quota-subagent)", "rescreened": 755, "total": 755, "flippedCleanToReject": 263, "keptClean": 492, "citedCards": {  "card:zh:ws:anki-1-100:1": 0,  "card:zh:ws:anki-1-118:2": 0,  "card:zh:ws:anki-1-40:2": 0,  "card:zh:ws:anki-1-474:2": 0 }, "distinctCleanTestedKcAfter": 373, "stoppedEarly": null, "at": "2026-08-24T15:29:53.643Z"}

The citedCards block is the calibration check. A manifest with a 1 in that block means the candidate passed a labeled defect and is disqualified for that class. The block reads the four ids' current clean flags, and the export walks only rows not yet judged at the current prompt version. So a later judge-model change reruns the walk under a new prompt version, and the four original cards go in as fixtures, since the regeneration has since replaced three of the four slots with new cards.

With the judge settled, I regenerated all 378 Haiku-drafted rows with Sonnet drafting and Opus judging. 272 of 378 passed the first round. The 106 rejects went through a second round, with Opus drafting on the judge's feedback, and 89 passed. Later rounds, together with a revision of the contracts the judges' findings kept pointing at (same-gloss distractor pairs, near-synonym cloze families with no forcing frame, wrong tested-word glosses, and inventories missing the word a natural sentence needs), filled the coverage gaps and gave every word a second card and 74 words a third. At the close of the integration milestone that followed, the bank held 1,058 clean cards covering all 492 words, and 0 of the clean rows were Haiku-drafted.

The last audit on the way to that count ran on the bank at 920 clean cards. A model different from both judges reviewed 40 random cards. Its three findings all traced to the card's contract rather than its text, and that was a disagreement on 3 of 40 (7.5%).

Limits

This is one task, one language and one model family's ladder of three sizes. I did not run another task or another family.

Neither the 65% nor the 34.8% is a measured defect prevalence for the bank. The 65% is 20 cards, and the 34.8% is a flip rate between two judges on 755.

The re-screen ran through the separate script above and the regeneration through its sibling, scripts/kc-regen-quota.mjs. The Sonnet minimum for the original pipeline script is recorded in the task's backlog: at the current head of the repository, scripts/kc-cardgen.mjs still names the Haiku model at the re-screen log line, at the judge call, in the re-screen's judge-model option and in the manifest's judge field (lines 172, 176, 180 and 191).

A second bank of 21,544 cards across 18 languages, built by a sibling of this pipeline with Haiku drafting and Haiku judging, has not been re-screened.

Which two of the four cited cards the Haiku judge passed under the v2 rubric can't be recovered: the per-card verdict files from those runs were working files and weren't kept. The figure shows its v1 verdicts, which are in the bank.

If your pipeline has one model drafting and another model judging, the first number to get is the judge's verdict on a handful of items you already know are wrong. Mine had scored one of its four 5 out of 5 with an empty findings list.