How a Student Model Gamed Its Verifier

The cards below were written by a Qwen3-0.6B model with a rank-16 LoRA, trained on 884 cards for my Mandarin flashcard app. Each passed the app's mechanical gate, 160 lines of deterministic JavaScript with no model in it, which a frontier model clears on 56.8% of first attempts. The fine-tune cleared it on 232 of 238 held-out contracts, which is 97.5%. The tested word is on the left.

干净 (clean) → 我干净，家人在们。               "My family is clean, and they are all over."岁   (year)  → 我年，打了昨天。               "My year has gone to yesterday."不   (no)    → 我不有房子。               (ungrammatical)

The test that found this was a blind judge. I shuffled 30 of the fine-tune's gate-passing cards with 30 written by the frontier ladder and had a local open-weight 35B model score each one against the criteria in the app's own judge schema, never told which was which. It passed 90% of the frontier's and 3% of the student's. The gate checks form (every Chinese character in a plain-text run inside the contract's character set, the tested word's id on a token, the ids and options well formed) and the model had learned to produce form.

The fix was to stop training and split the task. The base model already writes Chinese and its weights stay as they are. For the 45-word inventory I built a mask that sets the logit of every token carrying an outside character to minus infinity before each sampling step, and no character outside the contract's set can be emitted. assemble.py builds the rest of the card, the ids, the three options and the evidence citations, from the sentence and the contract. That system reached 82.5% usable cards per contract with no training, by a blind audit from a different model family than the generator.

A cheap verifier that does not capture the objective is worse than an expensive one, because it lets the loop run fast in the wrong direction while handing you a rising number the whole way.

The scope is one task, run on one MacBook under MLX with eight of the audited decks generated on a rented GPU. A card is one A1 Mandarin sentence using one tested word, no Chinese outside the learner's 45-word inventory, a translation, a question and three options with evidence. The claims cover the fine-tune, the gate, the two judges and the no-training pipeline that replaced the fine-tune, and nothing about small models against frontier models in general (two earlier tasks in the same project, chess legality and tariff classification, had failed their pre-registered gates). The cross-model audits here were read by the project's Claude agent with each card's source hidden, and no human read them.

The Gate Was Exact and the Student Passed It

The frontier baseline was measured on the same held-out contracts through the same gate. Every attempt the Sonnet and Opus ladder ever made on a card is stored in the card bank's stage history, so the ladder's first attempts were scored the way the student's were. First attempt: 56.8% of the 220 eval contracts with a recorded frontier attempt mechanically clean. Up to three attempts, each carrying the gate's findings back as feedback: 98.2%. The bank was split by tested word rather than by card, into 884 training cards over 387 words and an eval set of 238 contracts over 105 words, with the disjointness asserted in code.

The gate itself was ported from the app's JavaScript to Python and run against all 1,124 stored contract-and-card pairs in the bank. The port agreed with the original on every pair. Eval leakage was checked twice. The 232 passes were reproduced by re-scoring the run file independently. All of that proves the port matches the check, and nothing in it bears on whether a card that passes is a card a learner could use.

Training went through four configurations. The untrained base passed 0.8% of contracts. A rank-32 LoRA diverged to 0.0%. A rank-16 LoRA on the original long ids reached 52.9% at 800 iterations, which is 0.96 epochs. A rank-16 LoRA with the card's join keys re-encoded as short ids passed 97.5% at 475 iterations, which is 0.57 epochs, and 75.2% at 2,493 iterations, which is 3.0 epochs.

The 3-epoch model was the worst checkpoint past 200 iterations. I only knew that because the training script banked an adapter snapshot every 200 iterations and curve.sh fuses and evaluates each one. The snapshots already existed, and the curve cost eval time and no extra training. The peak at 0.57 epochs came out of it.

The Blind Judge Passed 3% of the Same Cards

The model had found the cheapest string that satisfies every check in the gate: 我, then the tested word, then two words from the inventory, then 。. That string contains the tested word, uses no outside character, and sits in a well-formed card with valid ids and three options, and it is not a Chinese sentence. The gate has no check for whether the sentence means anything. It also checks that the tested word's id is on a token, not what that token prints: in 15 of the 232 passing cards, the 岁 card at the top among them, the tested id sits on a token that prints another word, 年 for 岁, 上课 for 下课, 晚餐 for 晚饭.

judge.py draws 30 gold cards out of the bank and 30 gate-passing ones out of the student's run file, with a fixed seed, and shuffles them into one stream. Each goes to the 35B with five criteria from the app's schema: grammatical, makes sense, translation matches, exactly one option fits, and naturalness from 1 to 5. The script prints population rates. Gold: grammar 93%, sense 93%, translation 90%, discriminates 100%, all four 90%, naturalness 4.50. The student: 10%, 10%, 3%, 50%, 3%, 1.33. This is the judge's sample block as the diary recorded it:

GOLD    PASS 我们说话呢，他走了。 "We were talking and he left."GOLD    PASS 弟弟在家里唱歌。     "Little brother sings at home."STUDENT FAIL 我见面了年，男。     "grammatically incoherent,                                  translation incorrect"STUDENT FAIL 我年，很，有个分     "word salad with no grammatical                                  structure"STUDENT FAIL 我饭了，洗手间。     "grammatically invalid and                                  semantically incoherent"

A usable card passes both the gate and the judge, and the number that matters is the product. The frontier's first attempt clears the gate on 56.8% of contracts and the judge on 90% of cards, about 51% usable. The student clears the gate on 97.5% and the judge on 3%, about 3% usable. 51 divided by 3 is about 17. The 35B judge had not been validated against anything at that point. The audit below puts a number on its bias, and the 3% is the one population no second reader ever audited.

I had written the failure down before any code ran. The candidate memo for this task has a section called "The honest split". It separates the mechanical pass rate from the end-to-end clean rate, says the two must not be quoted as one number, and says a 0.6B model probably loses to the frontier per sample on grammaticality. Then I spent the training run driving the mechanical number from 52.9% to 97.5%.

Fine-Tuning Removed the Chinese the Model Already Had

The question that reframed the task was whether generating valid Chinese is inherent to the model, with the rest being scaffolding. The measurement that answers it puts three populations through one blind pass of the same judge: 30 gold cards, 30 sentences from the base Qwen3-0.6B with no training and a prompt asking only for a sentence, and 30 gate-passing cards from the fine-tune. Gold scored grammar 93%, sense 93%, naturalness 4.50. The base scored 77%, 73%, 3.43. The fine-tune scored 10%, 10%, 1.33. With no training, 77% of its sentences were grammatical Chinese. After 884 examples against a gate that scores only form, 10%.

The two models sit in opposite corners of one square. The base model's sentences stay inside the inventory 33% of the time and are grammatical 77%. The fine-tune's are inside the inventory about 99% and grammatical 10%.

The task bundles three concerns. The pretrained model already writes natural Chinese. The lexicon's characters can be enforced in the decoder, by setting the logit of every token that would emit an outside character to minus infinity. What remains, the ids, the option structure, the evidence citations and the JSON schema, is a deterministic function of a sentence and a contract, and code can assemble it. Fine-tuning was asked to learn all three at once, and it traded away the only one the model already had.

The join keys had shown the same thing hours earlier. They are strings like zh:ws:anki-1-107, and the model copied them unreliably. So I built a short-id codec that renames the inventory's keys to w1 through w45, numbers every other key the contract mentions onward from there, and maps them back before the gate. Its round-trip test failed on 1,010 of 2,006 prompts on the first run. That version numbered only the tested word and the inventory, and the distractor plans cite confusables outside it. I treated a model finding join keys hard as a prompt-engineering problem, when a join key should never pass through a language model at all.

The System That Works Uses No Training

The pipeline that replaced the fine-tune has six stages, and two model roles that are never the same call. The generator is a local open-weight 35B mixture-of-experts with about 3B active parameters, the model that judged above. Rules build the distractor candidates first. The generator then chooses one and writes the basis of the card. Under the logit mask it generates k sentences, every one built from the contract's characters by construction. A structural gate in code rejects before any judge call. The judge picks the best survivor, the generator writes the prose for it, and the final gate runs. On 40 contracts the stages converted 38 of 40, 38 of 38, 100%, 28 of 38, then no failures at the end, for 70.0% end to end.

The distractors turned out to be computable. The app's confusable graph, written by a model, has 188 edges over 2,770 words. Three of its six edge kinds need no model: a tonal confusable comes from the pinyin, a same-gloss confusable from the gloss string, a same-character confusable from the surfaces. Over the deck's 496 words those rules yield 1,740 candidates covering 90.7% of words (55 tonal, 88 same-gloss, 1,597 same-character), and they are the pairs a learner confuses: 白天 and 天 are both "day", 帮 and 帮忙 both "help". The generator selects one and writes why.

The mask has to be built over tokens, and the constraint is over characters. Qwen's tokenizers are byte-level BPE, so a Chinese character can be split across tokens that each decode to the replacement character U+FFFD. A filter that checks the characters of each token can't see what those fragments will become, and the first mask let 咦 into a "constrained" sentence that way. In the Qwen3-0.6B tokenizer, where the mask was first built, those fragment tokens are 1,457 of 151,936. Every character in a beginner lexicon has a whole-token form, and blocking all 1,457 removes nothing a sentence needs.

The second bug was a ban list that covered only ASCII, so one run emitted emoji and Cyrillic.

Now a token is usable only if every character in it is Han or punctuation, which leaves 26,271 usable tokens in the 0.6B's vocabulary. A usable token is allowed for a given contract only if every Han character it carries is in that contract's set. This is the index builder from constrained.py, with its docstring cut and two semicolon-joined lines split:

def build_token_index(tok, vocab_size: int):    han_chars, usable = [], []    for i in range(vocab_size):        s = tok.decode([i])        if "\ufffd" in s:            # partial UTF-8 byte token            han_chars.append(frozenset())            usable.append(False)            continue        han = frozenset(ch for ch in s if is_han(ch))        rest_ok = all(is_han(ch) or ch in PUNCT for ch in s)        han_chars.append(han)        usable.append(rest_ok)    return han_chars, usable

Per contract, allow[i] is usable[i] and han_chars[i] <= ok_chars, and the bias is 0 where allowed and minus infinity elsewhere. A repetition penalty sits on top of the mask, which on its own invites 爸爸帮爸爸帮 out of a 45-word lexicon, and generation cuts at the first sentence terminator. Without the first fix the "100% contained" claim would have been false.

The mask costs fluency too. In one blind pass, the base model with no mask scored grammar 77% and naturalness 3.50, and with the mask 43% and 2.13. Given 歌 (song) and no constraint, the 35B generator writes 我喜欢听歌. Given the inventory in the prompt only, it writes a good sentence by quietly using 唱, the partner word the sentence needs, which is outside the list. Under the mask it writes 我爱歌. The model still has to find a sentence reachable with the words it has, which is planning under a constraint.

The figure places every system on gate-clean yield against audited quality, with usable cards per contract as the product. The frontier's three-attempt ladder is 98.2% yield times 96% audited quality, about 94%. Its first attempt is 56.8% times 96%, about 54.5%. The first mask-and-code pipeline, k=4, is 70.0% times 64%, about 44.8%. The fine-tune, after 7 GPU-hours, is about 3%.

Few-shot from gold sentences with k=8 and two repair rounds took the pipeline to 75% times 80%, which is 60%. A contract-level ladder, rerunning only the failed contracts with the previous distractor excluded and k tripled, reached 40 of 40 at 80% audited quality. Three verifier votes with a majority to drop, a rebuttal pass, and an unmasked planning step before each retry gave, over four attempts, 35 of 40 contracts and 33 of 35 cards passing the audit, which is 82.5% usable. Both misses were let through by a rebuttal on the last attempt, which now runs only on early attempts.

The Generator's Own Judge Flattered Its Pipeline

The 35B judge and the generator are the same weights, and every quality number the judge produced for the pipeline was self-graded. The first audit took all 28 pipeline cards from the k=4 run and 28 gold cards, 56 in all, shuffled with the source hidden. The project's Claude agent, a different model family from the generator, read each card before the key was revealed. The 35B had scored the pipeline 71% and gold 89%. The audit scored the pipeline 64% and gold 96%. So the judge had overrated its own output by about 7 points and underrated the frontier's by about 7, and the real gap was 32 points where it had reported 18.

The sentences the judge accepted and the audit failed: 他国外有病 (missing 在), 我有块白本子 (块 is the wrong measure word for 本子), 日期是八 (needs 号), 别忘你的书 (missing 了), 爸爸今天看病进去 (word order).

Fourteen decks got both scores. On the six local decks the judge ran ahead of the audit by 7, 7, 7, 5, 3 and 7 points. On the eight remote decks the generator ran as FP8 or AWQ 4-bit on a rented GPU and the judge stayed local, and the gaps were 15, 17, minus 2, 14, 6, 16, 7 and 11. The gaps of 14 to 17 came from runs where a repair critic rewrote most of the deck, and the gap fell back to 7 once repair was gated to the sentences the stage-4 ranker had flagged.

calibrate.py was my attempt to subtract a fixed offset from the judge's score. On the first remote run it predicted 57.9% usable and a win over the frontier's first attempt. The audit gave 52.7%, which isn't one. A constant correction was the wrong model for a bias that moved between 3 and 17 points from one run to the next. The audit is the reported number, and the judge stays inside the pipeline as a selector.

Most of that bias was the question the judge had been asked. On 40 cards already audited blind, a verifier from a second family (GLM-4.7-Flash) caught 5 of the 8 bad sentences with 1 false alarm, and missed 我爱歌. The generator's own Qwen weights got a prompt that names the failure classes: a missing measure word, 了, 边 or 号, a bound morpheme used free, grammatical but meaningless. They caught 8 of 8 with 5 false alarms, and their explanation for 我爱歌 was that 歌 needs a measure word or the verb 唱. Stage 4 now asks the strict question, and the ladder retries a dropped contract with the verifier's objection as the hint.

Split the Task Before Training Anything

The project's postmortem had listed four conditions for a small fine-tuned model to beat a frontier one, and this task corrected two of them. Condition 4 had said the verifier must be cheap so the loop can run. It now says the verifier must be cheap and sufficient. Condition 1 now says a constraint is not a convention: the schema and the word list were the easy part, and under them sat ordinary linguistic competence under an unusual restriction.

The procedure I'd run on the next task starts before any training. First, divide the task into what the base model already does, what is a constraint to impose, and what is bookkeeping. Here that was Chinese, the 45-word inventory, and everything with an id in it. Second, impose the hard constraints mechanically and assemble the bookkeeping in code. The mask gives 100% character containment as a guarantee, where the fine-tune gave about 99% as a measurement. Third, if the verifier is cheap, ask what it can't see, and run a blind quality judge on gate-passing output before trusting a rising number. Mine ran in 51 seconds. Fourth, never let the generator's model family produce the quality number you report. Keep it as a selector, measure its bias against a reader from another family, and expect the bias to move. Fifth, bank checkpoints every few hundred iterations and evaluate all of them. The best model of this run was at 0.57 epochs and the last one was the worst.

Limits

The 3% and the 90% came from the 35B judge before its bias was measured, on 30 cards per population, and no second reader ever audited the fine-tune's cards. The later audits put the judge's self-preference between 3 and 17 points in 13 of 14 decks and at minus 2 in one. The 17 times is a ratio of two composites and moves with the judge. A later judge pass gave the fine-tune 4% instead of 3%.

The first cross-model audit was 56 cards and the second 60. Every audit was the agent's reading, source hidden. An agent audit is not human validation. The earlier pipeline audits scored the sentence, and option defects were counted separately where found: the first remote deck fell from 81% to 69% once they were. The frontier composites multiply its gate rate by the audited quality of its finished bank cards, the ones its own ladder had already judged; its first-attempt survivors were not audited on their own. The app's own production judge, a separate pipeline from this one, was calibrated on labeled defects in Calibrating an LLM Judge.

This is one task, one student base model (every fine-tune was Qwen3-0.6B) and one generator (the 35B, in three quantizations). Generalisation was tested on unseen tested words only, with inventories that overlap across contracts and a constant schema. A new deck or a new language is untested.

The frontier three-attempt ladder that produced the shipped bank is still ahead of the best local configuration, about 94% usable against 82.5%. Five contracts (歌, 块, 路, 日期, 地) never produce an audited-clean card locally. Their inventories contain no partner word the sentence needs, and the frontier needed two or three attempts on the same words and on one 块 contract never got a clean card.

The mechanical number went from 0.8% to 97.5% while grammatical output went from 77% to 10%, and the gate reported the same thing the whole time. If your loop has a cheap gate and a rising score, the next measurement is a blind judge on the cards the gate passed, read by a model that didn't write them. Mine passed 3% of the 30 cards it read, every one of which the gate had passed.