Andrew Fribush

fig. 2.15.3: fourteen decks scored by the generator’s own model and by a blind cross-model audit of the same cards

fig. 2.15.3 · fourteen decks scored by the generator’s own model and by a blind cross-model audit of the same cards.
GENERATOR’S JUDGE SCORED 3 TO 17 POINTS HIGHER IN 13 OF 14 AUDITSONE ROW IS ONE DECK OF GATE-CLEAN CARDS; INK IS THE BLIND CROSS-MODEL AUDIT,ACCENT THE 35B JUDGE ON THE SAME CARDS, GAP IN POINTS.60% OF THE DECK PASSED70%80%90%100%LOCAL DECKS · MLX 4-BIT, THE JUDGE’S OWN WEIGHTSREMOTE DECKS · THE SAME MODEL IN FP8 OR AWQ, JUDGED BY THE LOCAL WEIGHTSMASK + CODE, K=4AUDIT 64JUDGE 71 · +7 PTSFEW-SHOT, REPAIR8087 · +7DISTRACTOR GUARD, SEED 08390 · +7DISTRACTOR GUARD, SEED 18590 · +5RANKER-GATED REPAIR8083 · +3CONTRACT LADDER8087 · +7K=12, PENALTY 1.158196 · +15PENALTY 1.07693 · +17PRESENCE PENALTY 1.0-2 · 7375FP87993 · +14FP8, NO REPAIR8187 · +6FP8, EOS BLOCKED7187 · +16FP8, RANKER-GATED REPAIR8693 · +7CONTRACT LADDER8293 · +11ASKED FOR THE FAILURE CLASSES INSTEAD OF “IS THIS NATURAL MANDARIN?”, THE SAMEWEIGHTS CAUGHT 8 OF 8 BAD SENTENCES THE CROSS-MODEL AUDIT HAD FOUND, WITH 5 FALSEALARMS.

Every deck is the gate-clean cards of one pipeline run on the held-out contracts, scored on every criterion by the open-weight 35B that also generates, then read blind, source hidden, by the project’s Claude agent, a different model family: the cross-model audit, sentence-level. The gap is judge minus audit, in percentage points; the one negative gap is the presence-penalty deck.

source: minimodels DIARY.md:2715-2718, 2949-2952, 3185-3188, 3319, 3353, 3432, 3485-3488, 3547, 3607, 3713, 3767, 3901-3903, 4038-4043 · judgeBias.data.ts

View in context: How a student model gamed its verifier → · All figures

Andrew Fribush, software and computer engineer in Austin. I build consequential systems where software meets institutions: finance, healthcare, commerce, government data, and the mechanisms that govern AI agents. Taking new engagements: hello@andrewfribush.com.