fig. 2.7.3: three audits of one card bank on one axis, each at its own sample and its own bar fig. 2.7.3 · three audits of one card bank on one axis, each at its own sample and its own bar.⛶ THREE AUDITS, THREE BARS: 20%, 35% AND 65% KEPT APART ONE DOT IS ONE AUDIT OF THE CARD BANK, ITS AREA THE CARDS IT SAMPLED; RIGHT IS THE SHARE THAT FAILED THAT AUDIT’S BAR. 0% 25% 50% 75% 100% 20% ACCEPTANCE BAR CLOSING AUDIT 4 OF 20 · 20% SONNET RE-SCREEN 263 OF 755 · 34.8% OPUS AUDIT OF THE RE-SCREENED BANK 13 OF 20 · 65% HAIKU-GENERATED · 10 OF 11 · 90.9% SONNET-GENERATED · 3 OF 9 · 33.3% CLOSING AUDIT: 20 CARDS, EVERY 37TH BY ID · CLOSING REVIEW, A SECOND MODEL · A LOWER BOUND: NO DISAGREEMENT RECORDED ON DISCRIMINATION OR DISTRACTORS SONNET RE-SCREEN: ALL 755 CLEAN CARDS · V2 RUBRIC, NATIVE-SPEAKER BAR · JUDGE FLIPS: A SECOND JUDGE DISAGREEING WITH THE FIRST, NOT PROVEN DEFECTS OPUS AUDIT OF THE RE-SCREENED BANK: 20 RANDOM CARDS · THE SCRIPT’S AUDITOR PROMPT, OPUS AUDITOR · CLOSEST TO PREVALENCE; THE NUMBER THAT FORCED THE PIPELINE CHANGE
source: vocabinator research/2026-08-24-cardbank-rescreen-root-cause.md lines 3, 27; data/cardgen-audit.json; data/cardbank-rescreen.json; data/cardgen-audit-rescreen-opus.json; task T-20260819-09e2674a line 94 (repo e77a474) · judgeAudits.data.ts
View in context: Calibrating an LLM judge → · All figures
Andrew Fribush , software and computer engineer in Austin. I build consequential systems where software meets institutions: finance, healthcare, commerce, government data, and the mechanisms that govern AI agents. Taking new engagements: hello@andrewfribush.com .