Andrew Fribush
fig. 2.5.1 · twelve API failures in a 352-tool server, filed under the weakest test assertion that catches each.
WITH THE SILENT-EMPTY GUARD THE SUITE CATCHES 8 OF 12 BREAKS; CHECKINGFOR ERRORS ALONE CAUGHT 2ONE ROW IS ONE ASSERTION A TEST SUITE CAN MAKE, WEAKEST FIRST; ONE SQUARE IS ONE OFTHE 12 BREAKS, LIT WHEN THAT ASSERTION, OR A WEAKER ONE, CATCHES IT. CLICK ANASSERTION.CHECK FOR ERRORS ONLY · 2 OF 12ASSERT NON-EMPTY · 6 OF 12ASSERT NO ALL-EMPTY ROWS · 8 OF 12A HUMAN READS THE OUTPUT · 12 OF 12THE BREAK, UNDER THE WEAKEST ASSERTION THAT CATCHES ITWHAT THE SUITE SAWCHECK FOR ERRORS ONLYBEA WITHOUT A KEY: HTTP 200 WITH AN EMPTY BODY200 · EMPTY BODY, THE PARSER THREWOPENFDA, 25 TOOLS: ZERO MATCHES THREW AN ERROR404 · THROWNASSERT NON-EMPTYCMS PROVIDER DATA, 4 TOOLS: EVERY QUERY EMPTY200 · []CONGRESS COMMITTEE REPORT LISTS: ALWAYS EMPTY200 · []USPTO DECISIONS, 2 TOOLS: EMPTY FOR EVERY QUERY200 · NO DECISIONS FOUNDCFPB SUGGEST COMPANY: EMPTY FOR EVERY INPUT200 · []ASSERT NO ALL-EMPTY ROWSUSASPENDING OVER TIME: 37 PERIODS OF EMPTY ROWS200 · 37 × []CONGRESSIONAL RECORD: ROWS OF EMPTY OBJECTS200 · [{}, {}, …]A HUMAN READS THE OUTPUTBTS MONTHLY STATS: 2 OF EVERY 3 MONTHS DROPPED200 · 1 ROW IN 3, RIGHT SHAPEEDGAR "SINCE LAST WEEK": THE CORPUS BACK TO 2018200 · 10,000 ROWS, RIGHT SHAPERESOLVERS: CYRILLIC "США" RESOLVED TO CUBA200 · ONE PLAUSIBLE ROWCDC STATE FILTERS, 4 TOOLS: EMPTY ON ONE SPELLING200 · [] FOR "OH" OR "OHIO"

A suite that makes an assertion also makes every weaker one, so choosing a rung lights its rows and every row filed above it. The guard’s first run flagged 10 tools: 1 real bug and 9 stale canned arguments, fixed by maintaining the arguments rather than relaxing the guard.

source: us-gov-open-data-mcp docs/tool-audit.md lines 17-19, 37-39, 65-67, 80-84, 101-103, 157-158, 303-305, 347-352, 369-372, 408-412, 451-456; commits 1c336a4, fb2c96a, 8f16e46, a1ae253, 9177aa1, ef7e443, 6e0c7ee (repo 796ea9d) · silentEmpties.data.ts

View in context: Testing API wrappers for silent empty results → · All figures

Andrew Fribush, software and computer engineer in Austin. I build consequential systems where software meets institutions: finance, healthcare, commerce, government data, and the mechanisms that govern AI agents. Taking new engagements: hello@andrewfribush.com.