fig. 2.5.2 · the five checks the nightly runs on every tool call, in order, with the tags each failure carries.⛶ EVERY CALL ANSWERS FIVE QUESTIONS NIGHTLY; "DID IT THROW" IS THE FIRST ONE ROW, ONE CHECK ON EVERY TOOL CALL, IN THE SUITE'S ORDER, NIGHTLY, NEVER ON A PR. ACCENT: THE FAILURE'S TAG. 1 DID THE CALL THROW? [QUOTA EXHAUSTED] [REPO BUG (4XX)] [UPSTREAM (5XX)] [UPSTREAM (TIMEOUT)] [UPSTREAM (NETWORK/DNS)] [THREW] 2 IS THE PAYLOAD SHAPED LIKE AN ERROR? RETURNED AN ERROR PAYLOAD 3 DID IT RETURN ANYTHING AT ALL? RETURNED NOTHING 4 IS IT A SILENT EMPTY? [SILENT EMPTY] 5 DID THE COLUMNS DRIFT? [SCHEMA DRIFT] PASS THE ANSWER IS IN THE DISK CACHE; THE OFFLINE REPLAY RERUNS IT STRAIGHT AFTER, 455 OF 531 CASES IN ABOUT 80 S WITH ZERO NETWORK 1 THE MESSAGE NAMES A DAILY QUOTA [QUOTA EXHAUSTED]; HTTP 4XX [REPO BUG (4XX)]; HTTP 5XX [UPSTREAM (5XX)]; A TIMEOUT [UPSTREAM (TIMEOUT)]; DNS OR NETWORK [UPSTREAM (NETWORK/DNS)]; ANYTHING ELSE [THREW] 2 THE SUMMARY BEGINS "ERROR", "FAILED", "UNAUTHORIZED", "FORBIDDEN" OR "INVALID API KEY" RETURNED AN ERROR PAYLOAD 3 NOTHING PARSES OUT OF THE RESPONSE RETURNED NOTHING 4 SKIPPED FOR THE 25 TOOLS ON ALLOWED_EMPTY, EACH LISTED WITH ITS REASON; DATATYPE=EMPTY [SILENT EMPTY]; 0 ROWS [SILENT EMPTY]; 0 ITEMS [SILENT EMPTY]; N ROWS, ALL EMPTY [SILENT EMPTY]; N ITEMS, ALL EMPTY [SILENT EMPTY] 5 MISSING COLUMNS, AGAINST 102 RECORDED COLUMN SETS; A BASELINE COLUMN IS GONE [SCHEMA DRIFT]
source: us-gov-open-data-mcp tests/live/tools.smoke.test.ts lines 100-105, 152-222; tests/live/args.ts lines 398-422; tests/live/response-shapes.json; .github/workflows/live-smoke.yml lines 4, 9; commit e95252b (repo 796ea9d) · nightlyGauntlet.data.ts
View in context: Testing API wrappers for silent empty results → · All figures
Andrew Fribush , software and computer engineer in Austin. I build consequential systems where software meets institutions: finance, healthcare, commerce, government data, and the mechanisms that govern AI agents. Taking new engagements: hello@andrewfribush.com .