Testing API Wrappers for Silent Empty Results

The live test suite for an MCP server over US government data APIs calls every tool outside the bulk-ingest modules once against the real agency and checks that nothing threw. I added a second check: unless a tool is on a short allow list, its canned arguments point at data that exists, so an empty answer fails the test even though nothing errored. The suite tags that failure [SILENT EMPTY]. With that one assertion the suite catches 8 of the 12 upstream breaks one hardening pass found, where checking for errors alone caught 2.

The case it exists for is the CMS provider-data module. All four of its tools had been returning [] for every query. The DKAN metastore had stopped honoring show-reference-ids= with an empty value, so the distribution-UUID lookup resolved to an empty string and every query short-circuited to an empty list. Nothing threw, so the smoke suite stayed green, and I found it in a hands-on audit, calling each tool with arguments the suite had never used.

In a wrapper over other people's APIs, the most common failure is a well-formed empty or wrong result that looks like success and throws no error, so a test suite that only checks for errors passes it every time. The server had 352 tools across 42 modules on the audit date, and 387 by the end of the same audit. The suite that calls them runs nightly against the live APIs. The claims here are about the test suite: what one hardening pass found, what the guard now catches, and what it can't. Nothing in them is about the agencies' data or the MCP protocol.

The guard's first run flagged 10 tools, and they came down to one real bug and nine stale canned arguments. The real bug: the two committee-report list endpoints at Congress.gov answer with the key reports, and the SDK read committeeReports, which is the detail endpoint's key, so both list tools had always returned empty. The stale arguments: a bill constant pointed at 118 HR 3684, a minor bill, instead of the 117th Congress infrastructure act, so every bill-votes and related-bills probe was asking about a bill with nothing attached. A clinical-trial id had no posted results. A VIN was an invalid example. A GSA contract number had expired. An FEC contributions call ran unfiltered, which the tool refuses by design. Each stale argument needed a researched replacement before the guard would pass.

Twelve Breaks, and Eleven of Them Answered 200

The full catalogue from the pass is the figure below, with what came back under each break. It files each row under the weakest assertion that would have caught it: a suite that makes a given assertion also makes every weaker one, so choosing a rung lights its rows and every row above it.

Eleven of the twelve rows were an HTTP 200. Two of them were well-formed answers in the wrong shape: the CFPB suggest endpoint returns a bare string array where the tool expected an Elasticsearch suggest envelope, and the congressional record answers in PascalCase (Issue, Volume, PublishDate, Links), so its rows came back as empty objects. openFDA was the one 404, and it's the inverse case: it reports zero matches as 404 NOT_FOUND, so all 25 of its tools threw an error where an empty result was the right answer. It and BEA, whose empty 200 body made the JSON parser throw, are the 2 rows the weakest assertion the suite makes could catch, because a thrown error is the one thing "did it throw" sees.

The BTS dataset carries placeholder rows for months not yet published, and the tool filtered those out by checking whether general_economic_indicators was present. That column is quarterly GDP, so it exists in one month of every three. The filter dropped the other two, and monthly statistics came back quarterly, with the right shape and a monthly label. I replaced it with a future-date guard, which is the dataset's actual placeholder case.

The USPTO row couldn't be found until a key arrived, because the module had been audited only for clean keyless errors. Two decisions tools read patentTrialDecisionDataBag and petitionDecisionBag, and the API returns patentTrialDocumentDataBag and petitionDecisionDataBag. Both answered "No decisions found" for every query. With the right keys, the audit's verification queries found 19,838 trial decisions and 7,489 petition decisions, and both tools are now in the live-smoke arguments as regression guards.

EDGAR full-text search drops the start date when no end date is given, and the reverse. A query for filings since 2026-08-25 returned 10,000 results reaching back to 2018, with the right columns. A missing bound now defaults, the end date to today and the start date to the 2001 full-text-search epoch, and in the September verification the same lone start date returned 20 filings, the newest from 2026-08-31. (Congress.gov and the CFPB handle a lone bound correctly, so the quirk is EDGAR's.)

Assert Nonempty Results for Known-Good Queries

The guard runs after the throw check and after a check that the payload isn't error-shaped (a summary beginning error, failed, unauthorized, forbidden or invalid api key). It takes the parsed response and walks a ladder of reasons the result could be empty. The first reason that applies becomes the failure message. This is the whole of it, trimmed from tests/live/tools.smoke.test.ts, with long lines wrapped:

if (!ALLOWED_EMPTY.has(tool.name) && res && typeof res === "object") {  const r: any = res;  const allRowsEmpty = (arr: unknown[]) =>    arr.length > 0 && arr.every(x =>      Array.isArray(x) ? x.length === 0      : (x && typeof x === "object"          ? Object.keys(x).length === 0 : false));  const emptyBecause =    r.dataType === "empty" ? "dataType=empty"    : Array.isArray(r.data?.rows) && r.data.rows.length === 0      ? "0 rows"    : Array.isArray(r.data?.items) && r.data.items.length === 0      ? "0 items"    : Array.isArray(r.data?.rows) && allRowsEmpty(r.data.rows)      ? `${r.data.rows.length} rows, all empty (field-mapping bug?)`    : Array.isArray(r.data?.items) && allRowsEmpty(r.data.items)      ? `${r.data.items.length} items, all empty (field-mapping bug?)`    : null;  expect(emptyBecause,    `[SILENT EMPTY] ${tool.name} returned no data for its canned args ` +    `(${emptyBecause}). If this emptiness is legitimate, add the tool ` +    `to ALLOWED_EMPTY in tests/live/args.ts with a reason.`,  ).toBeNull();}

The two all-empty rungs weren't in the first version, which checked for dataType=empty and zero-length rows or items and nothing else. USAspending's over-time tool asked the response envelope for the raw API's field names after the SDK had renamed them, so each of 37 periods rendered as an empty row, []. Thirty-seven empty rows is a rows.length of 37, and the first guard passed it. I added the all-empty rungs, and the strengthened guard caught two tools: usa_spending_over_time, and congress_committee_report_text, whose fields were nested inside formats[]. The message says "field-mapping bug?" because in both cases the data arrived and the code read it from the wrong key.

The allow list has 25 tools on it, each under a comment giving the reason. Two weather-alert tools can be empty because the country can be quiet, and FDA drug shortages vary with the world. The Federal Register's public-inspection queue is empty on weekends and holidays. One FEC tool and five Open Payments tools refuse unfiltered queries by design. Fourteen Congress.gov tools are legitimately empty for most subjects (no amendments on an amendment, a committee with no pending nominations, recess weeks, and most amendments have no cosponsors). The last is the FBI use-of-force tool, which is dead upstream. The failure message names the file and asks for a reason, so an exemption without one doesn't get written.

The cost of the guard is the nine stale arguments. Each one was a canned call that had been passing for the wrong reason: the suite was checking that a query about a bill with no votes didn't throw, and it never did. Fixing them meant finding a bill with votes (the infrastructure act), a trial with posted results (the Pfizer COVID vaccine trial) and a real F-150 VIN. For the FEC it meant an indexed committee lookup, since a name search is an unindexed two-minute full scan. A stale argument is a canned call aimed at data that has gone away, and fixing it means swapping in a call aimed at data that exists, so the guard's check stays as written. A canned argument that stops returning data also tells me an agency changed something. The commit that added the guard landed with 1,000 unit tests and 478 live tests green.

The Nightly Run Tests the Live Agencies

The suite runs on cron 17 6 * * *, 06:17 UTC, and never on a pull request, because an agency's outage must not block a merge. Every failure carries a tag. A 4xx is [REPO BUG (4xx)] and fails at once, since a bad parameter or a renamed field is deterministic. A 5xx, a timeout and a network or DNS error are three [UPSTREAM] tags. They're retried twice more for a 5xx or a network error and once more for a timeout, and the retries are budgeted to fit inside the 90 s test timeout (a slow FBI endpoint had been surfacing as a bare vitest timeout). A daily-cap message is [QUOTA EXHAUSTED]. Anything else that throws is [THREW]. Then come two untagged checks, an error-shaped payload and a null result, then [SILENT EMPTY], then [SCHEMA DRIFT].

Schema drift compares the response's column set against a baseline recorded in tests/live/response-shapes.json. The file has 308 entries. All 308 record a dataType and 102 record a column set, and the check compares columns only when both the baseline and the response have them, so column drift is checkable for 102 tools. A baseline column that disappears fails the run, and the failure names the missing columns and any new ones, and the fix, once I've looked at what the agency did, is RECORD_SHAPES=1 npm run test:live.

A live run also warms the disk cache, and the offline suite replays from it with fetch stubbed to throw. There are no fixtures in the repo; the recorded responses are the cache entries the nightly run wrote. On the day I added it, 455 of 531 cases replayed end-to-end in about 80 s with zero network, and a case whose cache entry isn't warm skips rather than fails. The nightly runs the replay straight after the live suite, so the replay machinery is tested every night against the responses the agencies just gave.

A broken agency goes into known-upstream-failures.json with a date and a reason. The one entry today is fbi_use_of_force, since 2026-09-01: the FBI's Crime Data Explorer returns 404 for every tested /participation/*/uof* variant while its other endpoints still answer 200. The test runs with it.fails, so the nightly stays green while the outage lasts and turns red when the tool starts working again, which is the signal to delete the entry.

If you're wrapping someone else's data, the suite reduces to seven steps. Give every tool canned arguments that target data known to exist, and fail on empty unless the emptiness is on an allow list with a reason. Treat an array of empty rows as a field-mapping bug, and baseline the column set so drift fails. Run it nightly against the real thing and never gate merges on it, then replay offline from the warmed cache, so the replay itself is proven with zero network. Keep a dated list of known upstream failures, and make recovery turn the build red.

This Is Integration Testing With One Changed Assertion

The strongest objection is that this is integration testing, and the project already had it. The first half is right. The suite that missed the CMS outage was already calling the tools against the live APIs every night. The change is one assertion, that success has a shape other than "no exception", and it moved the nightly count from 2 of the 12 to 8.

The second objection is that allow lists rot. They do, which is why each entry carries a reason and the failure message names the file an exemption has to be written in.

Limits

The guard checks canned arguments only. A tool can be empty for real inputs it was never asked, which is how the CDC state filters stayed broken: the canned calls never set a state, and the audit found the bug by trying both "OH" and "Ohio". Wrong-but-nonempty results are outside it: the BTS quarterly-as-monthly case and the EDGAR whole-corpus case both had the right shape, and both were found by reading output with novel arguments. Schema drift catches some of that class, and no check in the suite catches the rest. Every tool and module count in this essay comes from the audit file. And the catalogue comes from one server and one hardening pass.

The old check asked whether the call threw, and the new one asks whether anything came back. Whether what came back is right still takes a person reading output from new arguments, which is how the BTS and EDGAR rows were found.

Appendix: Reject Inputs That Normalize to an Empty String

The resolver bug in the catalogue is its own class. A fuzzy resolver lowercases the input, strips everything that isn't an ASCII letter or a space, and substring-matches the result against every candidate name. Cyrillic, Chinese and symbol-only input strip to an empty string, and "".includes("") is true for every candidate, so the resolver returns its first or shortest match. World Bank resolved "США" to Cuba, the shortest country name. Census resolved "北京" to Parish village, New York. NHTSA's model resolver did the same with an empty model string. The fix is one line before the match, and I swept it across every resolver. This is the World Bank one, from src/apis/world-bank/sdk.ts, trimmed:

const norm = (s: string) =>  s.toLowerCase().replace(/[^a-z ]/g, " ").replace(/\s+/g, " ").trim();const nq = norm(q);// Non-Latin input normalizes to nothing, and "".includes matches every// country ("США" used to resolve to Cuba, the shortest name). Bail out.if (nq.length < 2) return null;const exact = countries.find(c => norm(c.name) === nq);if (exact) return { code: exact.iso2Code, name: exact.name };const partial = countries.filter(c =>  norm(c.name).includes(nq)  || (nq.length >= 4 && nq.includes(norm(c.name))));if (partial.length) {  partial.sort((a, b) => a.name.length - b.name.length);  return { code: partial[0].iso2Code, name: partial[0].name };}return null;

Census bails the same way when the normalized query is under two characters, with five-digit ZIP codes exempt since those resolve to their own ZCTA codes. Its variable search returns [] for an empty token list instead of every variable. A resolver that can't match returns nothing (null in World Bank, [] in Census), because the caller is usually a model, and a model will take Cuba and keep going.