What 1,097 Citation Checks Found in Fourteen Well-Known Papers

I built a pipeline that reads what a paper says its sources say, fetches the sources, and checks. Every verdict it returns quotes the source, and the pipeline checks that quote as a verbatim substring of the source text. This is its report on fourteen papers, written for anyone who reads a comparison table or a literature review as if someone had already opened the cited papers.

Across fourteen papers in NLP, ecology, sociology and the humanities, it checked 1,097 citations, reached a verdict on 612, and found 33 problems. Four papers have what I call a fire. All four have the same shape: a number in a comparison table, or a supporting citation in the framing, that is not in the paper it is attributed to. No cited source contradicted a core finding. A core result rests on the paper's own data and code, which a citation check doesn't read.

How the checks work

For each paper the pipeline pulls out every claim that rests on a citation, with any statistic printed next to it. It parses the bibliography (GROBID for author-year lists, a splitter that checks sequence numbers for numbered lists and footnotes). Then it fetches every cited source it can, through free routes that need no key. Those are arXiv, the ACL Anthology, Crossref, Unpaywall, Semantic Scholar, Europe PMC, NBER, OSF, the DOI landing page, the cited URL live and through the Wayback Machine, and Open Library and Internet Archive scans.

A judge model reads each attribution against the fetched source. It first decides whether the fetched text really is the cited work, because retrieval sometimes serves a licence page or a review at a PDF's URL. Then it returns a verdict with the span of source text it relied on. The pipeline takes every quoted span in every claim and verdict and looks for it, character for character, in the text it quotes. A span that isn't there fails the check, so a verdict can't cite a sentence the source doesn't contain. Where a source credits someone else for the point, the pipeline follows it one hop upstream.

Here is one verdict from the ELMo report, trimmed to the fields that matter. The line breaks are mine, and so is the colon where the report has a dash.

### V128 [56]: not_supported (0.70)Cited: Gated self-matching networks for reading comprehension and question answering- Paper says: Listed in Table 9 as r-net with 76.5 EM and 84.3 F1 on SQuAD.- Finding: The published source reports 71.3 EM for its single model (and 75.9 EM  for its ensemble) on the SQuAD test set; the 76.5 EM / 84.3 F1 that the citing  paper lists for r-net comes from a later, unpublished leaderboard submission  rather than from this paper.- Source text: > The single model achieves 71.3% on the evaluation metrics of exact  match on the hidden test set, while the ensemble model further boosts the results  to 75.9%.

None of the 612 verdicts has been checked against a human yet. Before any finding about a named paper goes public, I'm hand-adjudicating about sixty verdicts. They're stratified across the four fires and the clean papers, and precision on "not supported" and "contradicted" has to be high.

The counts

The fourteen are six NLP papers (the Transformer, BERT, ELMo, GloVe, SQuAD, and Sennrich, Haddow and Birch on subword units). The other eight are one ecology paper (Hallmann et al. 2017 on flying-insect biomass), four sociology papers from Sociological Science, and three humanities papers from the Journal of Cultural Analytics. I chose them because the papers themselves are open access. A "problem" is a verdict of contradicted, not supported, or misattributed, or an in-text citation with no bibliography entry.

612 of 1,097 is a little more than half. The other 485 checks had no verdict because the pipeline couldn't get the cited source.

Coverage depends on the field. For the six NLP papers, 183 of 247 references came back as full text (74%), 17 more as an abstract or landing page, and 47 as nothing. Sociology got full text for 37 of 216 (17%) and humanities for 24 of 182 (13%). The ecology paper got full text for 12 of its 67 references and an abstract for 22.

The residue differs by field too. In NLP it's books and closed IEEE and ACM proceedings. In sociology it's paywalled journals from before repositories existed, and books. In the humanities it's books, museum records and Goodreads pages.

I also ran five economics working papers through retrieval only, to test a guess that economics' working-paper culture would put it at 60 to 80%. It came in at 84 of 258 (33%): 57 closed articles with no open copy, 47 older articles and policy-journal pieces with no resolvable DOI, and 27 working papers whose titles differ from the published versions.

"Unverifiable" always means the source wasn't obtained, which says something about access, not about the paper. The Religion paper had 96 checks and a verdict on 17, so its clean row tells you very little. The same goes for every paper with a verdict on fewer than 40% of its checks.

The number is not in the paper it cites

A comparison table lists a baseline with a citation to the original paper, and the number came from a later re-evaluation, a revised row in another paper, or a leaderboard entry. Or a framing citation points at a paper about a different thing. I call this a traveled number when it's a figure. In each case below the judge read the cited source in full.

ELMo (Peters et al. 2018, about 18,000 citations). Results: Table 5 lists Iacobacci et al. 2016 as a state-of-the-art word sense disambiguation system at 70.1 F1. The source reports per-dataset scores and no all-words F1. The 70.1 comes from Raganato et al.'s 2017 re-evaluation.

Table 10 lists Pradhan et al. 2013 at 77.5 F1 on CoNLL-2012 semantic role labeling. The source's overall argument F1 is 75.53, and 77.5 appears there only as a word-sense figure. The 77.5 is He et al.'s "Pradhan (revised)" row.

Table 9 lists r-net at 76.5 EM and 84.3 F1 on SQuAD. The published r-net paper reports 71.3 EM for its single model and 75.9 for its ensemble, and 76.5 and 84.3 are a later, unpublished leaderboard entry.

ELMo also disagrees with itself in two places. The abstract states relative error reductions of "6 to 20%", while Table 1 lists 24.9% for SQuAD and 21% for named entity recognition. The SQuAD gain is 4.7% in the text and 4.2% in the supplement.

Hallmann et al. 2017, the insect-biomass paper (2,949 Crossref citations, and heavily reported). Results: Morecroft et al. 2002 is cited as evidence that climate change is a known driver of insect decline. Its abstract says "Most butterfly and moth species increased between 1994 and 1995" after the drought, with carabids increasing and decreasing in similar numbers. That one is contradicted.

Nilsson et al. 2008 is cited twice as data on wild bee decline, and it surveys butterflies and burnet moths. It has no bee data.

The insect paper's internal numbers don't match either. The text reports a DIC difference of 822.62 between the basic and null models. Table 3 gives the two models' values as 12465.08 and 13307.59, which differ by 842.51. The trend coefficient is minus 0.081 in the text and minus 0.080 in Table 4.

SQuAD (Rajpurkar et al. 2016, about 10,000 citations). Results: Table 1 gives the "Science" dataset as 855 multiple-choice questions from standardized tests, citing Clark and Etzioni's AI Magazine piece. That article never states a dataset size.

The paper credits Wang and Jiang's match-LSTM with 70.3% F1 on SQuAD v1.1. The cited paper reports 68.2 to 76.8 dev F1 and 73.7 or 77.0 test, and 70.3 isn't among them.

Three smaller ones. The Children's Book Test is given as 688K questions, where the source's figures sum to 687,343. The CNN/Daily Mail corpus is called "Daily News". Algebra.com problems are described as coming from standardized tests.

Subword units (Sennrich, Haddow and Birch 2016, about 10,000 citations). Results: Table 3's phrase-based SMT row gives 24.3 BLEU, 53.8 CHRF3 and unigram F1 of 56.0, 31.3 and 16.5 on English-Russian newstest2015, attributed to Haddow et al. 2015. Haddow et al. report only cased BLEU of 25.0 to 25.2, averaged over newstest2013 and 2014, and none of those figures.

Separately, Neubig et al. 2012 is cited for character-based translation being "especially successful for closely related languages." Neubig et al. attribute that finding to Vilar et al. 2007, Tiedemann 2009 and Sornlertlamvanich et al. 2008, and present their own contribution as extending the approach to distant pairs. So it's misattributed. The pipeline traced it upstream to Tiedemann 2009, which does support the point.

None of the traveled numbers changes the cited paper's own result. ELMo still beats the r-net row whether r-net's number is 76.5 or 71.3. The error is in the baseline column, and a baseline column is what the next paper copies. If you take ELMo's Table 9 as your source for r-net, you inherit a number that isn't in r-net's paper, attributed to that paper. A framing citation gets copied the same way, and Nilsson et al. 2008, which counted butterflies and burnet moths, is cited twice in the insect paper as data on bees.

The smoke

Two more papers have problems that are real but sit in the framing rather than in a result.

Gaddis 2017, on how people perceive the names used in audit studies. The paper says "since at least the 1960s, a small percentage of black parents have chosen certain first names for their children at much higher frequencies than white parents," citing Fryer and Levitt 2004. Fryer and Levitt's central finding is that Black and White naming was similar in the 1960s and diverged abruptly in the early 1970s, and that the shift involved most Black parents, not a small percentage. It's framing, and Gaddis's own result doesn't rest on it.

GloVe (Pennington et al. 2014). Three points are credited to the wrong one of Mikolov's 2013 papers. The three-quarter power scaling and frequent-word subsampling are in "Distributed representations" (2013b), not "Efficient estimation" (2013a). The skip-gram model is from 2013a, not "Linguistic regularities" (2013c), which uses an RNN language model. Two cited works, Manning et al. 2008 and Finkel et al. 2005, have no bibliography entry.

Everything else the pipeline found falls into a few kinds. Internal inconsistencies: the Transformer gives 41.8 BLEU for English-French in the abstract and Table 2 and 41.0 in the results text, and never states which BLEU variant it used. The Religion paper gives 11 percent in its conclusion where the body gives 12. Credit to an intermediary: BERT credits WordPiece to Wu et al., who credit Schuster and Nakajima.

Dataset sizes: BERT describes BooksCorpus as 800M words where the source's table gives 984,846,357. BERT also credits ELMo with a 95.7 dev F1 that appears nowhere in ELMo. A claim the source doesn't make: the Indigenous data paper says Mukurtu and its developers "have publicly recognized that it does not fulfill long-term preservation and storage needs." The cited interview describes a training gap. Missing entries: Glock and Stark 1968 in the Religion paper, three endnotes in the Goodreads paper, and three each in the discrimination meta-analysis and the incarceration paper, where the PDFs merge bibliography lines.

The tally is four papers with fire, two with smoke, and eight clean or merely sloppy. Five of the eight had a verdict on fewer than 40% of their checks, so for them clean mostly means unlit.

What none of this touches

The strongest objection is that zero contradictions of a core finding means the papers are fine and the pipeline found typos. The method checks citation fidelity, not results. It can't rerun an experiment, it can't see p-hacking, and it can't judge whether a comparison was fair. A paper's core result rests on its own data and code, and a citation check reads only what the paper says other papers say.

What it can test is whether the sentence "X reported Y" is true, and in 16 places it wasn't. The other 17 of the 33 problems are in-text citations with no bibliography entry. One paper is the exception, because its finding is about other papers. That's the discrimination meta-analysis, and it came through its 36 checks with a verdict without a hard problem.

The fires are in the scaffolding, the comparison rows and framing citations around a result. Scaffolding is the measurable proxy for how carefully a paper was assembled. It is also what a replicator inherits, the baselines and the dataset sizes, so a wrong 76.5 in a baseline column becomes the target someone tries to beat.

Three of the fourteen (ELMo, the insect paper and the Transformer) report their own core numbers inconsistently within their own pages. Whether any core finding reproduces is a question the pipeline can't ask yet, because the stage that would reconstruct and rerun an experiment doesn't exist.

The second objection is that a model judged all of this. That's correct, and it's why the essay waits on the hand adjudication described above. Treat every "not supported" here as a flag to read. Every number in the four fires can be checked by anyone with the citing paper and the cited paper open.

Who else does this

Checked on 2026-09-08. The ML Reproducibility Challenge has run yearly since 2018, has sent its reports to TMLR since 2023, and reproducibility is an official NeurIPS track for 2026. It covers a few dozen papers a year and re-runs the authors' method, not the baselines they cite. Raff 2019 attempted 255 papers from the text alone and reproduced 63.5%. Gundersen and Kjensmo 2018 found that 8% of AAAI and IJCAI papers shared code and 5% stated a hypothesis. In PaperBench (OpenAI 2025), agents replicating 20 ICML 2024 papers scored 21.0% at best, against 41.4% for PhD students given 48 hours.

ReproNLP and ReproHum reproduce human evaluations in NLP every year since 2021. The Institute for Replication re-runs economics and political science papers on their deposited data. It has done over 250 studies since 2022, through one-day replication games. scite labels whether a citing sentence supports, contrasts with or merely mentions the cited work, from full text, but it doesn't check whether the cited work says what is claimed. CiteAudit (2026), CiteGuard (2025), CiteVerifier and related tools check that references exist as real publications. They were prompted by over 100 hallucinated citations found in 53 accepted NeurIPS 2025 papers and by hallucinated citations in 50 ICLR 2026 papers. None of them reads the cited source and asks whether it supports the attributed claim.

The closest measurement is in medicine, where it's been done by hand. 16.9% of quotations misrepresent their source (95% CI 14.1 to 20.0), about half of those major, with no improvement over time. I didn't find anyone checking, at scale and against full text, what papers attribute to their sources. This pipeline reads the full text of every source it can get, and each verdict carries a verbatim span of that source. On these fourteen papers that came to 612 verdicts, sorted into fire and smoke.

Limits

The judge is unvalidated against humans as of this draft. Coverage below 40% makes a clean row uninformative, so sociology and the humanities are mostly unlit. Reference parsing still merges some entries in dense two-column bibliographies. Those show up as a missing reference rather than a wrong verdict, but they add a few to the problem counts of the sociology papers.

Retrieval sometimes serves the wrong document at a PDF URL (licence pages, reviews, handle pages, a student assignment in place of a 1997 paper). The judge's identity check caught every case that was noticed, and one it missed would show up as a false verdict. The NLP citation counts above are approximate; Semantic Scholar was rate-limiting when I queried it, and Crossref, used for the rest, undercounts Google Scholar by roughly half.

What changes for your tables

If you copy a baseline from a comparison table, copy it from the paper the table cites, and if the number isn't there, cite where it is. ELMo's r-net row is the example: 76.5 EM in ELMo's Table 9, 71.3 single and 75.9 ensemble in r-net's own paper. The cost of checking is now about an hour of agent time per paper plus two minutes of fetching. The next run is the fifty most-cited NLP papers, all open, every citation checked, with one count of how often a comparison number is in its cited source. If the traveled number holds up as a rate, the natural place for this check is at submission. A low rate would make four of fourteen a fluke of these four papers.

Citations

• Peters et al. (2018). Deep contextualized word representations. NAACL. arXiv:1802.05365
• Hallmann et al. (2017). More than 75 percent decline over 27 years in total flying insect biomass in protected areas. PLOS ONE. doi:10.1371/journal.pone.0185809
• Rajpurkar et al. (2016). SQuAD: 100,000+ Questions for Machine Comprehension of Text. arXiv:1606.05250
• Sennrich, Haddow and Birch (2016). Neural Machine Translation of Rare Words with Subword Units. arXiv:1508.07909
• Gaddis (2017). How Black Are Lakisha and Jamal? Racial Perceptions from Names Used in Correspondence Audit Studies. Sociological Science. doi:10.15195/v4.a19
• Pennington, Socher and Manning (2014). GloVe: Global Vectors for Word Representation. EMNLP. D14-1162
• Vaswani et al. (2017). Attention Is All You Need. arXiv:1706.03762
• Devlin et al. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805
• Quillian et al. (2019). Do Some Countries Discriminate More than Others? Sociological Science. doi:10.15195/v6.a18
• Hout and Fischer (2014). Explaining Why More Americans Have No Religious Preference. Sociological Science. doi:10.15195/v1.a24
• Haskins (2014). Unintended Consequences: Effects of Paternal Incarceration on Child School Readiness and Later Special Education Placement. Sociological Science. doi:10.15195/v1.a11
• Walsh and Antoniak (2021). The Goodreads "Classics". Journal of Cultural Analytics. doi:10.22148/001c.22221
• Stark and Hoffmann (2019). Data Is the New What? Journal of Cultural Analytics. doi:10.22148/16.036
• Guiliano and Heitman (2019). Difficult Heritage and the Complexities of Indigenous Data. Journal of Cultural Analytics. doi:10.22148/16.044
• Raff (2019). A Step Toward Quantifying Independently Reproducible Machine Learning Research. arXiv:1909.06674
• Gundersen and Kjensmo (2018). State of the Art: Reproducibility in Artificial Intelligence. AAAI.
• OpenAI (2025). PaperBench. openai.com/index/paperbench
• CiteAudit (2026). arXiv:2602.23452
• CiteGuard (2025). arXiv:2510.17853
• Quotation accuracy meta-analysis (2025). Research Integrity and Peer Review. doi:10.1186/s41073-025-00173-z