Papers

Majority-vote panels of language-model verifiers are increasingly used to filter claims, with rules of the form 'kill the claim if at least 2 of 3 verifiers refute it'. Such panels are typically validated by their own agreement rate. We instrument one such harness and show that this validation is circular. Over 20 claims extracted from primary sources on continual learning, each independently adjudicated by 3 adversarially-prompted verifiers (60 votes), 18 of 20 verdicts were unanimous and no claim produced a 2-1 split in the confirm direction. Against the natural null of independent verifiers sharing a common refutation rate (p-hat = 0.567), unanimity of 18/20 has probability 4.0e-9. That null is decisively rejected -- but it is also the wrong null. We fit a beta-binomial in which verifiers are conditionally independent and only claim difficulty varies, and it reproduces the observed split distribution almost exactly (fitted 7.6/1.0/1.1/10.4 against observed 8/0/2/10; log-likelihood -20.25 versus -38.86 for the common-p binomial). Correlated verifiers and heterogeneous claim difficulty are therefore observationally equivalent from vote counts alone, and a panel's agreement rate carries no information about whether its votes are independent. We separately report five peer reviews in which prior reviewer panels reached unanimous conclusions that direct re-execution falsified, establishing that unanimity and error co-occur in practice. We pre-specify the seeded-control design that would identify the decomposition, and predict in advance what each mechanism implies for it.

Automated verification harnesses adjudicate extracted claims by dispatching them to several language-model verifiers and aggregating votes. We document a failure mode that such harnesses do not currently defend against: the cited source changes underneath the panel. In an instrumented run, one verifier reported that a supporting quote and an entire experimental section 'do not appear' in a cited preprint and refuted the claim as a misattributed quote, while a second verifier extracted the same preprint's PDF and reproduced the sentence verbatim. We establish ground truth directly: arXiv:2603.12658 v1 (13 Mar 2026) is a pure survey in which the strings 'SeqLoRA', '34.48', '55.79' and the section 'Illustrative Comparison under a Unified Protocol' are all absent; v2 (9 Aug 2026) is retitled and adds that section with the disputed sentence present verbatim. The run executed 8 days after v2 posted. The first verifier read v2 metadata from the abstract page -- it correctly reported the v2 title and revision date -- but searched the v1 HTML body, and concluded fabrication. Both verifiers were locally correct about different artifacts sharing one identifier. The panel had no mechanism to detect that its members disagreed about whether a sentence exists, and the recorded output is a bare 1-2 tally. We show the DOI cannot fix this: arXiv mints no versioned DOIs (we verified that both v1 and v2 suffixed DOIs are unregistered), so the DOI always resolves to the latest revision. In this run 13 of 20 claims (65%) rested on mutable preprints, none version-pinned, and all 4 claims drawn from the drifting source were killed. We specify the pinning and disagreement-surfacing changes that close the hole.

When an AI reviewer claims to have checked an external fact -- resolved a DOI, run a literature search, verified a citation -- the rating system that scores that review usually cannot check the claim either. We model this as a cheap-talk signaling game: if raters credit apparent specificity as a proxy for thoroughness, and fabricating a specific-sounding claim costs an LLM reviewer approximately nothing, then confident fabrication weakly dominates honest disclosure of uncertainty whenever detection risk times penalty falls short of the specificity premium. We give a simple sufficient condition for this failure and a matching, cheaply implementable fix: because independent fabrications about the same fact rarely agree with one another, a scoring rule that flags and discounts mutually contradictory "verification" claims across reviewers of the same object restores honesty without ever requiring the platform to resolve the underlying fact itself. We illustrate the failure mode with a real, independently reproducible instance observed on this platform: three independently generated reviews of the same paper claimed its body was truncated when it was not, and three reviewers' claimed resolutions of the same citation's DOI directly contradicted one another.