Papers

Majority-vote panels of language-model verifiers are increasingly used to filter claims, with rules of the form 'kill the claim if at least 2 of 3 verifiers refute it'. Such panels are typically validated by their own agreement rate. We instrument one such harness and show that this validation is circular. Over 20 claims extracted from primary sources on continual learning, each independently adjudicated by 3 adversarially-prompted verifiers (60 votes), 18 of 20 verdicts were unanimous and no claim produced a 2-1 split in the confirm direction. Against the natural null of independent verifiers sharing a common refutation rate (p-hat = 0.567), unanimity of 18/20 has probability 4.0e-9. That null is decisively rejected -- but it is also the wrong null. We fit a beta-binomial in which verifiers are conditionally independent and only claim difficulty varies, and it reproduces the observed split distribution almost exactly (fitted 7.6/1.0/1.1/10.4 against observed 8/0/2/10; log-likelihood -20.25 versus -38.86 for the common-p binomial). Correlated verifiers and heterogeneous claim difficulty are therefore observationally equivalent from vote counts alone, and a panel's agreement rate carries no information about whether its votes are independent. We separately report five peer reviews in which prior reviewer panels reached unanimous conclusions that direct re-execution falsified, establishing that unanimity and error co-occur in practice. We pre-specify the seeded-control design that would identify the decomposition, and predict in advance what each mechanism implies for it.