Papers

A published law placed 94% of a sealed hold-out within a factor of two: t_gen = 850.5 * lr^-1 * wd^-1 * train_frac^-4 * p^-2, read as saying that the cumulative weight-decay shrinkage at generalisation is set by the task alone. We ran the two falsification tests that paper named. The first, an iso-dose grid holding lr*wd fixed while varying lr over a decade, REFUTES the exact form: the slope of log10(t_gen) on log10(lr) is -0.1070 (95% CI [-0.169, -0.045]), so the exponents are near-equal but measurably unequal. The second, three weight-decay schedules delivering an identical total dose at different times, CONFIRMS the mechanism decisively: front-loading halves t_gen (ratio 0.510, CI [0.412, 0.630]) and back-loading delays it (1.499, CI [1.213, 1.853]), both excluding the elapsed-time prediction of 1.000, with all three arms landing within 6.4% of numbers fixed before execution. Grokking is governed by accumulated decay, not elapsed time; the accumulating quantity is close to but not exactly lr*wd*t. We also report two methodological measurements. Neither test was among the 1,848 hypotheses the original search proposed, and we argue this is structural rather than accidental: a grid holding lr*wd fixed reduces variance in the feature the proposer is scored on, so it is a worse grid by the proposer's own criterion. And scoring every round's discovered law against the vault retrospectively shows the search saturated at round 10: rounds 1-10 cost $24 and reached a vault median error of 0.0792; rounds 11-114 cost $252 and reached 0.0784.

Predictive claims across several fields are validated by reporting the fraction of held-out points falling within a factor of T of the prediction. That statistic has a null model which is almost never reported: a CONSTANT predictor ignoring the inputs entirely. We give the null in closed form. If log10 of the held-out target has standard deviation s, the constant's absolute log error is half-normal, so its expected pass fraction is p_null(T,s) = 2*Phi(log10(T)/s) - 1. Monte Carlo over 21 (T,s) cells reproduces this to a maximum absolute error of 0.0007 against a 0.0020 tolerance derived from the Monte Carlo standard error rather than chosen. Two usable outputs follow. First, a requirement table: at tolerance factor 2, a constant scores at or above 0.90 unless the held-out target spans more than 0.183 dex, and at or above 0.60 unless it spans more than 0.358 dex. A study whose held-out target is narrower than that cannot distinguish its law from a constant however good the law is, and the pass fraction it reports is uninformative rather than merely weak. Second, a sample-size table: separating a law that is right 95% of the time from its constant baseline requires 191 held-out rows when the target spans 0.20 dex, and 33 when it spans 0.30 dex. Corpora in this area typically hold tens. Applied to a published round that reported 24/25 = 0.960 inside a factor of two against a committed bar of 0.60, the constant scored 21/25 = 0.840 on the same rows, implying a held-out spread of 0.214 dex - 1.67x too narrow for the 0.60 bar to be falsifiable, and 3.9x too few rows to separate the two figures. We also report a methodological incident: the derivation's first validation failed by 25 sigma because of a floating-point defect in a linear congruential generator, and was caught only because the acceptance threshold had been derived from the standard error instead of set to a round number.