Papers

We pre-registered a quantitative theory of grokking, hash-froze it before any held-out run executed, ran the held-out set once, and report that it failed its own falsification condition. The theory predicts the grokking step zero-shot from the configuration dictionary alone, with no probe of the run being predicted. Its clock is derived rather than fitted: under AdamW the normalised update has unit modulus wherever the second moment exceeds epsilon, so gradient magnitude, loss scale and parameter count divide out and decoupled decay is the only unnormalised rate left, giving t_grok = A/(eta*lambda) with exponent exactly -1, confirmed at -0.98, -1.11, -0.73 and -0.96 across four training fractions. On 80 configurations it placed 0.550 of rows within a factor of two against a committed 0.61: falsified. The failure is structured, not diffuse. Inside the task families it was fitted on it is unbiased (mean error +0.006 dex over 42 runs) and extrapolates far outside its fitting range, predicting the canonical modular-addition configurations at p=97 and p=113 to within factors of 1.2 to 1.7 having never seen a modulus above 59 or a two-layer model. On eight unseen task families it is biased +1.004 dex, 68 percent one-sided late: max(a,b) predicted at 7700 steps against a true 28, sparse parity at 164778 against 307. The diagnosed cause is that its data term counts cells in the lookup table when what sets the time is the difficulty of the rule, and the two coincide only inside modular arithmetic. A second cycle partially repairs this with a vocabulary-normalised rank exponent, reaching 0.538 under leave-one-task-family-out and turning parity from 2.677 dex wrong into a median error of 0.030 -- but its features were selected on that metric, so a second held-out set of ten never-run families is hash-committed and executing. We release the harness, 396 runs and both registries.

Grokking is normally reported as a step number, as if delayed generalisation happened at a determinate time. It does not. Across 85 training runs that are identical in task, architecture, optimiser, hyperparameters and data split, and differ only in the initialisation seed, the grokking step scatters with a standard deviation of 0.19 in log10 -- a factor of 1.6 -- and in the worst cell by a factor of 15.2. The scatter is not an artefact of a threshold crossing: all runs in that cell show the same smooth sigmoid rise off a common test-accuracy plateau with training accuracy pinned at 1.0, and differ only in when it starts. The scatter is also not a property of grokking. It is a property of the architecture. Attention-free networks grok at a nearly determinate step, sigma = 0.0382 over 19 runs, with no cell exceeding a 1.4x spread; transformers on the same tasks at the same hyperparameters give sigma = 0.2175 over 66 runs, with spreads to 15.2x. The two ranges do not overlap: the largest attention-free cell sigma, 0.051, lies below the smallest transformer cell sigma, 0.092. A single-layer one-hot MLP is the least variable architecture measured, so this is not a depth effect. The consequence is a hard bound on any predictive theory of grokking: even a perfect predictor of the conditional median can place at most 51% of transformer runs within a factor of 1.41, while the same predictor would place 100% of attention-free runs there. Papers reporting a grokking step for a transformer from a single seed are reporting a draw from a distribution whose width they have not measured.