Papers

Grokking is normally reported as a step number, as if delayed generalisation happened at a determinate time. It does not. Across 85 training runs that are identical in task, architecture, optimiser, hyperparameters and data split, and differ only in the initialisation seed, the grokking step scatters with a standard deviation of 0.19 in log10 -- a factor of 1.6 -- and in the worst cell by a factor of 15.2. The scatter is not an artefact of a threshold crossing: all runs in that cell show the same smooth sigmoid rise off a common test-accuracy plateau with training accuracy pinned at 1.0, and differ only in when it starts. The scatter is also not a property of grokking. It is a property of the architecture. Attention-free networks grok at a nearly determinate step, sigma = 0.0382 over 19 runs, with no cell exceeding a 1.4x spread; transformers on the same tasks at the same hyperparameters give sigma = 0.2175 over 66 runs, with spreads to 15.2x. The two ranges do not overlap: the largest attention-free cell sigma, 0.051, lies below the smallest transformer cell sigma, 0.092. A single-layer one-hot MLP is the least variable architecture measured, so this is not a depth effect. The consequence is a hard bound on any predictive theory of grokking: even a perfect predictor of the conditional median can place at most 51% of transformer runs within a factor of 1.41, while the same predictor would place 100% of attention-free runs there. Papers reporting a grokking step for a transformer from a single seed are reporting a draw from a distribution whose width they have not measured.