This paper measures something the grokking literature has needed and skipped: it treats t_grok as a random variable and estimates its within-configuration dispersion. The core object is right — replicate cells fixing task, architecture, every hyperparameter and split_seed, varying only init_seed — and the hygiene is better than typical. The harness is released and content-addressed, and the determinism check (same configuration twice gives a bitwise-identical trace) is precisely the control that makes a seed-variance claim meaningful rather than a report of nondeterministic kernels. Section 5 is unusually candid, and the note that an exponential waiting time would force sd(log10 t) = 0.557 universally, which no cell approaches, is a correct and informative negative check (I verified (pi/sqrt6)/ln10 = 0.557). My difficulty is that the title, abstract and Section 4 make claims of escalating strength and the evidence supports only the weakest.
Three claims must be separated. (a) t_grok varies across seeds: established, and the Section 3.2 trace analysis — common 0.57 plateau, training accuracy pinned at 1.0 from step ~130, same sigmoid shape, runs differing only in onset — is the right way to do it and does rule out threshold-hovering. (b) Attention-bearing architectures show larger dispersion than the two MLPs: supported as a correlation, and the paper is stronger here than it argues. Two settings match families on task and hyperparameters exactly — modadd p=31 / 0.60 / 9.0e-3 gives onehot_mlp 0.021, embed_mlp 0.051, transformer 0.359; 0.50 / 3.0e-3 gives embed_mlp 0.039 against transformer 0.130 and 0.243. That the embedding MLP is also low is a real control, ruling out "has an embedding layer". (c) "Attention sets its variance": this is causal language and nothing here is an intervention. The transformer differs from the MLPs in attention but also in softmax mixing over positions, causal masking, positional structure, tokenised input, output head, parameter count, and the absence of LayerNorm. The paper's own Section 5 hypothesis — a symmetry-breaking choice of which position a head reads — names the experiment it declines to run. Two within-architecture interventions would settle it and are affordable at 85 CPU-scale runs: seed the attention parameters while holding the MLP/embedding initialisation fixed and vice versa, decomposing variance by module; and replace attention with a static or uniform mixing matrix in an otherwise identical transformer. Absent that, the defensible title is "attention-bearing architectures have higher grokking-time variance."
Statistical support is thinner than presented, because a variance needs far more replicates than a mean. Cells have n = 3 to 10 and no confidence interval appears anywhere in a paper whose subject is a variance. The chi-square interval for sigma at n=3 spans roughly a factor 0.52 to 6.3, so the 0.118 cell is compatible with 0.061 to 0.74. This undermines the load-bearing sentence "The ranges are disjoint... There is no overlap and no ambiguous case." That compares two extremum statistics — max of four estimates against min of eleven — the quantities most sensitive to sampling noise. The 0.051 cell (n=6) has a 95% interval of roughly [0.032, 0.125] and the 0.092 cell (n=5) roughly [0.055, 0.264]; these overlap heavily. The group-level contrast does survive, and the paper should have argued that instead: a permutation test over the 15 cell sigmas, asking how often a random 4-of-15 split puts all four attention-free cells lowest, gives p = 1/C(15,4) = 7.3e-4. That test is not run.
I could not reproduce the pooled figures from the tabulated cell sigmas. Under the standard df-weighted pooled estimator I get 0.236 for transformers and 0.042 for attention-free, against the reported 0.2175 and 0.0382. Both gaps track sqrt(N/(N-k)) — 1.095 and 1.125 — consistent with dividing the sum of squares by total runs rather than degrees of freedom. Rounding of three-decimal cell sigmas cannot produce an 8-10% gap in the same direction twice. This deflates both groups so the contrast survives, but a variance paper should use and name an unbiased estimator.
There is also an internal arithmetic contradiction. Section 3.2 gives the worst cell's runs as 307, 636, 636, 1370, 2529, 5239; 5239/307 = 17.06, but the table reports max/min = 15.2 for that cell, and since the cell holds ten runs its true ratio can only exceed 17.06. Either the six values, the tabulated 15.2, or the cell identification is wrong. Relatedly, "Reporting t_grok = 3308... when the cell median is 689" borrows 3308 from a different row's median.
Measurement and censoring are under-reported. Robustness at theta = 0.5 through 0.99 is asserted "of the same magnitude" with no table, and magnitude is not the criterion — what must survive is the ordering between families. More seriously, "cells with at least three grokked runs are used" selects on the outcome. The step budget is never stated, runs attempted are never given, and no count of non-grokked runs appears, so right-censored data is analysed as complete. The direction plausibly favours the paper — censoring should truncate the transformer tail harder — but readers cannot check without the denominator. Report attempted/grokked per cell, or treat this as survival analysis. Grid density is likewise unstated, though two runs sharing 636 shows it produces ties; the smallest sigma (0.021) may sit near the quantisation floor.
The analysis is entirely exploratory and should say so. Cells were selected post hoc from a corpus assembled for another purpose — n ranging 3 to 10 is the signature of opportunistic mining, not a designed replicate study — and there is no confirmatory hold-out. The pre-registration in Section 4 belongs to a different study, and this measurement is used to retroactively excuse that study's missed target, exactly the configuration where a sealed set matters most. If this corpus later feeds a predictive model, splits must be by configuration, never by run: seeds within a cell are replicates and a row-wise split leaks. Two rows also carry identical listed configurations (transformer, modadd p=31, 0.50, 3.0e-3, 1 layer) yet differ in n, median and sigma (0.130 vs 0.243) — so the displayed columns do not identify a cell, and sigma varies by 1.9x across nominally identical settings, undercutting "transformer sigma" as a single parameter.
Section 4 compounds this. The 51% figure requires log-normality, which Section 5 explicitly disclaims ("85 runs across 15 cells constrain a width, not a tail"); the erf arithmetic is correct given normality, but normality is the contested part. It also pools cells spanning 0.092 to 0.359. A predictor conditioning on hyperparameters faces the per-cell spread: against sigma = 0.092 the ceiling at +/-0.15 is about 90%, not 51%. The principle is right; the number is an artefact of pooling heterogeneous cells, and "bounds every predictive theory of grokking, including ones not yet written" is overreach.
On scores: novelty 6, since seed sensitivity is folklore but measuring the width, contrasting it across architectures and converting it into a predictor ceiling is a fresh framing rather than a new primitive. Rigour 4 — real runs, released code and a determinism check, against no confidence intervals in a variance paper, n as low as 3, unreproducible pooled statistics, an internal numerical contradiction, unreported censoring, post-hoc cell selection with no confirmatory set, and a causal title with no intervention behind it. Clarity 7: well organised and mostly re-implementable, held back by unstated grid density and budget, a table whose columns do not identify its rows, and a promised robustness result with no numbers. Significance 6: the qualitative finding should change reporting practice in a subfield that routinely quotes single-seed grokking steps, but scope is three architectures, one optimiser and toy algorithmic tasks.