1. What is being replicated, and why this particular thing
Grokking — generalization arriving long after the training set has been fit — was reported by Power et al. (2022) and has attracted a family of competing explanations. Among these, Omnigrok (Liu, Michaud & Tegmark, ICLR 2023; arXiv:2210.01117) is unusually attractive because it is quantitative. Its "LU mechanism" says that reduced training loss is L-shaped and reduced test loss U-shaped against the weight norm w, that generalizing solutions live near a critical norm w_c set by the loss landscape, and that grokking is simply the time taken for regularization to drag an over-large norm down to w_c. The paper's own derivation is explicit:
"weight decay gamma causes w(t) ≈ exp(−gamma t) w_0, when w_0 > w_c, so it takes t ≈ ln(w_0/w_c)/gamma to generalize."
This is a real, falsifiable mechanism, and the paper tests one of its consequences: t ∝ gamma^−1, shown in Appendix C for a transformer on modular addition and an MLP on MNIST.
But the derivation entails more than the scaling law, and the additional entailments are strictly sharper because they are parameter-free. Under AdamW's decoupled weight decay (Loshchilov & Hutter, 2019) the per-step multiplicative factor is (1 − eta*gamma), so on a post-overfitting plateau where the training loss and its gradient are negligible the mechanism requires:
- (C3a) d log w / dt = −eta*gamma throughout the plateau;
- (C3b) w at the moment of generalization equals w_c, which is a property of the data and architecture and therefore independent of gamma.
Neither is tested in the original paper. They are the discriminating tests, because a world in which the scaling law holds but C3a and C3b fail is a world where Omnigrok has found a real regularity and attached the wrong explanation to it. That is the possibility this replication was designed to detect, and it is what we found.
We also test the paper's flanking claims: that memorization time is independent of gamma (Fig. 2c), that gamma = 0 produces no generalization at all, and that "small initializations w < w_c can always generalize fast, regardless of regularization."
2. Pre-registration
The full protocol — configuration, sweep grid, test statistics, and numeric decision thresholds — was written and hashed before any confirmatory run.
PREREGISTRATION.txt SHA-256 cbdd2c73511ccf26b94010a66bb71f29f280066a4f9a0b638389744cfcc0c887
grok.py SHA-256 bfa65c16ade842ceba28f2141525f3aced058c64734229e521047698e414a219
frozen at 2026-08-07 12:01:49 UTC
A pilot was run before the pre-registration, on p=31 with train_frac ∈ {0.5,…,0.9}, d ∈ {128,256}, eta ∈ {1e-3, 3e-3}, gamma=1, alpha=1, seed 0 only, for the sole purpose of finding a cell that groks within the compute budget. Pilot data are used as evidence for nothing; the selected cell was re-run from scratch. This is disclosed in §5 of the protocol and repeated here because a replication that hid its pilot would be worth little.
Setup. Modular addition mod p = 31 (all 961 pairs, 70/30 split). Two-layer ReLU MLP without biases: x = [onehot(a); onehot(b)] ∈ R^62 → W1 (62×256) → ReLU → W2 (256×31). Softmax cross-entropy, full batch. AdamW, eta = 1e-3, betas (0.9, 0.98), eps 1e-8, decoupled weight decay gamma. Initialization U(±1/√fan_in) scaled by alpha. w = ‖[W1, W2]‖₂. t_mem = first step with train accuracy ≥ 0.99; t_gen = first step with test accuracy ≥ 0.95. Logging every 25 steps, budget 60000 steps, three seeds per cell.
MLPs of this kind grok reliably on modular arithmetic (Gromov, 2023), which is why one suffices here. This is a conceptual replication: Omnigrok used a 1-layer transformer and an MLP on MNIST, so a failure of the empirical scaling claim here would not by itself refute it for transformers. C3a and C3b are different — they follow from weight-norm dynamics alone, with no architectural content, and are therefore testable wherever grokking occurs.
Implementation checks before any run. Analytic gradients were verified against central finite differences in float64 (worst relative error 3.5e-5 over 226 randomly chosen coordinates). The decoupled decay was verified to reproduce (1 − eta*gamma)^t exactly on a zero-gradient parameter.
3. Results
3.1 Sweep A: weight decay (alpha = 1, 21 runs)
Cell means over three seeds. w_mem is the norm at t_mem.
| gamma | t_mem | t_gen | w_mem | w_peak | w_gen |
|---|
| 0.1 | 225 | 18300 | 37.3 | 164.6 | 143.3 |
| 0.2 | 225 | 8933 | 36.9 | 88.7 | 78.8 |
| 0.3 | 225 | 6450 | 36.5 | 86.3 | 75.7 |
| 0.5 | 225 | 4425 | 35.7 | 82.4 | 73.8 |
| 1.0 | 242 | 2958 | 35.4 | 75.1 | 71.7 |
| 2.0 | 258 | 3117 | 32.8 | 64.9 | 63.5 |
| 3.0 | 300 | 6350 | 31.3 | 48.5 | 48.3 |
T1 (C1: t_gen ∝ gamma^−1) — FAILS as pre-registered. OLS on log–log gives b_gen = −0.379 ± 0.081 (R² = 0.538), outside the pre-registered window [−1.20, −0.80]. The fit is poor because the relation is not a power law at all: t_gen is non-monotonic, falling to a minimum near gamma ≈ 1 and rising again by a factor of two by gamma = 3.
Omnigrok's footnote 3 anticipates part of this — "gamma should not be too large, otherwise it will bring the weights to a trivial solution" — so the honest reading restricts attention to the monotone branch. Restricted to gamma ≤ 1 the fit is excellent and the exponent is b_gen = −0.786 ± 0.036 (R² = 0.974). This is a strong regularity, and it is not gamma^−1: −0.786 sits six standard errors from −1. Robustness at other thresholds: −0.437 ± 0.058 (0.90), −0.435 ± 0.066 (0.99, 13/21 cells reaching it). The qualitative claim "smaller weight decay means much longer delay" replicates; the specific exponent does not.
T2 (C2: t_mem independent of gamma) — REPLICATES. b_mem = +0.076 ± 0.012, inside the pre-registered |b| ≤ 0.20. Memorization takes 225–300 steps across a 30-fold range of gamma.
T3 (C3b: w_gen is a gamma-independent constant) — FAILS. b_w = −0.232 ± 0.030 (R² = 0.754), far outside |b| ≤ 0.10, with Spearman −0.991 — a near-perfect monotone dependence. w_gen falls from 143 to 48 as gamma rises from 0.1 to 3. Within each cell it is extremely tight (e.g. 71.7 ± 0.4 at gamma = 1). So the norm at generalization is a sharply reproducible quantity that is set by the optimizer's regularization strength, not a landscape constant that regularization merely transports the model to.
T4 (C3a: plateau log-norm decays at eta*gamma) — FAILS, 0/21. As pre-registered the window [t_mem, t_gen] gives a median ratio of −0.081: the sign is negative because over that window the norm is growing, not decaying. Because the pre-registered window turns out to straddle the norm's rise and fall, we also report the more charitable window [t_peak, t_gen], which isolates the decay phase and exists in only 12 of 21 runs; there the median ratio is 0.083, still an order of magnitude too slow, and still 0/21 inside [0.5, 2.0]. The mechanism fails on both the strict and the generous reading.
T5 (C3: mechanism's time budget) — FAILS, 0/21. The pre-registered ratio is undefined in 9 runs because t_peak ≥ t_gen — the norm has not even begun to fall when generalization occurs. Using the always-defined form, ln(w_mem/w_gen)/(eta*gamma) against t_gen − t_mem, the ratio is negative in 21 of 21 runs, because w_gen > w_mem everywhere. The mechanism predicts a negative delay in every single run.
This last point is the core empirical finding and deserves stating plainly. In this setting the network memorizes at a norm of ≈ 35, the norm then grows through the plateau to a peak of 48–165, and generalization occurs at a norm of 48–143 — always above the memorization norm. Grokking here is not a descent to a critical norm. It coincides with the arrest and mild reversal of norm growth.
3.2 Control: gamma = 0 (3 runs, 60000 steps)
T6 (C4) — HOLDS. No run generalizes: best test accuracy 0.181–0.229, final 0.003–0.014, while training accuracy is reached at step 200–225 and the weight norm grows without bound to ≈ 2200. Weight decay is genuinely necessary here. Note this also rules out the possibility that our MLP simply generalizes for free.
3.3 Sweep B: initialization scale (gamma = 1, 15 runs + 6 exploratory)
| alpha | w_0 | t_mem | t_gen | t_gen/t_mem | w_gen |
|---|
| 0.3 | 2.9 | 242 | 2808 | 11.6 | 71.5 |
| 0.5 | 4.9 | 233 | 2833 | 12.2 | 71.5 |
| 1.0 | 9.8 | 242 | 2958 | 12.3 | 71.7 |
| 2.0 | 19.5 | 242 | 3208 | 13.3 | 72.0 |
| 4.0 | 39.1 | 258 | 3567 | 13.8 | 72.0 |
| 8.0 | 78.2 | 308 | 3975 | 12.9 | 72.2 |
| 12.0 | 117.3 | 367 | 4300 | 11.7 | 72.3 |
(Italic rows are exploratory, added after the confirmatory runs; see §4.)
T7 (C5: small init generalizes fast regardless of regularization) — FAILS. At alpha = 0.3 the initial norm is 2.9, a factor of 25 below w_c ≈ 72, and the model nevertheless groks with t_gen/t_mem = 11.6, far outside the pre-registered ≤ 2. It groks essentially as slowly as alpha = 12, whose initial norm is 40 times larger.
The reason is visible in the trajectories: whatever the initialization, the norm is driven during memorization to the same neighbourhood and peaks at 74–79. The initial norm is almost entirely forgotten. "Did you initialize above or below w_c" is therefore not the variable governing the dynamics in this setting.
w_c is real, but it is not what the mechanism says it is. Across a 40-fold range of w_0, w_gen = 71.90 ± 0.43. That is a striking invariant and it supports the existence of a well-defined critical norm at fixed gamma. But the same quantity moves from 143 to 48 under gamma (§3.1). A quantity that ignores initialization but tracks the regularizer is not a property of the loss landscape.
3.4 The mechanism in its own home regime
A fair objection to §3.3 is that the confirmatory grid never reached w_0 > w_c, which is precisely the regime the LU derivation addresses. We therefore added alpha ∈ {8, 12} (w_0 = 78 and 117, against w_c = 71.9). At alpha = 12 the norm does decay monotonically from initialization, exactly as the mechanism describes. Regressing across the full alpha range:
observed : t_gen = 3916 + 408 * ln(w_0/w_c) (R^2 = 0.856)
LU : t_gen = 0 + 1000 * ln(w_0/w_c)
The slope is 2.5× too shallow, and — far more importantly — there is a ~3900-step intercept that the mechanism assigns zero. At w_0 = w_c the mechanism predicts no delay whatsoever; the observed delay is 3900 steps. Even at alpha = 12, deep inside the regime the derivation was built for, norm decay accounts for 11.8% of the observed delay (2.1% at alpha = 8).
So the mechanism is not merely mis-scaled. It captures a genuine second-order effect — raising the initial norm does lengthen grokking, and roughly in the right direction — while missing the entire first-order phenomenon.
4. Deviations, limitations, and what would overturn this
Deviations. One: the alpha ∈ {8, 12} runs in §3.4 are exploratory, added after the confirmatory sweep, and are excluded from every pre-registered test (T7 is decided only on alpha ∈ {0.3, 0.5} as registered, and fails there). They are reported because omitting the regime most favourable to the hypothesis would be a worse sin than declaring an extension. Two: T4's pre-registered window proved ill-chosen; both the registered result and the charitable variant are reported, and both fail.
Limitations. (i) One architecture, one task, one p. C1 and C5 are empirical claims that could behave differently for transformers, and we do not claim to have refuted them there. (ii) In Omnigrok's transformer (their Fig. 7a) the norm at generalization drops below the initialization norm; in our MLP it does not, because our standard init is small relative to the solution norm. This is a genuine setting difference and it is why §3.4 exists — pushing w_0 above w_c reproduces their qualitative picture and still leaves ~88% of the delay unexplained. (iii) t_gen is threshold-defined; we report three thresholds. (iv) Three seeds per cell is few, though within-cell spread is small relative to the effects (e.g. w_gen ± 0.4 against a 3× range across gamma).
What would overturn this. A demonstration that in a transformer at p = 113 the plateau log-norm decays at eta*gamma within a factor of two, and that w_gen is gamma-independent, would confine our result to MLPs and restore the mechanism in its original setting. We consider that the most valuable follow-up, and note it costs a single sweep.
5. Interpretation
Two claims of Omnigrok replicate cleanly (gamma-independence of memorization; no generalization without regularization). One replicates in weakened form: delay does fall steeply with weight decay over gamma ∈ [0.1, 1], but as gamma^−0.79 rather than gamma^−1, and the relation reverses above gamma ≈ 1. The mechanism advanced to explain it does not survive on any test we could construct.
What the data suggest instead — offered as a hypothesis, not a result — is a ceiling rather than a descent. With gamma = 0 the norm grows without bound and generalization never arrives. Weight decay caps that growth; the larger gamma, the lower the ceiling (w_peak: 165 → 48 as gamma goes 0.1 → 3) and the sooner growth is arrested. Generalization coincides with arrest, not with arrival at a landscape-determined sphere. And the upturn at gamma = 3 is what a ceiling picture predicts and a descent picture does not: too tight a cap starts to obstruct the generalizing solution itself, which is consistent with test accuracy at gamma = 3 barely clearing threshold (0.955–0.965) where every other cell reaches 0.99–1.00.
This sits naturally beside Kumar et al. (2024), who dispute Omnigrok's norm story on independent theoretical grounds — agreeing that initial weight norm can induce grokking while denying that generalizing solutions lie on smaller-norm spheres. Our §3.1 is a direct empirical statement of their objection: the generalizing solution here sits at a larger norm than the memorizing one. It is also compatible with the circuit-efficiency account of Varma et al. (2023), in which the relevant competition is between memorizing and generalizing circuits with weight decay setting the exchange rate, rather than between norms.
The narrower methodological point is worth separating from the grokking-specific one. Omnigrok's gamma^−1 plot is real and reproduces qualitatively; had we tested only that, we would have logged a successful replication. The mechanism failed only under the two entailments the original paper did not plot. Deriving a mechanism's parameter-free consequences and testing those, rather than the scaling law it was introduced to explain, is cheap and discriminating, and we would encourage it as a default move in replication work.
6. Reproducibility
45 runs, all on one CPU core in under an hour of compute, pure NumPy with no framework dependency. Released: PREREGISTRATION.txt (hashed pre-run), grok.py (training, hashed pre-run), confirm.py and control.py (sweep drivers), analyse.py (implements T1–T7 exactly as registered), and per-run JSON logs containing the full accuracy and weight-norm trace at 25-step resolution for every run. Every number in this paper is recomputable from those logs by python analyse.py.