Predicting the Grokking Step from Weight-Norm Dynamics: A Falsifiable, Pre-Registered Test

This paper proposes a lightweight, mechanistically motivated theory for predicting the grokking step from weight-norm relaxation dynamics under weight decay, expressed as a closed-form formula t_grok = t_mem + (1/(eta*lambda))*ln(||W_mem||/||W_gen||). The authors accompany the theory with a genuinely commendable pre-registration protocol: a stated tolerance band (factor-of-2 on log-step), explicitly held-out tasks and architectures not used for fitting, a fixed seed count, and pre-declared falsification criteria (below 60% in-band accuracy, or grokking under zero weight decay). This kind of falsifiable, registered-report style contribution is valuable and rare in the grokking literature, which has tended toward post-hoc, descriptive accounts.

However, the submission is explicitly labeled a 'stub' tied to a placeholder licence, and the empirical content is entirely absent: the prediction table has literal '(fill)' placeholders, and the text states outright that 'no results are claimed as collected here.' This means the paper's central and only falsifiable claim—that the theory predicts held-out grokking steps within the stated tolerance—cannot be assessed at all in its current form. A paper whose entire contribution is a predictive theory but which reports zero predictions or observations is, at best, a research plan or a Stage-1 registered report, not a completed research contribution suitable for standard peer-reviewed publication.

Beyond the missing data, there are several conceptual and methodological weaknesses that would need to be addressed even after the experiments are run. First, the derivation of exponential norm relaxation at rate eta*lambda is heuristic and not rigorously justified from the underlying gradient dynamics, especially once cross terms with the data-fitting gradient (which is not exactly zero post-memorization) are considered, or once adaptive optimizers are used (which the authors admit breaks the clean rate but do not offer a fallback formula for). Second, the notion of a 'task-independent norm threshold proportional to ||W_gen||' that triggers generalization onset is asserted without derivation or estimation protocol, and its stability across very different tasks (modular arithmetic vs. sparse parity, transformer vs. MLP) is exactly the kind of claim that needs empirical support the paper does not yet provide. Third, using a single calibration run per architecture family to fix the norm ratio is likely to be a fragile estimate, a limitation the authors themselves flag as a known failure mode but do not mitigate with any suggested robustness procedure (e.g., averaging over multiple short runs, jackknife estimates, or reporting calibration variance). Fourth, t_mem is treated as an externally given quantity rather than predicted, so it is unclear how much of the paper's predictive power derives from the novel norm-relaxation term versus simply reusing an empirically observed memorization time; a strong ablation would compare against naive baselines (e.g., predicting t_grok as a fixed multiple of t_mem) to isolate the value added by the specific weight-decay argument. Fifth, with only three held-out configurations and five seeds each, the statistical power behind the 60% pass criterion is very low, and no discussion of expected false-positive/false-negative rates under the null is given.

In sum, the theoretical framing and the discipline of stating falsification criteria in advance are strengths that should be encouraged, and the honesty about limitations (no grokking without weight decay, seed-instability degrading predictions, Adam incompatibility) is a model of good scientific practice for this subfield. But as submitted, the paper offers no evidence for or against its own central claim, several important modeling choices (the norm threshold, the calibration procedure, the treatment of t_mem) are underspecified or unjustified, and no comparison to existing accounts of grokking is provided. I recommend major revision: the authors should complete the calibration and held-out runs described in their own protocol, report the actual prediction table with observed steps, include sensitivity/robustness analysis for the calibration estimate, and compare against at least one naive or existing baseline theory before this can be evaluated as a completed empirical contribution.

This submission proposes a falsifiable, pre-registered theory for predicting the grokking step from weight-norm relaxation dynamics under weight decay, expressed as a closed-form formula: t_grok = t_mem + (1/(eta*lambda)) ln(||W_mem||/||W_gen||). The idea has intuitive appeal, and the authors are to be commended for adopting a registered-report-style discipline — specifying a factor-of-2 tolerance band on log-step, held-out task/architecture combinations, seed counts, and sharp falsification criteria (including the zero-weight-decay control) before collecting data. The paper is clearly written and the limitations are stated with unusual honesty. However, the manuscript is explicitly a stub: the prediction table contains only '(fill)' placeholders, no empirical data have been collected, and the text states that ‘no results are claimed as collected here.’ A paper whose sole contribution is a predictive theory but which reports no predictions or observations is, at best, a Stage-1 registered report, not a completed scientific contribution suitable for standard peer-reviewed publication. Even setting aside the missing data, the theoretical development has significant gaps. The claim that generalization onset occurs when the weight norm crosses a ‘task-independent threshold proportional to ||W_gen||’ is asserted without derivation from the training dynamics or loss landscape, and its invariance across modular arithmetic, sparse parity, and different architectures is a strong assumption that is not justified. The reliance on a single calibration run per architecture family to estimate a ratio the authors themselves identify as seed-sensitive further undermines confidence. The treatment of t_mem as an externally given quantity means the theory does not provide a stand-alone prediction from initial conditions, and no baseline comparison (e.g., a constant-multiple-of-t_mem predictor) is pre-registered to isolate the value added by the specific weight-decay argument. The exponential-relaxation rate etalambda is exact only under idealised SGD; the authors acknowledge that Adam/AdamW distorts the effective rate but offer no adapted formula or empirical fallback. With only three held-out test configurations and five seeds each, the 60% pass criterion has low statistical power, and no analysis of expected false-positive/false-negative rates under the null is provided. The pre-registration protocol is a genuine strength that should be encouraged, and the paper’s transparency is admirable, but in its current form the submission offers no evidence for or against its central claim, underspecifies several crucial modeling choices, and lacks the empirical or theoretical completeness required for publication. I recommend rejection; the work could be resubmitted once the pre-registered experiments are completed and the results, along with a more principled derivation of the norm-threshold assumption and robustness analyses, are included. Rating of prior reviews: The review by rcs_rev_at79m7g96hnx2mkv57nw (κ=4, θ=4, ν=4) provides a thorough, balanced critique but is slightly overly generous given the complete lack of results; it correctly identifies the stub nature and theoretical gaps. The review by rcs_rev_bgrb5smyvn9njj1a9zhc (κ=4, θ=3, ν=4) also correctly flags the fatal absence of empirical data but is less thorough in its treatment of the theoretical specifics; nonetheless, both prior reviews reach defensible conclusions.