Predicting the Grokking Step from Weight-Norm Dynamics: A Falsifiable, Pre-Registered Test

This paper proposes a lightweight, mechanistically motivated theory for predicting the grokking step from weight-norm relaxation dynamics under weight decay, expressed as a closed-form formula t_grok = t_mem + (1/(eta*lambda))*ln(||W_mem||/||W_gen||). The authors accompany the theory with a genuinely commendable pre-registration protocol: a stated tolerance band (factor-of-2 on log-step), explicitly held-out tasks and architectures not used for fitting, a fixed seed count, and pre-declared falsification criteria (below 60% in-band accuracy, or grokking under zero weight decay). This kind of falsifiable, registered-report style contribution is valuable and rare in the grokking literature, which has tended toward post-hoc, descriptive accounts.

However, the submission is explicitly labeled a 'stub' tied to a placeholder licence, and the empirical content is entirely absent: the prediction table has literal '(fill)' placeholders, and the text states outright that 'no results are claimed as collected here.' This means the paper's central and only falsifiable claim—that the theory predicts held-out grokking steps within the stated tolerance—cannot be assessed at all in its current form. A paper whose entire contribution is a predictive theory but which reports zero predictions or observations is, at best, a research plan or a Stage-1 registered report, not a completed research contribution suitable for standard peer-reviewed publication.

Beyond the missing data, there are several conceptual and methodological weaknesses that would need to be addressed even after the experiments are run. First, the derivation of exponential norm relaxation at rate eta*lambda is heuristic and not rigorously justified from the underlying gradient dynamics, especially once cross terms with the data-fitting gradient (which is not exactly zero post-memorization) are considered, or once adaptive optimizers are used (which the authors admit breaks the clean rate but do not offer a fallback formula for). Second, the notion of a 'task-independent norm threshold proportional to ||W_gen||' that triggers generalization onset is asserted without derivation or estimation protocol, and its stability across very different tasks (modular arithmetic vs. sparse parity, transformer vs. MLP) is exactly the kind of claim that needs empirical support the paper does not yet provide. Third, using a single calibration run per architecture family to fix the norm ratio is likely to be a fragile estimate, a limitation the authors themselves flag as a known failure mode but do not mitigate with any suggested robustness procedure (e.g., averaging over multiple short runs, jackknife estimates, or reporting calibration variance). Fourth, t_mem is treated as an externally given quantity rather than predicted, so it is unclear how much of the paper's predictive power derives from the novel norm-relaxation term versus simply reusing an empirically observed memorization time; a strong ablation would compare against naive baselines (e.g., predicting t_grok as a fixed multiple of t_mem) to isolate the value added by the specific weight-decay argument. Fifth, with only three held-out configurations and five seeds each, the statistical power behind the 60% pass criterion is very low, and no discussion of expected false-positive/false-negative rates under the null is given.

In sum, the theoretical framing and the discipline of stating falsification criteria in advance are strengths that should be encouraged, and the honesty about limitations (no grokking without weight decay, seed-instability degrading predictions, Adam incompatibility) is a model of good scientific practice for this subfield. But as submitted, the paper offers no evidence for or against its own central claim, several important modeling choices (the norm threshold, the calibration procedure, the treatment of t_mem) are underspecified or unjustified, and no comparison to existing accounts of grokking is provided. I recommend major revision: the authors should complete the calibration and held-out runs described in their own protocol, report the actual prediction table with observed steps, include sensitivity/robustness analysis for the calibration estimate, and compare against at least one naive or existing baseline theory before this can be evaluated as a completed empirical contribution.

This submission presents a lightweight theoretical proposal for predicting the grokking step from weight-norm relaxation dynamics under weight decay, packaged as a pre-registered, falsifiable protocol. The idea—that the delay between memorization and generalization is governed by an exponential norm-decay timescale set by eta*lambda, with the norm ratio calibrated from a single short run—has intuitive appeal and is stated in a commendably explicit, testable form, complete with a stated tolerance band, held-out tasks, seed counts, and falsification criteria (including the sharp, easily-checked prediction that zero weight decay should preclude grokking).

However, as the authors themselves state plainly, this is a stub submission: the prediction table is unfilled, no empirical validation has been performed, and the abstract explicitly disclaims that any results are yet collected. This is the central and disqualifying issue for evaluation as a scientific contribution: there is no evidence presented that the theory works, fails, or even produces sensible numbers when the formula is applied to the stated configurations. A pre-registration document has scientific value only when submitted alongside or as a precursor to the actual results; as a standalone paper it cannot be assessed on its empirical merits because there are none to assess.

Beyond the missing data, the theoretical development itself is under-specified. The two-phase story (rapid memorization, then exponential norm relaxation toward a task-independent generalization threshold) is stated as a modeling assumption without derivation from the underlying training dynamics, loss landscape, or known analyses of grokking (e.g., how weight decay interacts with the direction along which memorized vs. generalizing solutions differ). The claim that a generalization-norm threshold is task-independent across radically different tasks (modular arithmetic, sparse parity) and architectures (transformer, MLP) is a strong and unsupported assumption that will likely be the dominant source of prediction error, yet it receives no justification or sensitivity analysis. The reliance on a single calibration run per architecture family to estimate a ratio the paper itself admits is seed-sensitive further undermines confidence in the eventual predictions. Critically, essentially all major empirical grokking work uses Adam/AdamW, for which the simple eta*lambda decay-rate assumption is explicitly acknowledged to break down, yet no adapted formula or empirical substitute is pre-registered for this common and important setting, limiting the practical applicability of the theory even once tested.

The honesty and transparency of the limitations section, and the explicit falsification design, are genuine strengths reflecting good scientific practice, and the writing is clear. But as submitted, the paper offers no results, thin theoretical justification, and a protocol whose success or failure cannot currently be evaluated. I recommend rejection in its current stub form; the work would need to be resubmitted with the pre-registered experiments actually completed (with results, not schema) and ideally with a more principled derivation of the exponential-relaxation assumption and the task-independent threshold claim before it can be judged as a scientific contribution.