The Regularized-Gate Clock: Exact Step Prediction for Delayed Generalization in Mixture-of-Paths Models

1. Claim, status, and scope

Grokking is the delayed appearance of generalization long after a model has fit its training set [@power2022]. A useful theory must do more than name a mechanism after inspecting a curve: it must produce the transition step before the run, state a narrow tolerance, and survive configurations that were not used to select the theory.

This paper derives such a predictor for one explicit architecture class: a smooth gate mixes a high-capacity, training-example memory path with a lower-dimensional rule path shared by train and test inputs. The result is exact within the registered model, optimizer, task generator, and numerical semantics. It is not a claim about arbitrary neural networks. In particular, it does not establish that the same clock governs ungated transformers, minibatch SGD, Adam, or representation discovery outside a fixed rule dictionary.

The scientific result has two parts. First, the full training dynamics admit an exact low-dimensional recurrence. This is a theorem, not a fitted surrogate. Second, those dynamics exhibit a slow-fast ignition: memory fits rapidly, regularization creates a persistent residual, gate decay seeds the shared rule, and a learned gate-rule feedback causes rule takeover. Continuous held-out risk can transition sharply even though neither accuracy quantization nor a hard discontinuity is used.

The empirical status is equally important. All numerical results in this article are development observations. The final five-family, three-architecture registry was specified but not executed at the time of submission. We therefore report an exact theory, independent implementation checks, development evidence, and a falsifiable prospective protocol—not a completed held-out validation. This disclosure prevents development results from being misrepresented as evidence for the registered holdout.

2. Model

Let Phi_T in R^(n x K) be the training rule matrix and Phi_H a held-out rule matrix. The target is generated by a public rule vector theta, so y_T=Phi_T theta and y_H=Phi_H theta. There are three trainable components:

1. a shared rule coefficient q in R^K;
2. one memory parameter m_i for each training example, with no corresponding test parameter;
3. a scalar gate logit z.

For gate power p, slope beta, and B^2=mu n, define

s(z) = sigmoid(beta z),        g_p(z) = s(z)^p,
f_T  = (1-g_p) B m + g_p Phi_T q,
f_H  = g_p Phi_H q.

The objective is

J = ||f_T-y_T||^2/(2n)
    + lambda_q ||q||^2/2
    + lambda_m ||m||^2/2
    + lambda_z z^2/2.

We use simultaneous full-batch gradient descent: every gradient is evaluated at the same pre-update state. Writing r=f_T-y_T and

g'_p(z) = p beta g_p(z) [1-s(z)],

the exact full updates are

q+ = (1-eta lambda_q)q - eta g_p Phi_T^T r/n,
m+ = (1-eta lambda_m)m - eta (1-g_p)B r/n,
z+ = z - eta {g'_p <Phi_T q-Bm,r>/n + lambda_z z}.

Only training residuals appear in these updates. The held-out matrix and labels are used solely to score a prediction after the optimizer state is advanced. Appending or removing held-out identities cannot change training, because no held-out memory coordinate exists.

3. Exact reduced predictor

Assume m_0=0 and the target lies in the rule span. Define a coefficient a by Bm=Phi_T a. This relationship is preserved exactly by the full update. Let

G = Phi_T^T Phi_T/n,
e = (1-g_p)a + g_p q - theta.

Then the full (K+n+1)-parameter system closes on (q,a,z):

q+ = (1-eta lambda_q)q - eta g_p G e,
a+ = (1-eta lambda_m)a - eta mu(1-g_p)e,
z+ = z - eta {g'_p (q-a)^T G e + lambda_z z}.

This recurrence is the predictor. Its inputs are fixed before training: the task and split, Phi_T, theta, initialization, architecture, optimizer, and horizon. It has no learned clock constant, no checkpoint-derived order parameter, and no per-task refit. Evaluating g_p Phi_H q supplies a step-by-step held-out-risk forecast, but the held-out population never affects the state update.

The reduction is exact because the memory update is a linear combination of its previous value and the training residual, both of which remain in the column space of Phi_T. Substitution into the full gradient gives the three equations above. Thus equality between the reduced and full implementations is an algebraic prediction of the theorem; it is a software-equivalence check, not itself evidence of transfer.

For targets outside the rule span, memory remains in span(Phi_T,y_T). Writing Bm=Phi_T a+c y_T yields an exact 2K+2 closure. Development binary sign teachers under that extension were generally gradual or censored and are treated as negative controls, not as positive evidence.

4. Mechanism and approximate clock

The exact recurrence gives integer predictions. A slow-manifold approximation explains why it can produce a delayed, sharp transition. In a scalar RMS-one mode, the memory logit is rapidly attracted near

m_bar(g,q) = mu(1-g)(1-gq) / [mu(1-g)^2 + lambda_m].

Substitution into gradient flow gives

dq/dtau = g[1-(1-g)m_bar-gq] - lambda_q q,
dz/dtau = g'(m_bar-q)[1-(1-g)m_bar-gq] - lambda_z z.

Initially z_0<0 and beta|z_0| is large, so g is exponentially small. Memory learns quickly while the rule is suppressed. Memory decay prevents a zero training residual and gate decay slowly moves z toward zero. The resulting small rule update seeds q. Eventually the loss-gradient term in z and the growing q reinforce one another, causing rapid gate and rule takeover. This is a trajectory-dependent ignition, not a fixed-state saddle-node claim.

An interpretable approximate clock follows by defining the right-hand side of the z equation as F(z,q), solving dq/dz=(dq/dtau)/F, and integrating

t(z) approximately eta^(-1) integral[z_0 to z] du/F(u,q(u)).

The integral exposes the decay and feedback timescales, but all registered predictions use the exact discrete recurrence.

5. Fixed point and stability certificate

Let gamma=2^(-p). With lambda_q=0, the zero-risk point is

q*=theta/gamma=2^p theta,   m*=0,   z*=0.

It is stationary even when lambda_m and lambda_z are positive because their parameters vanish at the point. In a scalar mode, the exact Hessian is

H = v v^T + diag(lambda_q,lambda_m,lambda_z),
v = (gamma, 1-gamma, p beta/2).

In the full model, the residual-Jacobian blocks are

J_q = gamma Phi_T,
J_m = (1-gamma)sqrt(mu n) I_n,
J_z = (p beta/2)y_T,
H = J^T J/n + diag(lambda_q I_K,lambda_m I_n,lambda_z).

Because the residual is zero, no residual-weighted second-derivative term is omitted. Plain simultaneous gradient descent is locally stable when 0<eta<2/lambda_max(H). The prospective protocol uses the stricter pre-run certificate eta lambda_max(H)<=1.5; a failing cell is recorded, not retuned.

The three registered architecture profiles are:

All use q_0=m_0=0, z_0=-2, 250,000 steps, and learned smooth gates.

6. A threshold-robust grokking endpoint

Accuracy can manufacture apparent jumps when many margins are identical or the test population is small. We therefore classify suddenness using continuous held-out mean-squared risk.

Let Y_T=mean_T(y^2). Memorization time t_mem is the first start of 500 consecutive steps with train MSE at most 0.01Y_T; any later violation is a train relapse. Because the teacher is in-span, the structural reference risk R_ref=0 is known analytically rather than estimated from the observed final curve. With E_mem=R_H(t_mem), define unclipped progress

P_t = [E_mem-R_H(t)]/[E_mem-R_ref].

For c in {.1,.5,.9}, T_c is the first start at or after t_mem of 500 consecutive steps with P>=c. Define

D = T_90-t_mem,        W = (T_90-T_10)/D.

A cell is genuine only when D>=max(1000,t_mem), W<=.25, no later train or test relapse occurs, and at least max(500,ceil(.25 T_90)) later steps remain. The horizon may not be extended. Failure precedence is fixed:

invalid hash/schema > nonfinite > no stable memorization > train relapse
> no stable T90 > insufficient tail > test relapse > not delayed
> not sudden > genuine.

The primary numeric target is T_90, with an absolute tolerance of two optimizer steps. The tolerance is intentionally severe because the theory predicts the same deterministic recurrence implemented by the trainer.

7. Development evidence and falsifiers

Two independently written engines were used: a NumPy implementation of the reduced recurrence and a PyTorch-autograd implementation with one explicit memory coordinate per training example. Development tests across all three gate powers agreed to atol=rtol=2e-10; randomized short trajectories had discrepancies far below that bound.

Stable long development predictions on dense rotated teachers were:

These configurations were inspected during development and are not held-out evidence.

The mechanism makes strong negative predictions. At a scalar p1 development point, the complete learned update gave T10/T50/T90=10089/12682/12985, W=.223. If the gate received weight decay only, the transition occurred at 922/1420/2983 with W=.691: early and gradual. With loss gradient only (lambda_z=0) or a frozen gate, the shared rule did not take over. Hence explicit exponential decay is neither sufficient nor interchangeable with learned feedback.

The boundary is also sharp. At nearby development settings, beta=12 gave W=.338, lambda_m=.006 gave W=.323, and lambda_z=.06 gave W=.302; all fail the frozen suddenness criterion. Initial q_0=.1 gave W=.263, while q_0=-2 did not reach the high-rule solution. Equal rule and memory decay often prevented 90% progress. Target rescaling changed both time and suddenness, so RMS normalization is part of the theory, not a cosmetic preprocessing choice.

These failures distinguish the theory from the assertion that any decaying nuisance path will eventually reveal a rule.

8. Prospective registry

The unexecuted registry crosses three gate powers with five task families never used in development:

1. 5-subsets of 10 with coordinate-membership rules (N=252);
2. set partitions of 7 with pair-co-block rules (N=877);
3. Dyck paths of semilength 7 with step and prefix-height rules (N=429);
4. perfect matchings of K_10 with edge-membership rules (N=945);
5. Prüfer trees on 6 labelled vertices with edge-membership rules (N=1296).

Each target is a hash-selected, normalized combination of five generic rules. Each family shares exactly one target and split across p1/p2/p3. Feature construction uses fixed-order scalar two-pass Gram-Schmidt; every array, split, source file, and prediction has a canonical SHA-256 representation. Rank failure, instability, nonfinite output, insufficient tail, or a cross-engine hash mismatch is a recorded failure with no replacement.

Before observations, the predictor must classify at least 10 of 15 cells as genuine, at least two per family, and at least three per architecture. Otherwise the prospective claim fails without running observations. If authorized, success requires exact category agreement in all 15 cells and |T90_observed-T90_predicted|<=2 for at least 14 of 15, at least two of three in every family, and at least four of five for each gate power. No configuration, threshold, horizon, or tolerance may change after predictions are sealed.

At submission this registry remained unexecuted. Reporting that fact is part of the protocol: a later prospective result can be evaluated without retroactively turning development decisions into preregistration.

9. Relation to prior work

Power et al. established the modern grokking phenomenon and its delayed train-test separation [@power2022]. Exact modal learning dynamics in deep linear networks show how plateaus and rapid transitions can arise from nonlinear factorization [@saxe2014], while neural tangent kernel theory gives exact linearized trajectories in a different fixed-feature regime [@jacot2018]. Mixtures of local experts provide the architectural ancestry for learned competition between specialist paths [@jacobs1991].

The present contribution is narrower and more operational: a path-competition model whose complete optimizer trajectory closes exactly, plus an integer-step endpoint and failure protocol. It differs from a post-hoc mechanism account because every quantity needed by the predictor is available before the run. It differs from a generic NTK forecast because the gate-rule coupling is nonlinear and causal to the sharp transition.

10. Limitations and conclusion

The strongest limitation is architectural construction. A smooth shared gate and asymmetric path regularization are explicit inductive biases. The rule dictionary is fixed and the teacher is realizable within it; the theory predicts selection and takeover, not the discovery of a representation absent at initialization. Full-batch deterministic optimization removes SGD escape phenomena. Exact zero shared initialization is consequential. These restrictions make the result falsifiable, but prevent a universal claim about grokking.

Within that scope, the theory is complete: the optimizer has an exact reduced recurrence, the transition has a slow-fast causal account, the event definition resists accuracy quantization, fixed-point stability is checked before the run, and the absolute error tolerance is two steps. Development evidence shows a nontrivial open regime and decisive negative controls. The remaining scientific question is prospective transfer to the frozen five-family registry. Until that run is performed, this is an exact predictive theory and preregistration, not a held-out-validated solution to grokking in general.

References

[@power2022] Power, A., Burda, Y., Edwards, H., Babuschkin, I., and Misra, V. (2022). “Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets.” arXiv:2201.02177.

[@saxe2014] Saxe, A. M., McClelland, J. L., and Ganguli, S. (2014). “Exact solutions to the nonlinear dynamics of learning in deep linear neural networks.” arXiv:1312.6120.

[@jacot2018] Jacot, A., Gabriel, F., and Hongler, C. (2018). “Neural Tangent Kernel: Convergence and Generalization in Neural Networks.” arXiv:1806.07572.

[@jacobs1991] Jacobs, R. A., Jordan, M. I., Nowlan, S. J., and Hinton, G. E. (1991). “Adaptive Mixtures of Local Experts.” Neural Computation 3(1):79–87. doi:10.1162/neco.1991.3.1.79.