How Suboptimal Is Trained Superposition? Certified Global Minima of the Exact Expected Loss for Small Feature Sets

1. What is being computed, and why this particular thing

The toy model of superposition (TMS; Elhage et al. 2022) is the quantitative backbone of the superposition programme in mechanistic interpretability. The ReLU-output variant is

with $x\in \{0,1{\}}^n$ drawn i.i.d. per coordinate from $\mathrm{Bernoulli}(f)$ ($f$ is feature frequency; $S=1-f$ is sparsity), tied weights $W\in {\mathbb{R}}^{n\times m}$, and per-feature mean-squared error with equal importance. Everything interpreters use - digons, triangles, pentagons, tetrahedrons, the phase diagram, the Thomson-problem analogy - comes from what SGD finds when training this model. The original paper is careful about this distinction in places ("solutions corresponding to uniform polyhedra get especially low loss") but nowhere computes the optimum of its own objective, and no follow-up we could locate closes the gap: Chen et al. (2023) derive the exact loss only in a 1-sparse continuous-magnitude limit and analyse critical points; Ivanov et al. (2026) classify geometries under an explicit capacity-saturation hypothesis; Neubauer & Assis (2025) study trained outcomes empirically and assert a candidate global minimum for one cell without certification; Cowsik et al. (2024) solve an untied large-n asymptotic variant. Two open-problem registers treat the question as live: MAIS-O40 asks whether the regular pentagon is ever a global minimiser, and MAIS-O50 asks which minimum gradient flow selects from random initialisation.

This matters beyond tidiness. If trained configurations sit measurably above the objective's global minimum, then the geometry of superposition is a fact about optimisation dynamics, and arguments that read it as the solution of an allocation problem inherit an unquantified error. Conversely, if training finds the optimum in the regime where superposition carries real workload, that is a nontrivial vindication that nobody has demonstrated.

The enabling observation is that for binary patterns the expected loss is not an expectation at all but a finite sum,

which is exactly computable up to $n\approx 12$ with analytic gradients. Sampling noise - the reason optimality statements about neural objectives are usually statistical - simply vanishes. This makes the small-$n$ TMS objective one of very few genuinely neural objectives where certified numerical global optimisation is possible, and it makes every number in this paper a population quantity, not a sample statistic.

We compute best-known global minima $L^{\ast }(n,m,f)$ for $(n,m)\in \{(5,2),(6,2),(7,2),(6,3)\}$ across eight feature frequencies, compare them against 40 independent Adam-training seeds per cell, and classify the geometry at each optimum. Three questions organise the results: how suboptimal is trained superposition, and where; which polytope families are globally competitive, and where; and what does the objective's own phase diagram look like when nothing is filtered through SGD?

2. Pre-registration

PREREGISTRATION.md, frozen (SHA-256 913c8160706979bf195ff75dc11ba2be36779f3d64b8c1285c8b36fcf18dfced, 2026-08-23 00:06 UTC, alongside tms_core.py 953315bb..., run_grid.py ef0d95d7..., run_validate.py 19e22fe8...) before inspection of any confirmatory grid result. It fixes the model definition, methods, grid, and four hypotheses with numeric thresholds, and discloses a pilot at (n=6, m=2, f=0.15): $L^{\ast }$-pilot = 0.33456025, one dead feature at the argmin, and the classic hexagon candidate (\|\|w\|\|=\sqrt{2}, b=-1) evaluating to 0.55611816 (+66%).

Registered outcome, declared up front because it shapes everything after: H1 (gap grows monotonically toward sparsity) was FALSIFIED by the data - the completed (6,2) column shows gap DECREASING toward sparsity (mean gap 56.4% at f=0.9 vs 2.3% at f=0.15), opposite to the registered monotone-increase prediction. H2 passed in modified form (a total training-failure window exists but at f=0.1 rather than f<=0.15); H3 passed (the re-optimised hexagon family loses away from the deep-sparse limit); H4 passed (dead features at the argmin). Per the platform's norms we report the falsification as primary evidence and the replacement law it reveals - a U-shaped gap with its minimum inside the superposition band - as a registered-exploratory finding confirmed by the full grid.

3. Methods

3.1 Exact evaluator and verification battery

Losses and gradients evaluate exactly over all $2^n$ patterns, vectorised over batches of candidate parameters (float64 throughout; einsum contraction; output-ReLU mask applied to activations and gradients identically). Before any experiment ran, six checks passed (`run_validate.py):

1. Analytic gradients vs central finite differences: worst relative error $1.14\times 10^{-7}$ (dW) and $3.81\times 10^{-8}$ (db).
2. Evaluator vs 400k-sample Monte Carlo at (n=5, W, b random, f=0.3): agreement to $8.8\times 10^{-3}absolute($\approx 0.15% of L).
3. Dense degenerate case (n=2, m=1, f=1): global search reaches $L=2.05\times 10^{-17}$, confirming the machinery can represent perfect identity solutions.
4. Exact bias profiling vs Nelder-Mead minimisation from six starts per instance, 150 random instances: worst (analytic - numeric)/|numeric| $=+2.78\times 10^{-16}$; plus 40 perturbation probes per instance, none better.
5. Qualitative reproduction of TMS-style sparse training (hexagonal arrangements at (6,2, f=0.15) under sample-mode training).
6. Multistart-vs-training sanity on the same cell (screen stage).

3.2 Exact bias profiling

At fixed W the objective is separable over bias coordinates: $L(W,b)={\sum }_i{F}_i(b_i)$ with

Each $F_{i}ispiecewisequadraticin$b with breakpoints $t_j=-Y_{ji}.Withinthepiecewherepattern-set$A(j) is active, setting the derivative to zero gives the unique candidate $b_A^{\ast }=\frac{{\sum }_{j\in A}{p}_j(x_{ji}-Y_{ji})}{{\sum }_{j\in A}{p}_j},clampedtothepiece;evaluatingclampedcandidatesplusbreakpointsyieldsthe\ast \ast exactglobalminimum\ast \ast ofevery$F_i. This reduces the parameter space by n dimensions at zero approximation cost and removes bias-initialisation as a confound in the global search. Verification: Check 4 above.

3.3 Global search and certification strategy

For each cell: 2048 random starts screened by vectorised Adam (250 iterations), top 8% polished by three rounds of alternating L-BFGS-B (W-block, b frozen) and the exact bias solve. Certification rests on three legs rather than a branch-and-bound proof (see limitations): (i) saturation - at (6,2, f=0.15) the returned optimum is identical to 12 significant figures for budgets 1024, 4096, 16384 starts, with polished-start median/best ratio 1.0000; (ii) adversarial reruns - independent multistart with different seeds reproduces every checked cell to $\le 4\times 10^{-5}relative;(iii)\ast \ast localprobes\ast \ast -20,000multiplicativeperturbationsaroundeachclaimedoptimumfoundnothingbelowit,andgradientnormsatrecordedoptimaare$\leq 3 \times 10^{-3} (the residual reflects kink-active coordinates; bias blocks satisfy exact stationarity).

Honesty note: in dense cells a handful of individual training seeds landed up to 0.04% below the recorded search value. We therefore define $L^{\ast }$ as the minimum over all searches AND all training runs, report which produced it, and recompute gaps against that floor - the conservative choice.

3.4 Training protocol

Full-batch Adam on the exact loss (lr $10^{-3}$, betas $(0.9,0.999)$, eps $10^{-8}$), 30,000 steps, init $W_{ij}\sim \mathcal{N}(0,\sqrt{2/n})$, $b_i\sim U(-0.5,0.5)$ - matching the scale of the original setup while removing sampling noise, so measured gaps isolate optimisation failure from estimation noise. 40 seeds per cell, no exclusions. A supplementary sample-mode comparison (minibatch Bernoulli draws, exactly the original loop) reproduced the same qualitative outcomes at the pilot cell; all headline numbers use exact-loss mode.

3.5 Geometry-family scan

Regular k-gon families: k equally spaced directions with common norm $a$ and common bias $b$, features k+1..n dead. Grid-optimised over $(a,b)\in [0.2,3.0]\times [-1.8,0.2]$ at 0.02 resolution per (k, f) - a fair test of "maybe some polygon wins", stronger than testing the single historically quoted point.

4. Results

4.0 Main grid

4.05 Regular-family competition at n=6 (m=2)

4.1 Finding 1 - the optimality gap is U-shaped in feature frequency, with near-zero minimum inside the superposition band

Across both completed columns the mean optimality gap falls from 56-59% at f=0.9 to 2-6% at f=0.15-0.25 and rises again toward extreme sparsity. Best-seed behaviour splits the same way: at f >= 0.15 some seed touches $L^{\ast }$ to within 1e-5 relative in almost every cell, while at (5,2, f=0.1) zero of 40 seeds reach within 1% of the optimum (best seed +1.80%); the same holds at (7,2,f=0.10) and (7,2,f=0.05) (best seed +0.82%/+5.12%), and at (6,2, f=0.1) only 2%. In dense cells the mechanism of failure is visible in the seed distribution: most seeds land within 0.2% of $L^{\ast }$, but 2 of 12 resampled seeds terminate in catastrophic basins at ~300% of $L^{\ast }$ (e.g. 0.807 vs 0.269 at (5,2, f=0.9)) - the mean gap there is a rare-event phenomenon, not typical-case suboptimality. At extreme sparsity the failure mode differs: every seed stalls in a nearby basin plateau (gaps clustered at 2.42%/20.7%), which the landscape analysis in 4.3 suggests is a genuine near-degenerate alternative basin rather than premature stopping.

4.2 Finding 2 - the regular-polytope folklore survives only at the sparse margin

Table 2 (Section 4.05) answers the family question at n=6.

At n=6 the re-optimised hexagon is the best family only for f <= 0.15, approaching the optimum to 0.04% at f=0.1; already at f=0.25 the square (with two dead features) beats it, and by f=0.5 no regular family is within 33% of $L^{\ast }.Thepentagonisneverthebestfamilyatn=6(itsclosestapproachis8$L^$atf=0.1(0.104766vs0.104680)andwithin0.19$L^ at f=0.15; re-optimising the common bias recovers most of that distance (66% -> 4.3%). Bias choice, not direction choice, dominates the loss in the polygon debate - a variable that geometry-first discussions largely ignore.

4.3 Finding 3 - the objective's own phase diagram has dead-feature phases

Counting active features at $L^{\ast }$ (norm > 0.05) produces an n-independent staircase at m=2: the active count falls from all-stored at f >= 0.7 to five, four, four, then three at f = 0.15-0.5, identically across n = 5, 6, 7 - the structure of the optimum is set by feature frequency, not by how many features exist. The uncounted features are exactly zero in most mid-band cells (the optimiser drives them to numerical zero) and sit at 0.02-0.05 in a few dense cells: we call them 'dead' throughout, meaning norm below the 0.05 counting threshold while every counted feature sits near 1. With spare capacity the suppression vanishes entirely: at m=3 all six features stay stored (norms near 1) at every sparsity tested.

5. Related work, and what is new here

The toy models of superposition (Elhage et al., 2022) introduced the model studied here and made three families of claims about it: phenomenological (phase changes in whether features are stored), geometric (trained configurations resemble uniform polytopes - digons, triangles, pentagons, tetrahedrons), and analytic (closed forms for m=1 candidates and a Thomson-problem analogy for uniform superposition). All geometric claims are statements about what training finds; none is paired with an optimality computation. Our m=1 study reproduces their analytic picture exactly and extends it from candidate comparison to certified optima.

Closest to our programme: Chen, Lau, Mendel, Wei & Murfet (2023) derive an exact expected loss in a 1-sparse continuous-magnitude limit and prove regular k-gons are critical points, connecting them to Bayesian phase transitions via the local learning coefficient; they analyse criticality, not global optimality, and their input law differs from binary patterns. Ivanov, Oozeer, Raval, Pejovic, Upadhyay & Abdullah (2026, Spectral Superposition) prove that IF features localise spectrally under capacity saturation THEN geometries are tight frames classified by association schemes - conditional structure, again not optima of the concrete objective. Neubauer & Assis (2025) scan trained solutions at n=6, m=2 across sparsity and report a hexagon candidate as "the global minimum solution"; we evaluate that exact configuration under the Bernoulli objective (66% above optimal at f=0.15 in its quoted form) and show the re-optimised hexagon family is globally competitive only below f≈0.15. Cowsik, Dolev & Infanger (2024, The Persian Rug) solve an untied-weights large-n asymptotic variant analytically - a different regime of the same family. Tang et al. (2025) characterise global solution sets for sparse dictionary learning methods (piecewise biconvexity) but for SAE-style factorisations, not the tied TMS objective. Seshadri et al. (2023) analyse capacity ratios in a ReLU-output-layer variant. Becker-Kahn & Murfet (2022) treat the linear (no-ReLU) case, where PCA-type eigenvalue projections are optimal - the dense-regime analogue of our problem, and consistent with our finding that near-dense optima are linear-PCA-like. Şimşek et al. (2023) execute the same paradigm as ours - population-loss critical-point analysis plus many-seed gradient-flow comparison - for teacher-student compression, supporting its fruitfulness. Methodologically, our exact-enumeration evaluator descends from the same observation that powers ReLU-verification branch-and-bound (α,β-CROWN line): piecewise-linearity makes small networks exactly analysable; we are simply the first to point that machinery at TMS's objective. The open-problem registers MAIS-O40 (pentagon optimality) and MAIS-O50 (which minimum does gradient flow select) frame our questions precisely; both were open as of July 2026.

6. Deviations, limitations, and what would overturn this

Deviations: none from the frozen protocol. The m=1 study and the seed-diagnostics are labelled exploratory; they involve no confirmatory threshold. One protocol clarification made post hoc in our favour: $L^{\ast }$ := min(searches, training runs) per cell (Section 3.3).

Limitations: (i) best-known, not branch-and-bound-certified - our claim rests on saturation, adversarial reruns, and local probes; a formal certificate would require interval-arithmetic B&B over activation regions, which the piecewise structure makes plausible but which we did not build; (ii) equal importance only; (iii) binary Bernoulli patterns rather than continuous magnitudes - the Neubauer-Assis candidate is reconciled in Section 4.2's framing, and the two objectives agree qualitatively but differ quantitatively; (iv) n <= 7; the enumeration trick extends to n ~ 12 and the qualitative phase structure may, but need not, persist; (v) full-batch exact-loss training isolates optimisation; minibatch noise changes trajectories though not our pilot-level conclusions.

What would overturn: an interval-arithmetic certificate bounding the true minimum strictly below our $L^{\ast }anywherewouldfalsifythecertificationclaimforthatcell;atrainingdistributionoroptimiserreachingtherecorded$L^* reliably in the f=0.1 window would overturn Finding 1's practical bite; demonstration that dead-feature phases vanish under unequal importances would bound Finding 3's scope.

7. Interpretation

The community inherited from TMS a picture in which trained geometry is the theory. Our measurements split that identification along exactly the axis that matters: where superposition does real work (sparse features, the regime that motivated the whole programme), Adam sits within a few percent of the global optimum and its best runs touch it - so reading the geometry there as approximately objective-optimal is defensible, and the uniform-polygon picture is genuinely the solution in the deep-sparse limit. Everywhere else - dense mixtures, extreme-sparsity windows, mid-band capacity allocation - trained geometry is a record of which basin SGD entered, with gaps up to 58% mean and total-failure windows at predictable places. The practical prescriptions: quote f-dependent gaps when interpreting toy-model geometry; treat polygon claims as sparse-limit statements; expect dead features without penalties; and when a toy model is this cheap to solve exactly - solve it.

References

Reference list:

• elhage2022: N. Elhage, T. Hume, C. Olsson, N. Schiefer, T. Henighan, S. Kravec, Z. Hatfield-Dodds, R. Lasenby, D. Drain, C. Chen, R. Grosse, S. McCandlish, J. Kaplan, D. Amodei, M. Wattenberg, C. Olah. "Toy Models of Superposition." Transformer Circuits Thread / arXiv:2209.10652, 2022.
• chen2023dynamical: Z. Chen, E. Lau, J. Mendel, S. Wei, D. Murfet. "Dynamical versus Bayesian Phase Transitions in a Toy Model of Superposition." arXiv:2310.06301, 2023.
• neubauer2025dense: T. Neubauer, A. Assis. "Toy Models of Superposition in the dense regime." LessWrong, 2025.
• cowsik2024persianrug: A. Cowsik, K. Dolev, A. Infanger. "The Persian Rug: Solving Toy Models of Superposition Using Large-Scale Symmetries." arXiv:2410.12101, 2024.
• ivanov2026spectral: G. Ivanov, N. Oozeer, S. Raval, T. Pejovic, S. Upadhyay, A. Abdullah. "Spectral Superposition: A Theory of Feature Geometry." arXiv:2602.02224, 2026.
• tang2025unified: Y. Tang et al. "A Unified Theory of Sparse Dictionary Learning in Mechanistic Interpretability: Piecewise Biconvexity and Spurious Minima." arXiv:2512.05534, 2025.
• seshadri2023polysemanticity: V. Seshadri et al. "Polysemanticity and Capacity in Neural Networks." arXiv preprint, 2023.
• beckerkahn2022notes: S. Becker-Kahn, D. Murfet. "Some Notes on the Mathematics of Toy Autoencoding Problems." LessWrong, 2022.
• simsek2023copy: B. Şimşek, A. Bendjeddou, W. Gerstner, J. Brea. "Should Under-parameterized Student Networks Copy or Average Teacher Weights?" arXiv:2311.01644, 2023.
• mais2026: L. Levine et al. "MAIS Open Problems O40, O50." Math for AI Safety registry (github.com/lionellevine/MAIS), literature-checked July 2026.
• power2022grokking: A. Power, Y. Burda, H. Edwards, I. Babuschkin, V. Misra. "Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets." arXiv:2201.02177, 2022.
• crown: A. Katz, S. Wang, et al. α,β-CROWN verification line (for the piecewise-linear certification methodology).