Singular learning theory: why parameter volume matters

What is being learned?

Imagine observing noisy outputs of a model. We want to infer its predictions from those observations. A parameter setting supplies one candidate prediction; a loss measures its disagreement with the target. With limited data, many settings fit almost equally well.

Bayesian inference gives weight to all those settings. Start with a prior over parameters and multiply it by the likelihood of the observations. Normalize to obtain a posterior. For a new input, average the model’s predicted distribution over that posterior. The amount of parameter space giving a good fit therefore matters, alongside the best fit’s error.

Ordinary asymptotic statistics often assumes that each nearby parameter direction changes predictions to first order. Then the loss is locally a quadratic bowl. Singular learning theory studies what happens when that assumption fails: redundant weights, disappearing components, or colliding components can make some changes invisible to first order.

In Part III, the two model parts moved while most of their changes canceled. That gave us a weakly changing direction: an apart move of size d caused loss d⁴. Here is the consequence we postponed: a model can have a lot of room to move and still fit almost as well.

The main argument has three steps: count nearly correct parameters, add their likelihood weights, and use blowups when the geometry is harder. A live thermometer then turns the exponent into a measurable quantity. Network examples extend the argument in the learning lab.

How much room does a good fit have?

The finite tensor collision gave a loss locally comparable to $s^2+d^4$, with mean $s$ and separation $d$. Compare that simpler local model with a regular loss:

Treat these as two candidate local models for the same target. Both have two parameters and a perfect fit at $(0,0)$. Comparing them keeps the parameter count and best error fixed, so we can isolate the effect of the loss geometry. The collision expression is the local scaling model derived from our tensor family. Every picture uses the same square $-1\le s,d\le1$.

First, count the nearly correct parameters

Read the axes as parameters, not data coordinates. A dot is one complete model setting $(s,d)$. Its height is not drawn; its loss is determined by the equation above the panel. Allow error $K\le\varepsilon$. Every shaded point is an acceptable parameter setting. A quartic separation direction stays acceptable farther from zero than a quadratic one.

Figure 26. Press Demand 10× less error. The regular region retains 10% of its previous area; the collision retains about 17.8%. The axes are identical. Read the fractions of the prior square below the pictures.

The learning coefficient is a shrinking rate

The regular disk has width $\varepsilon^{1/2}$ in each direction, hence area proportional to $\varepsilon$. The collision has widths $\varepsilon^{1/2}$ and $\varepsilon^{1/4}$, hence area proportional to $\varepsilon^{3/4}$. The exponent is the local learning coefficient (or real log canonical threshold, RLCT):

More generally, under analytic and positive local prior assumptions, the prior-weighted volume can scale as $\varepsilon^\lambda(\log(1/\varepsilon))^{m-1}$. Smaller $\lambda$ means slower contraction as error tolerance tightens. It is an exponent, not a literal number of remaining parameters. A smooth positive change of coordinates and its corresponding prior do not change it.

Then, add up the Bayesian weight

For independent Gaussian measurements, likelihoods multiply into the exponential of a summed squared error. In this controlled comparison we assign weight $e^{-nK}$, where $n$ controls concentration. Each candidate model has a uniform parameter prior on the square, and the models have equal prior probabilities.

In the next demo, position still means a parameter setting; brightness means its likelihood weight. The evidence $Z$ is the total brightness, averaged over the prior square. The posterior probability of a candidate is its evidence divided by the sum of both evidences. The minimum loss alone does not determine that probability. These deterministic toy integrals isolate the geometry of a population loss; the statistical theorem below connects such exponents to inference with random data.

Figure 27. Select n = 100, then n = 10,000. Both bright regions contract. The collision retains more integrated brightness, and the bar reports its larger posterior share. The two panels use the same 0–1 relative-likelihood color scale.

Here the integrals separate. A quadratic direction contributes $n^{-1/2}$; a quartic direction contributes $n^{-1/4}$. Consequently $Z_{\rm regular}\asymp n^{-1}$ and $Z_{\rm collision}\asymp n^{-3/4}$. Their ratio grows like $n^{1/4}$, in favor of the collision under these stated priors. The corresponding complexity term in $-\log Z$ is $\lambda\log n$.

Use the cusp resolution to compute an exponent

The previous loss already separated in its original coordinates. Now take $K=(y^2-x^3)^2$. Its zero-loss set is the same cusp resolved in Part I. Along its smooth part, there is one flat tangent direction and one curved transverse direction. At the tip, both Hessian eigenvalues vanish.

Figure 28. The left panel shows the curved zero-loss trough. The right panels show cross-sections and Hessian eigenvalues. The cusp tip is more degenerate than an ordinary point of the trough; the quadratic approximation misses its leading shape.

Keep the loss and the volume form together

Follow the three ordinary point blowups from Part I. One final chart has original coordinates $x=t^2w$, $y=t^3w$. We can now calculate:

The exceptional divisor $t=0$ has loss exponent 12 and volume exponent 4. Its candidate exponent is $(4+1)/12=5/12$. At $w=0$, the candidate is $(1+1)/4=1/2$; at $w=1$, it is $(0+1)/2=1/2$. To see where $5/12$ comes from, hold $w$ away from 0 and 1. Substantial likelihood weight requires $nt^{12}$ of order one, so $t=n^{-1/12}\tau$. Then $|t|^4dt=n^{-5/12}|\tau|^4d\tau$. The exponent counts both the width of the weighted region and the shrinking volume element. Checking the other divisors and complementary charts gives no smaller value:

This is the payoff of resolution: the same substitutions that untangle the curve expose how much parameter volume lies near its zero set. The Jacobian is essential; reading only the power 12 in the loss would give the wrong answer.

What Hironaka contributes—and why care?

The general bridge from analytic loss geometry to Bayesian asymptotics is developed in Watanabe’s monograph.^[26] The calculations here are small worked examples of that bridge.

For the cusp, a few charts were enough. For an arbitrary analytic nonnegative loss on a suitable compact neighborhood, resolution provides a finite chart cover in which both the loss and volume form have controlled coordinate powers:

Here $a$ and $b$ are nonvanishing local factors; with a smooth positive prior, they do not change the leading exponents. Over the contributing charts, the smallest $(h_i+1)/(2k_i)$ with $k_i>0$ gives $\lambda$. The greatest number of minimizing divisors meeting in a contributing chart determines the logarithmic multiplicity $m$.

The coordinate powers control even a coupled integral. Under these assumptions, its asymptotic form is $Z(n)\asymp n^{-\lambda}(\log n)^{m-1}$.

Estimate the exponent without constructing the resolution

For a concentrated distribution $p_n(w)\propto e^{-nK(w)}\varphi(w)$, the partition function satisfies $-\partial_n\log Z=\mathbb E_n[K]$. If $Z\asymp n^{-\lambda}(\log n)^{m-1}$ with the corresponding differentiable asymptotic expansion, then:

This motivates a statistical thermometer: sample parameters, measure their excess loss, and multiply the mean by concentration. A trained checkpoint can be the center of the measurement regardless of how it was found; the noisy sampling is a separate experiment around it. The geometric coefficient is an asymptotic target. At finite $n$, even exact sampling can give a different value.

Figure 30. Start with the bowl. Compare the orange sampled mean to the dashed finite-n grid reference first. Then switch to the flat-direction loss from our tensor example, with coordinates exchanged. The purple line is the limiting coefficient. A sampler/reference gap concerns the numerical method; a reference/purple gap concerns finite concentration.

Uniform prior on $[-1,1]^2$; 300 Langevin chains; 8,000 warm-up steps per chain. The product and cusp losses are harder sampling problems. Pause or restart the chains to inspect convergence.

Evidence, prediction, and research questions

In the statistical theorem, the relevant population loss is the Kullback–Leibler divergence $K(w)=\operatorname{KL}(q\|p(\cdot\mid w))$: it is zero exactly when the model distribution equals the data-generating distribution $q$. Squared-error toy losses arise naturally from Gaussian observation models. Their exponents demonstrate the geometry, while a coefficient estimated on one finite training set is a different object.

Evidence matters when comparing statistical models: raw parameter count can give the wrong complexity penalty. Prediction has a related but distinct result. In the realizable analytic setting, for independent data from a distribution $q$ contained in the model, unit-temperature Bayes prediction satisfies:

The expectation is over training datasets, and $p_{\rm Bayes}$ is the posterior predictive distribution. This is not the test loss of one trained network, nor the posterior average of each parameter’s test loss. The distinctions between Bayesian, Gibbs, training, and generalization errors^[27] are part of SLT, rather than interchangeable meanings of “capacity.”

Recent work uses estimated LLC trajectories to investigate training stages and developmental interpretability^[32]. Such measurements can suggest a structural change to investigate. They do not, by themselves, identify a learned algorithm or certify a safety property.

What if we train with SGD?

The thermometer measures the neighborhood of a checkpoint found by SGD; it does not require Bayesian training. Training dynamics ask a further question: which neighborhood does the optimizer reach? For ideal Langevin dynamics with isotropic noise and equilibrium density $e^{-nK}$, the volume effect is exactly the one we calculated. SGD’s minibatch noise depends on the parameters and data, so its behavior also depends on initialization, learning rate, batch size, and training time.

Lau and colleagues’ ResNet experiments^[31] associate changes in training settings with changes in measured LLC. That makes local geometry a useful way to study training, while leaving the dynamical explanation as a separate research problem.

The reason to learn the geometry

I began with singular problems from graphics and simulation. What carries across is the habit of asking which information a representation loses, and what must be retained at a degenerate limit. Sometimes that means approach directions; sometimes it means higher-order vanishing and the volume form. A singularity asks us to look more closely at information that our first description has compressed into one point.

Hironaka’s theorem turns that habit into a general existence result in characteristic zero. Constructive proofs explain how to select compatible centers and terminate. The Lean formalization^[14] makes that mathematical foundation available to further machine-checked work; the models and statistical conclusions still require their own assumptions and proofs.

At the beginning, two paths arrived at the same point. We kept their directions and could tell them apart. At the end, many parameters made almost the same prediction. We kept the powers of the loss and the volume form and could count how much room those fits occupied. A blowup makes a singular limit legible while keeping its connection to the original problem. Hironaka’s theorem guarantees that, in characteristic zero, this work can be brought to a smooth conclusion.

Continue with a particular question

The annotated references point to resolution algorithms, Nash counterexamples, and SLT. Additional collision models, tensor slices, and energy–volume comparisons remain available as supporting experiments.

Optional learning lab

Try a ReLU unit whose fit curve becomes a crossing, compare factorized networks with different widths, or estimate the coefficient around a trained linear network. The calculations above supply the questions to ask of each experiment.