What is… a blowup? · Part III of IV
A fit that hides a change
Two parts move in opposite directions. Most of their changes cancel. That small observation is our bridge from geometry to learning.
At a crossing, two paths share one position. We kept their directions to tell them apart. Now consider a model made of two parts: the parts can change while their combined prediction hardly changes. The hidden distinction is between what the parts do and what we can observe.
We will make that happen with eight numbers. Then we will return to blowups: the error is a polynomial in the model’s knobs, and the geometry of that polynomial tells us how much room there is for a good fit.
The job: copy eight numbers
A tensor here is just a little block of numbers. Ours can be drawn as two 2 × 2 grids: eight entries altogether. The data we want to copy contain a single 1 and seven 0s.
Our model builds its answer from two simple patterns, A and B. A has one adjustable knob, a; B has another, b. To make a prediction, add the two patterns entry by entry. To fit the model is to turn the knobs until that sum matches the data.
Set both knobs to zero. Each part supplies half the required 1 and nothing elsewhere. Their sum is exactly right. We have used two identical parts to do a job that needs only one. This is the component collision. The demo begins with this known solution and nudges it so we can inspect its neighborhood.
How the two simple patterns are made
Each pattern is built from three two-number lists. For A, every list is [1, a]; for B, every list is [1, b]. To fill a cell, choose one number from each list, multiply the three numbers, then multiply by ½. The row, column, and grid select which entries to use. This is a rank-one tensor; adding such patterns is called a CP decomposition.[34]
The indices i, j, k each take values 0 or 1. The fixed target has $T^*_{000}=1$ and all other entries zero. We measure mismatch by $K=\frac13\sum_{ijk}(T_{ijk}-T^*_{ijk})^2$. The factor ⅓ makes the later quartic coefficient equal to 1. Each part’s three lists are deliberately tied together, and the weights stay fixed: this is a two-knob teaching model. At the fit the sum has rank 1; a nonzero apart move gives rank 2, despite using two allowed terms throughout. The tutorial by Rabanser, Shchur, and Günnemann, §§3.1 and 4.1 explains the general construction.
Two ways to spoil a perfect fit
Move together: turn both knobs up by the same amount. Their changes add, so the prediction moves noticeably away from the data.
Pull apart: turn one knob up and the other down by the same amount. Some changes cancel. For example, one part contributes +0.15 to a cell while the other contributes −0.15. The sum stays at the correct value, 0.
The cancellation is strong, but incomplete. In another cell both contributions are positive: 0.045 + 0.045 = 0.09. The parts moved by 0.3, yet the remaining prediction error is only 0.3² = 0.09.
Loss is our score for how wrong the prediction is: square the entry errors, add them, then divide by three to set the units. Zero means a perfect fit. When the prediction error is as small as the square of a move, squaring that error makes the loss as small as the move’s fourth power.
Figure 17. Start at Exact fit: A and B add up to the fixed data. Press Move together, then Pull apart, watching the outlined cell. In Pull apart, select 011 · error that remains to see what did not cancel. Finally press Halve δ. The comparison below shows why for small together moves the loss falls about 4×, and for apart moves exactly 16×. Here δ means the size of the proposed knob movement.
Read left to right: part A + part B = our prediction; compare it with the data. Each card contains the same eight positions. A cell’s three-digit address means row, column, grid: 001 is row 0, column 0, grid 1. The outlines track the same cell across all four cards. Blue is negative; orange is positive. Click a cell to follow its arithmetic.
Both rows propose moves from the exact fit. The cards above show the currently selected move.
A half-sized move has ¼ the square and ¹⁄₁₆ the fourth power. The exact formulas and the contribution from each entry are available below.
Inspect every entry, the loss breakdown, and keyboard controls
Arrow keys change the row or column of the selected cell; Page Up/Down changes the grid. The highlighted numbers in the three lists are multiplied to make that cell.
Larger models can contain repeated parts too: two neurons may compute the same feature, or two mixture components may describe the same cluster. The full models have more knobs, but redundancy and cancellation are recurring sources of singularity.[26]
Why the hidden change matters
Real data come with noise. We usually have many nearly right fits, rather than one visibly perfect answer. A weakly changing direction lets the knobs move farther while the prediction remains acceptable. To understand the model, we need to know how much room those nearly right settings occupy.
Use two new knob coordinates: the mean $s=(a+b)/2$ tells how far the parts move together; the separation $d=(a-b)/2$ tells how far they pull apart. Close to the perfect fit, the loss behaves like:
The square describes the together move; the fourth power describes the apart move. A quadratic approximation—the Hessian—sees the first term and reports zero curvature for the second. It misses the size of the remaining error. This is the local degeneracy that makes the fit singular.
Check the exact tensor loss and its Hessian
Entries with one index equal to 1 have value s; those with two have value $s^2+d^2$; the last entry has value $s^3+3sd^2$. There are three entries of each of the first two types. Thus:
Along the together move, $K(\delta,0)=\delta^2+\delta^4+\delta^6/3$. Along the apart move, $K(0,\delta)=\delta^4$ exactly. The symbol $\asymp$ means the exact loss is bounded above and below by positive constant multiples of the simpler one near the fit. This is the loss of the specified two-parameter family, not a formula for every tensor model.
Bring back the blowup
The object has changed: we are now studying a polynomial in knob settings. Its zeros are perfect fits. The tool is familiar: choose new coordinates that expose what happens near the zero.
Blow up the parameter plane around its perfect-fit point. Two of the point blowups from Part I combine into the substitution $d=r$, $s=r^2v$. The perfect fit becomes an exceptional line $r=0$: every value of v on that line maps back to the one old setting $(s,d)=(0,0)$. In this chart the loss becomes:
Now the vanishing is easy to see. For bounded v, the factor $1+v^2$ stays positive. The small factor is $r^4$.
Work through the two ordinary point blowups
- Set $s=dv_1$. Then $K_0=d^2(v_1^2+d^2)$. After extracting $d^2$, the remaining factor still vanishes at $(d,v_1)=(0,0)$.
- Blow up that point: set $d=r$, $v_1=rv$. This gives $s=r^2v$, $d=r$, hence $K_0=r^4(1+v^2)$. The map is invertible for $r\ne0$ in this chart.
There is also a link to Part I’s touching parabolas: over the complex numbers, $s^2+d^4=(s-id^2)(s+id^2)$. The same two blowups handle those conjugate branches. On the real parameter plane, the loss has just one zero.
To count settings, we must also track area. A little rectangle in (r, v) does not keep its area when mapped back to (s, d). Its area is multiplied by $|r|^2$, the absolute Jacobian determinant:
So the useful answer has two parts: how fast the error grows, and how much area each new coordinate patch represents. Blowups expose both. In this small example we can also calculate directly without a blowup; that gives us a check. The curved cusp in Part IV needs more geometric work.
Follow the powers into the statistical integral
On a bounded v-sector, substantial weight lies at $r\sim n^{-1/4}$. The radial width contributes $n^{-1/4}$ and the area factor contributes $n^{-1/2}$, giving $n^{-3/4}$. Other charts cover the remaining directions. The direct separated integral gives the same $n^{-1/2}n^{-1/4}=n^{-3/4}$ over the full square.
The actual tensor loss, under the same substitution, is:
The bracket stays positive and bounded away from zero near a bounded piece of r = 0, so the leading exponent is unchanged. The mean–separation change of coordinates has a constant nonzero Jacobian.
That is our route back to Hironaka. Resolution turns difficult vanishing into powers of coordinates, and the map tells us how to count volume correctly. Part IV asks what that volume means for learning: when many settings fit the data, how should we weigh their predictions?
Optional: other meanings of “singularity”
The experiments below motivated the essay, but they ask different questions. A contact law needs limiting histories; a tensor approximation may lack an attainable optimum; a field can have a topological obstruction. Read these as separate case studies, rather than steps needed for the learning-theory argument.
Mechanical side trip: simultaneous impacts and inelastic collapse Figures 18–19
Rigid-body collisions: the opening demo, unpacked
The three balls from Part I provide our first mechanical blowup. In an elastic collision between two balls of masses $m_1, m_2$, conservation of momentum and kinetic energy uniquely determines the outgoing velocities:
For an approaching pair, momentum and energy conservation together with the outgoing-contact condition select the usual elastic impact. With three simultaneous contacts, conservation alone leaves multiple outgoing possibilities. A contact model or a limiting procedure must provide additional information.
Configuration space in a wedge: Write the gaps between neighbors as $g_1 = x_2 - x_1 - 2r \ge 0$ and $g_2 = x_3 - x_2 - 2r \ge 0$. In mass-weighted coordinates $q_i = \sqrt{m_i} x_i$, kinetic energy is standard Euclidean $\frac{1}{2}|\dot{q}|^2$. Removing uniform translation reduces the three-ball system to a single point bouncing inside a planar wedge between the constraint walls $g_1 = 0$ and $g_2 = 0$, with wedge angle:
A sequence of pairwise collisions is a billiard path bouncing between the walls. Simultaneous impact is the corner $g_1=g_2=0$; the equation $g_1g_2=0$ describes the union of both walls.
What the bar keeps. Fix the incoming velocities and vary the ratio of two small positive gaps. The pairwise impact model gives intervals with common limiting outcomes. Equal masses give the same outgoing velocity pattern for this experiment; a light middle ball can give different patterns. With a uniform prior over the displayed gap ratio, each interval’s width is its outcome’s probability share. A different prior gives different shares.
Figure 18. Left: three balls colliding simultaneously, with the gap-ratio bar colored by outcome. Right: the same motion mapped to a single point in a wedge. Drag the marker or click an outcome below to test incoming directions.
A second difficulty: inelastic collapse. For equal masses and constant restitution $e$, the three-ball model admits trajectories with infinitely many impacts in finite time below $7-4\sqrt3\approx0.072$. The figure follows one such setup with a finite event cutoff. All’s Well That Ends Well[19] gives a separate terminating impact algorithm. Its guarantee concerns that collision model; it is not an application of Hironaka’s theorem.
Figure 19. Inelastic collapse. Below $e \approx 0.072$, the model can accumulate infinitely many impacts in finite time. The computed trace stops at its event cutoff; separation is not asserted in the collapsing case.
Tensor and field side trips: border rank, cubics, and topological defects Figures 20–25
A fit that needs an unreachable limit
Here is another way two parts can cancel. Change the data: now three cells should be 1 and the rest 0. This target, W, needs three simple patterns for an exact match. Insist on using only two, and an optimizer can get arbitrarily close without ever reaching it.
The trick is to make the two parts large and almost opposite. One supplies large positive numbers; the other supplies large negative numbers. Most of that cancels, leaving a small, increasingly accurate answer. Watch the sum improve while the individual pieces grow.
This is called border rank 2: the target is a limit of two-part predictions, although its rank is 3. Every finite two-part fit still has some error.
The exact target, objective, and limiting construction
Set $W_{001}=W_{010}=W_{100}=1$ and all other entries zero. The two-rank-one-term fit minimizes:
Writing $e_0=(1,0)$ and $e_1=(0,1)$ gives an explicit sequence of rank-two approximations:
The quotient has the three desired entries equal to 1, the three double-1 entries equal to h, and entry 111 equal to h². Its limit is W. The two summands individually diverge while their difference converges.
Figure 20. Press Play. This recorded fitting run tries to match W using only two simple parts. Look at each part, then their sum, then the target. The error plot falls while the size plot rises. The construction above explains why such fits can keep improving without reaching a finite exact answer.
Each piece is aₖ ⊗ bₖ ⊗ cₖ: entry i j k is the product of the three corresponding vector entries. The vectors and tensor slices update together at the selected gradient step. Their scaling is the stored optimizer’s choice; redistributing scale between factors can leave their tensor unchanged.
Describe the limit directly. Keep a common pattern, the direction in which the two parts differ, and a small separation h. After expanding their difference, the formula has a finite value even at h = 0. This new description includes the limiting tensor that the original two-part model could only approach.
The finite formula at the boundary
Expand $[(a+ha')\otimes(b+hb')\otimes(c+hc')-a\otimes b\otimes c]/h$. Every numerator term contains h, so division leaves a polynomial. At h = 0 its value is $a'\otimes b\otimes c+a\otimes b'\otimes c+a\otimes b\otimes c'$, a tangent tensor. This enlarged parameterization includes border-rank limits; at its boundary it goes beyond the original finite rank-two model.
Figure 21. Orange uses the original two-part parameters. Violet uses the description that includes the limiting pattern at h = 0. Compare the falling loss with the size of the parameters. The second description gives this optimizer access to a finite exact endpoint.
Symmetric tensors as binary cubics: two direction records
A symmetric $2 \times 2 \times 2$ tensor corresponds to a homogeneous binary cubic with four coefficients. On the chart where the leading coefficient is nonzero, normalize it to one and write $t^3 + a t^2 + b t + c$. For distinct roots, its real symmetric rank is determined by the cubic discriminant $\Delta$:
- $\Delta > 0$: three distinct real roots, real symmetric rank 3;
- $\Delta < 0$: one real root, real symmetric rank 2;
- $\Delta = 0$: repeated root (tangent developable boundary);
- Triple root $(t - r)^3$: rank one (the twisted cubic curve $(a, b, c) = (-3r, 3r^2, -r^3)$).
The boundary $\Delta = 0$ is swept out by tangent lines to the twisted cubic—a tangent developable creased along the rank-one curve into a sharp cuspidal edge $(e^2, 2e^3)$. Transverse to the rank-one locus, the boundary has a cusp. The point-blowup calculation from Part I applies along that locus: the strict transform becomes smooth first, and the exceptional boundary requires further steps. Recording the repeated root $r$ gives the smooth Nash lift for this cuspidal boundary. This is a special example, not a termination argument for general tensors.
Figure 22. Monic cubics $(a, b, c)$. The creased boundary $\Delta = 0$ unfolds into a smooth sheet under Nash's lift by $r$. The slice panel on the right shows Hironaka resolving the cuspidal crease step-by-step.
Cross fields: topology constrains what can be repaired
A quad mesh needs a cross field: four directions spaced by quarter turns at each regular point. Tangent boundary alignment on a disk forces total cross-field index $+1$. The representation $u=e^{4i\theta}$ has boundary degree 4. Four simple index-$+1/4$ defects realize that total; they can also merge into a higher-index defect. Topology fixes the total index, not a universal lower bound of four distinct points.
Figure 23. Drag singular points on a disk. The bars show a point-vortex energy approximation with calibrated core costs. Compare a merged degree-4 defect with four degree-1 defects. The leading core cost is quadratic in degree, while pair interactions depend logarithmically on separation.
Core structure and energy: Standard numerical optimizers minimize the Ginzburg–Landau energy:
Here $u$ is a complex representation of the cross, not the cross angle itself. Away from the cores, $|u|\approx1$; at a zero, its argument—and hence the cross direction—is undefined. For an isolated axisymmetric degree-$d$ vortex, the leading energy is $\pi d^2\log(1/\varepsilon)$. This favors splitting a degree-4 core into four degree-1 cores when the boundary and interactions permit it.
Figure 24. A Ginzburg–Landau vortex. Left: cross field over $|u|$. Middle: energy per logarithmic radial interval, which approaches a plateau outside the core. Right: four simple cores cost substantially less than one 4-fold core, driving core splitting.
Three strategies for frame fields:
- Ginzburg–Landau Penalty: Allows the field amplitude $|u|$ to soften to zero at the core. The penalty parameter $\varepsilon$ controls the core size.
- Drill a hole (Blowup): Changes the domain to an annulus: the winding can live on the inner boundary. Shrinking a circular boundary while retaining its angle is related to a real oriented blowup. A finite drilled hole is not an algebraic resolution.
- Lift to a sphere (Burer–Monteiro): Changes the target to unit vectors $v\in S^2$. The field can leave the equator; at a pole its planar projection is zero, so projecting back still leaves an undefined cross direction. This relaxation can improve optimization without removing the original topological obstruction.
Figure 25. Three strategies side by side: Ginzburg–Landau splitting (left), drilled hole blowup (middle), and Burer–Monteiro sphere lift (right).
What changes in each problem?
The remedy depends on the task. Retaining a gap ratio, enlarging a tensor model, and changing a field’s domain solve different problems:
| Problem | What fails? | What the construction provides |
|---|---|---|
| Simultaneous impact | The pairwise law leaves the coincident event underdetermined. | Gap ratios keep distinct limiting histories. Selecting one physical outcome needs a contact model. |
| Finite tensor collision | The parameter map loses first-order rank. | Mean and separation expose a quartic loss; its local volume is computable. |
| Border rank | The rank-two image is not closed. | A tangent-limit parameterization includes the missing boundary tensor. |
| Cross field | Boundary winding forbids a nonvanishing extension. | Regularization resolves core structure; holes or sphere lifts change the domain or target. |
These examples motivate retaining limiting information. For the direct link from algebraic blowups to statistical inference, return to the two-chart loss calculation and continue to Part IV.