18  Flat minima: the free energy of a basin

Recommended reading

Core (~1 h). Chaudhari et al. (2017) — entropy-SGD, the algorithm that turns this lecture’s thesis into a training objective: read §3–§4 for the local free energy and its inner sampler. Alongside it, Keskar et al. (2017) — the empirical anchor: large-batch training finds sharp minima that generalize worse; read for the evidence, especially the sharpness–gap plots.

Optional background. Hochreiter and Schmidhuber (1997) — Flat Minima, the original argument (via minimum description length), made twenty years before deep learning could test it. Hinton and Camp (1993) — the MDL/Bayesian ancestor: free energy as a complexity penalty on weights. MacKay (2003), Ch. 28 — the Occam factor done cleanly from the Bayesian side. Dinh et al. (2017) — the essential caveat, Sharp Minima Can Generalize for Deep Nets; best read last.

Prerequisite reminder. All of the previous lecture — SGD samples \(e^{-L/T}\) in the isotropic approximation (Equation 17.4) at temperature \(T \propto \eta/|B|\) (Equation 17.2), with anisotropic noise (Section 17.5); Laplace’s method Equation 2.2 and the Gaussian integral Equation 2.1 (Week 1); the free energy \(F = \langle E \rangle - TS\) (Week 1) and its career as an objective (Weeks 5 and 7, Equation 14.6); SGLD as the deliberate sampler (Welling and Teh 2011). New content: flat versus sharp minima, the occupancy of a basin as a free-energy cost, the Occam factor and basin entropy, SGD’s free-energy preference, and entropy-SGD.

18.1 Which minimum?

The previous lecture read training as sampling: in the isotropic approximation SGD draws from \(e^{-L/T}\), thermalizing around minima rather than reaching them. A modern network’s loss landscape holds astronomically many minima, and gradient training will find one of them — but the minima are not interchangeable. Two parameter vectors with the same training loss can differ substantially on test data; whatever separates them is invisible to the loss value itself. The empirical discriminator has been known, in outline, for a long time: flat minima tend to generalize better than sharp ones. A minimum sitting in a wide, low-curvature basin tolerates perturbation — including the perturbation that matters, the shift from the training loss to the test loss — while a needle-thin minimum of identical depth is brittle. Which basin the optimizer reaches therefore affects generalization, and the previous lecture’s thermodynamics bears on which basin that is.

The history assembles across three decades. Hochreiter and Schmidhuber (1997) argued that flat minima generalize because they can be described cheaply — a minimum-description-length argument with Hinton and Camp (1993) as its ancestor, where the free energy of the weights is literally a complexity penalty; both are the Bayesian Occam’s razor of MacKay (2003), translated to network weights. Twenty years later Keskar et al. (2017) supplied the modern evidence: large-batch training — which the previous lecture taught us to read as low-temperature training — systematically finds sharper minima with worse test error. And Chaudhari et al. (2017) closed the loop by building the preference into an algorithm, entropy-SGD, whose objective is a local free energy rather than the loss, the same object this course has carried since Week 1.

Two questions follow: why flat minima generalize, and why SGD, which is given no information about flatness, finds them. The second has a thermodynamic answer that we derive today. A finite-temperature sampler visits a basin in proportion to the Boltzmann mass the basin holds, and that mass depends on the basin’s width as well as its depth. To the extent that SGD is such a sampler, its temperature \(\eta/|B|\) sets how strongly width competes with depth. The first question has a separate, geometric answer, and the two meet only through the curvature.

Take-home 1

Minima with equal training loss can generalize very differently, and the discriminator is flatness: a wide basin is robust to the train→test shift, a sharp one is brittle. A finite-temperature sampler weights basins by the Boltzmann mass they hold, which combines loss and width.

18.2 Sharp or flat, loss or free energy

The robustness argument needs one line and no new machinery. Model the passage from training to test as a small displacement of the loss landscape — the test loss is the training loss evaluated on slightly different data, so its minima sit slightly elsewhere. At a training minimum with Hessian \(H\), a shift of effective size \(\delta\) raises the loss by roughly \(\frac{1}{2} \delta^{\mathsf{T}} H \delta\): the generalization gap scales with the curvature. A sharp minimum (\(H\) large) converts a small distribution shift into a large loss increase, a flat one into a small increase. Figure 18.1 draws the argument: two minima of exactly equal training loss, one narrow and one wide, under a shifted test curve — the sharp minimum’s loss jumps by \(\lambda \delta^2 / 2 = 0.40\), the flat one’s by \(0.02\), a factor of \(\lambda_{\text{sharp}}/\lambda_{\text{flat}} = 25\) with nothing else different between them. (The modern rigorous version of this argument is a PAC-Bayes bound, where flatness controls the generalization gap through a KL term; we keep to the physics telling and name the rigorous one.)

Figure 18.1: Generalization as robustness to the train→test shift. Two minima of the training loss (navy) with exactly equal depth — sharp (\(\lambda = 25\), left) and flat (\(\lambda = 1\), right) — under the test loss (orange dashed), modeled as the same landscape shifted by \(\delta = 0.18\). The gap at each minimum is \(\frac{\lambda \delta^2}{2}\): the sharp minimum jumps by \(0.40\) while the flat one moves by \(0.02\) — a factor \(25\), the curvature ratio, at identical training loss. The shaded bands mark each basin’s width: the volume that the next section converts into an entropy.

Ranking minima by loss ignores this geometry. A sampler ranks them differently, by the probability mass each basin holds, and the next section computes that mass. The two rankings are different quantities: the gap above is the response of the loss to a displacement, \(\frac{1}{2}\delta^{\mathsf{T}}H\delta\), while the basin mass will turn out to depend on \(\log\det H\). Both grow with the curvature, and this shared monotonicity is the whole of the link between a sampler’s preference and robustness.

18.3 Flatness as low free energy

The previous lecture left us with a stationary law over parameter space, \(p_\infty(\theta) = e^{-L(\theta)/T}/Z\) with \(Z = \int \mathrm{d}\theta\, e^{-L(\theta)/T}\) (Equation 17.4). The question “which minimum?” is a question about this distribution: what is the probability of finding the iterate somewhere in basin \(\alpha\)? That is a partial observation of the kind met in Week 5 (Equation 9.2), where we asked for the probability of the visible units and traced out the rest. Here we record only which basin and integrate over the position within it,

\[ P_\alpha \;=\; \int_{\text{basin } \alpha} \mathrm{d}\theta\; p_\infty(\theta) \;=\; \frac{Z_\alpha}{Z}, \qquad Z_\alpha \;=\; \int_{\text{basin } \alpha} \mathrm{d}\theta\; e^{-L(\theta)/T}, \tag{18.1}\]

basin occupancy as a ratio of partition functions

the partition function of the system restricted to the basin, divided by the full one. As in Week 5, the global \(Z\) is common to every basin and cancels from any comparison: \(P_\alpha / P_\beta = Z_\alpha / Z_\beta\). Writing \(Z_\alpha = e^{-F_\alpha/T}\), which defines the free energy \(F_\alpha = -T \log Z_\alpha\) of the restricted system, the occupancy is a free-energy cost, \(P_\alpha = e^{-(F_\alpha - F)/T}\).

The restricted integral \(Z_\alpha\) is not tractable in general, but at low temperature it is dominated by the neighborhood of the minimum, and Laplace’s method (Equation 2.2) applies. Expanding \(L(\theta) \approx L(\theta^\ast) + \frac{1}{2}(\theta - \theta^\ast)^{\mathsf{T}} H (\theta - \theta^\ast)\) with \(H = \nabla^2 L(\theta^\ast)\) and doing the \(d\)-dimensional Gaussian integral (Equation 2.1),

\[ Z_\alpha \;\approx\; e^{-L(\theta^\ast)/T} \, \frac{(2\pi T)^{d/2}}{\sqrt{\det H}}, \tag{18.2}\]

Laplace estimate of a basin’s partition function

each Hessian eigendirection contributing its one-dimensional width \(\sqrt{2\pi T/\lambda_i}\). Two basins of equal depth do not hold equal mass: the flat one holds more, in proportion to the volume of parameter settings inside it that have low loss (Figure 18.2, left). Taking \(-T\log\) of Equation 18.2 and dropping the term \(-\frac{dT}{2}\log(2\pi T)\), which is the same for every basin,

\[ \boxed{\; F_\alpha \;\approx\; L(\theta^\ast) + \frac{T}{2} \log \det H + \text{const}, \;} \tag{18.3}\]

free energy of a basin

the loss at the bottom plus a curvature penalty that grows with temperature, \(\log\det H = \sum_i \log\lambda_i\) charging every sharp direction additively.

Two names attach to what has been derived. The prefactor \((2\pi T)^{d/2}/\sqrt{\det H}\) is the volume of the basin, and its logarithm is an entropy: comparing Equation 18.3 with the Week 1 identity \(F = \langle E \rangle - TS\) identifies the basin entropy \(S_\alpha = -\frac{1}{2}\log\det H + \text{const}\), large for flat minima and small for sharp ones. (The mean energy of the Gaussian basin is \(L(\theta^\ast) + \frac{dT}{2}\), by equipartition; the extra \(\frac{dT}{2}\) is common to all basins and sits in the constant.) On the Bayesian side the same prefactor is MacKay’s Occam factor (MacKay 2003). There the weight is a posterior \(\propto e^{-NL(\theta)}\), with \(L\) the per-sample loss of the previous lecture and a flat prior over the basin assumed, so the Bayesian evidence of a basin is Equation 18.2 at \(T = 1/N\), up to the prior width. The factor penalizes needlessly precise solutions, which is Occam’s preference for the less fine-tuned explanation.

The consequence, drawn in Figure 18.2 (right): a deeper but narrower minimum can have higher free energy than a shallower, wider one. At \(T = 0.1\), a sharp basin with \(\lambda = 25\) and perfect training loss \(L^\ast = 0\) has \(F = 0.161\); a flat basin with \(\lambda = 1\) and worse training loss \(L^\ast = 0.08\) has \(F = 0.080\) — the flat minimum has the lower free energy, \(0.080\) against \(0.161\), a difference invisible in the loss. Whether that ranking governs anything depends on the optimizer being a finite-temperature sampler, which the previous lecture established in the isotropic approximation.

Figure 18.2: Laplace turns basin geometry into statistical weight. Left: the Gaussian factor \(e^{-\frac{\lambda}{2}(\theta - \theta^\ast)^2}\) at a sharp (\(\lambda = 25\)) and a flat (\(\lambda = 1\)) minimum; the enclosed area — the one-dimensional Occam factor \(\sqrt{2\pi/\lambda}\) — is \(0.50\) against \(2.51\): the flat basin holds five times the volume. Right: the free energy Equation 18.3 at \(T = 0.1\) for a sharp-but-deeper minimum (\(L^\ast = 0\), hatched curvature penalty \(\frac{T}{2}\log 25 = 0.161\)) against a flat-but-shallower one (\(L^\ast = 0.08\), penalty zero): the flat minimum has the lower free energy, \(0.080\) against \(0.161\), despite the higher loss.

18.4 SGD’s basin weight

In the isotropic approximation of the previous lecture, SGD run at fixed \(\eta\) and \(|B|\) samples the Boltzmann distribution \(p_\infty \propto e^{-L/T}\) over parameter space (Equation 17.4). The occupancy Equation 18.1, evaluated with the Laplace estimate Equation 18.2, is therefore the probability that a long SGD run is found in a given basin:

\[ \boxed{\; P_\alpha \;\propto\; e^{-F_\alpha/T} \;\sim\; \frac{e^{-L(\theta^\ast)/T}}{\sqrt{\det H}}, \;} \tag{18.4}\]

SGD’s basin weight

energy and volume together, with the temperature \(T \propto \eta/|B|\) setting their balance. Hot training (small batches, large learning rate) weights volume heavily: the sampler rarely visits narrow wells, however deep, and settles in wide basins. Cold training (large batches) weights depth and finds the sharp minima that Keskar’s experiments associated with poorer generalization (Keskar et al. 2017). Equation 18.4 read at low temperature explains why large-batch training lands in sharper minima; that sharper minima then generalize worse is the separate robustness argument of Section 18.2, which shares with Equation 18.4 only the dependence on curvature. The anisotropy of the previous lecture points the same way: the gradient noise is largest along the steep directions of the landscape, so SGD leaves sharp wells faster than a uniform thermostat would, by a Kramers escape-rate argument we state without derivation.

Entropy-SGD: the preference as an objective. If flat minima are what we want, we need not rely on SGD’s implicit thermodynamics — optimize for flatness directly. Chaudhari et al. (2017) replace the loss by the local free energy

\[ F_\gamma(\theta) \;=\; -\log \int \mathrm{d}\theta' \; e^{-L(\theta') \,-\, \frac{\gamma}{2} \|\theta - \theta'\|^2}, \tag{18.5}\]

entropy-SGD objective (local free energy)

a log-partition-function over a Gaussian neighborhood of scope \(1/\sqrt{\gamma}\) around \(\theta\) (conventions in the literature differ by harmless prefactors). Small \(F_\gamma\) requires a large volume of low loss near \(\theta\): narrow wells contribute almost nothing to the integral and are smoothed away, while wide basins remain. The gradient has a form that makes the objective trainable:

\[ \nabla F_\gamma(\theta) \;=\; \gamma\, \big( \theta - \langle \theta' \rangle_{\rho} \big), \qquad \rho(\theta') \propto e^{-L(\theta') - \frac{\gamma}{2}\|\theta - \theta'\|^2}, \]

the displacement of \(\theta\) from the mean of the local Gibbs distribution — estimated, in the algorithm, by an inner SGLD chain (Welling and Teh 2011): Week 8’s sampler running as a subroutine inside every outer step, with the scope \(\gamma\) annealed as training proceeds. The free energy entered as a bound (Week 5), became a training loss by amortization (Week 7, Equation 14.6), and now appears as an objective computed by sampling — a partition function estimated with the very MCMC machinery of this module.

Figure 18.3 shows the objective on a test landscape: a narrow global minimum (lower loss) against a wide valley (higher loss), plus small-scale ruggedness. The smoothed \(F_\gamma\) removes the wiggles and the needle together, and its global minimum moves to the wide valley, so an optimizer following \(F_\gamma\) avoids the deepest but sharpest point of the landscape.

Show code 
th = np.linspace(-2.5, 3.5, 2401)
dth = th[1] - th[0]
L = smooth_min(60 * (th + 1) ** 2 - 0.12, 0.15 * (th - 1) ** 2, 0.05) \
    + 0.015 * (1 - np.cos(15 * th))

fig, ax = plt.subplots(figsize=(8.4, 4.2))
ax.plot(th, L, color=NAVY, lw=1.6, label=r"loss $L(\theta)$")
ax.plot(th[L.argmin()], L.min(), "o", color=NAVY, ms=8)

for gam, col, ls in [(6.0, ORANGE, "-"), (1.0, RED, "--")]:
    Fg = np.empty_like(th)
    for i, t in enumerate(th):
        Fg[i] = -np.log(np.exp(-L - gam / 2 * (t - th) ** 2).sum() * dth)
    Fg -= Fg.min()                          # align minima for comparison
    ax.plot(th, Fg, color=col, lw=1.8, ls=ls,
            label=rf"$F_\gamma$, scope $1/\sqrt{{\gamma}} = {1/np.sqrt(gam):.1f}$")
    ax.plot(th[Fg.argmin()], 0, "s", color=col, ms=8)

ax.set_xlabel(r"$\theta$")
ax.set_ylabel("objective (shifted)")
ax.set_ylim(-0.25, 1.3)
ax.legend(frameon=False, fontsize=9.5, loc="upper center")
fig.tight_layout(); plt.show()
Figure 18.3: The entropy-SGD objective Equation 18.5 on a test landscape: the loss \(L(\theta)\) (navy) has its global minimum in a needle at \(\theta = -1\) (depth \(-0.09\), curvature \(\sim 120\)) next to a wide valley at \(\theta = +1\) (minimum \(0\), curvature \(0.3\)), with small wiggles throughout. The local free energy \(F_\gamma\) (each curve shifted so its minimum sits at zero) smooths over a Gaussian neighborhood: at scope \(1/\sqrt{\gamma} = 0.4\) (orange) the wiggles and the needle are smoothed away and the global minimum (square) moves to the wide valley at \(\theta \approx 1.0\); a wider scope (red dashed) does the same more aggressively. An optimizer descending \(F_\gamma\) avoids the deepest point of the landscape.
Take-home 2

To the extent that SGD samples \(e^{-L/T}\), flat minima are favored, with a strength set by \(T \propto \eta/|B|\); large-batch (cold) training finds sharper minima (Keskar et al. 2017), and the anisotropic noise accelerates escape from sharp wells. Entropy-SGD makes the preference explicit: its objective \(F_\gamma\) (Equation 18.5) is a local free energy whose gradient is computed by an inner SGLD sampler.

18.5 The caveat: flatness is not invariant

The picture above is powerful, and one qualification bounds the claim. Flatness, measured by raw Hessian curvature, is not a property of the function the network computes — it depends on the parameterization. Dinh et al. (2017) make this sharp with a one-line construction: in a ReLU network, scaling one layer’s weights by \(c\) and the next layer’s by \(1/c\) leaves the input–output function — and therefore the test error — exactly unchanged, while rescaling the Hessian’s eigenvalues arbitrarily. Any minimum can be made to look as sharp as desired without touching what the network does; hence the paper’s title, sharp minima can generalize. The free-energy account inherits the same subtlety through its measure: the basin “volume” in Equation 18.2 is a volume in \(\mathrm{d}\theta\), and a reparameterization that reshapes \(\mathrm{d}\theta\) reshapes the entropy with it.

The repair is to measure curvature in a metric invariant under reparameterization: the Fisher-information (natural-gradient) geometry, normalized flatness measures, or the PAC-Bayes route, where the generalization bound is stated invariantly from the start. Sharpness-aware minimization (SAM, Foret et al. (2021)), a related practical method, minimizes the worst loss in a perturbation ball; its sharpness measure is also affected by reparameterization, which the scale-invariant ASAM of Kwon et al. (2021) corrects. We name these and leave them to the literature. The claim that flat minima generalize better is therefore a robust empirical regularity with a clear physical mechanism, and a leading picture rather than a theorem, the same standing this course gave mean field before its correction (Weeks 5–6) and contrastive divergence’s bias (Week 3).

Trap

Flatness is reparameterization-dependent. Rescaling adjacent ReLU layers by \(c\) and \(1/c\) changes the Hessian arbitrarily without changing the function (Dinh et al. 2017) — raw curvature, and the basin volume behind Equation 18.3, are coordinate quantities. Sharp minima can generalize. The heuristic becomes reliable only in an invariant formulation (Fisher metric, PAC-Bayes, relative sharpness); quote raw \(\det H\) comparisons only between minima of the same parameterization, as done throughout this lecture.

18.6 Example: two minima, one temperature dial

We take a landscape small enough to integrate exactly: a one-dimensional loss with a sharp minimum (\(\lambda_s = 25\)) and a flat one (\(\lambda_f = 1\)), with the two experiments of Figure 18.4.

The left panel tests the robustness claim at equal depth: shifting the landscape by \(\delta = 0.1\) raises the loss at the sharp minimum by \(\lambda_s \delta^2/2 = 0.125\) and at the flat one by \(0.005\) — the generalization gap is twenty-five times larger at the sharp minimum, the curvature ratio exactly, with the training losses identical. In the right panel the sharp minimum is made deeper by \(\Delta L = 0.08\), so a zero-temperature loss minimizer would choose it every time. The exact Boltzmann occupation of the flat basin, \(P_{\text{flat}}(T)\), computed by quadrature, depends on temperature: cold, the sampler sits in the sharp well (\(P_{\text{flat}} = 0.08\) at \(T = 0.02\)); hot, it sits in the flat one (\(0.77\) at \(T = 0.2\)); and the crossover sits where Equation 18.3 says the two free energies cross,

\[ T^\ast \;=\; \frac{2\, \Delta L}{\log(\lambda_s / \lambda_f)} \;=\; \frac{0.16}{\log 25} \;\approx\; 0.050, \]

with the Laplace prediction \(P_{\text{flat}} = [1 + e^{\Delta F / T}]^{-1}\) lying on top of the exact curve across the whole range. Translated through \(T \propto \eta/|B|\), this is Keskar’s experiment in one dimension: batch size and learning rate move the training run along this curve, and which minimum “SGD converged to” is a point on it.

Show code 
lam_s, lam_f = 25.0, 1.0
th = np.linspace(-2.5, 3.5, 2401)
dth = th[1] - th[0]

fig, (a1, a2) = plt.subplots(1, 2, figsize=(10.4, 4.2))

# (a) equal depth + test shift
delta = 0.1
Ltr = smooth_min(lam_s / 2 * (th + 1) ** 2, lam_f / 2 * (th - 1) ** 2, 0.15)
Lte = smooth_min(lam_s / 2 * (th + 1 - delta) ** 2,
                 lam_f / 2 * (th - 1 - delta) ** 2, 0.15)
a1.plot(th, Ltr, color=NAVY, lw=2.0, label="train")
a1.plot(th, Lte, color=ORANGE, lw=1.6, ls="--", label=rf"test ($\delta = {delta}$)")
for m, lam in [(-1.0, lam_s), (1.0, lam_f)]:
    i = np.argmin(np.abs(th - m))
    a1.annotate("", xy=(m, Lte[i]), xytext=(m, Ltr[i]),
                arrowprops=dict(arrowstyle="-|>", color=RED, lw=1.6))
    a1.text(m + 0.08, Ltr[i] + (Lte[i] - Ltr[i]) / 2,
            rf"${Lte[i] - Ltr[i]:.3f}$", fontsize=10, color=RED)
a1.set_xlim(-2.0, 3.6)
a1.set_ylim(-0.02, 0.55)
a1.set_xlabel(r"$\theta$")
a1.set_ylabel("loss")
a1.set_title("equal depth: the gap tracks curvature", fontsize=11.5)
a1.legend(frameon=False, fontsize=9.5, loc="upper right", bbox_to_anchor=(1.02, 0.5))

# (b) sharp deeper by DL: occupation of the flat basin vs temperature
DL = 0.08
L2 = smooth_min(lam_s / 2 * (th + 1) ** 2, lam_f / 2 * (th - 1) ** 2 + DL, 0.15)
mid = (th > -1) & (th < 1)
th_b = th[mid][np.argmax(L2[mid])]              # barrier top splits the basins

Ts = np.linspace(0.005, 0.25, 120)
P_exact, P_lap = [], []
for T in Ts:
    w = np.exp(-(L2 - L2.min()) / T)
    P_exact.append(w[th > th_b].sum() / w.sum())
    r = np.sqrt(lam_s / lam_f) * np.exp(-DL / T)  # Laplace: P_flat / P_sharp
    P_lap.append(r / (1 + r))
Tstar = 2 * DL / np.log(lam_s / lam_f)

a2.plot(Ts, P_exact, color=NAVY, lw=2.0, label="exact (quadrature)")
a2.plot(Ts, P_lap, color=ORANGE, lw=1.8, ls="--", label="Laplace / free energy")
a2.axvline(Tstar, color=GRAY, ls=":", lw=1.2)
a2.axhline(0.5, color=GRAY, ls=":", lw=0.8)
a2.text(Tstar + 0.005, 0.08, rf"$T^* = {Tstar:.3f}$", fontsize=10, color=GRAY)
a2.text(0.16, 0.60, "hot: flat basin\n(small batch / large lr)",
        fontsize=9.5, color=NAVY)
a2.text(0.012, 0.30, "cold: sharp,\ndeeper basin\n(large batch)",
        fontsize=9.5, color=NAVY)
a2.set_xlabel(r"temperature $T \propto \eta / |B|$")
a2.set_ylabel(r"$P_\mathrm{flat}$")
a2.set_title("the temperature picks the basin", fontsize=11.5)
a2.legend(frameon=False, fontsize=9.5, loc="lower right")
fig.tight_layout(); plt.show()
Figure 18.4: The worked example: a double-well loss with a sharp (\(\lambda_s = 25\)) and a flat (\(\lambda_f = 1\)) minimum. Left: at equal depth, the train→test shift \(\delta = 0.1\) opens a gap \(\lambda \delta^2/2\) at each minimum — \(0.125\) (sharp) against \(0.005\) (flat), a factor \(25\) at identical training loss. Right: with the sharp minimum now deeper by \(\Delta L = 0.08\), the equilibrium probability that a sampler of \(e^{-L/T}\) occupies the flat basin, computed exactly by quadrature (navy) and by the Laplace/free-energy formula \(P = [1 + e^{\Delta F/T}]^{-1}\) with \(\Delta F = \Delta L - \frac{T}{2}\log(\lambda_s/\lambda_f)\) (orange dashed, indistinguishable): cold (\(T = 0.02\)) the sampler sits in the sharp-but-deep well (\(P_\mathrm{flat} = 0.08\)); hot (\(T = 0.2\)) in the flat-but-shallow one (\(0.77\)); the crossover \(T^\ast = 2\Delta L / \log(\lambda_s/\lambda_f) \approx 0.050\) (gray dotted) is where the two free energies cross. Through \(T \propto \eta/|B|\), batch size and learning rate select a point on this curve.

18.7 Outlook

Week 8 established that the intractable \(Z\) need not be computed to be used — Metropolis made it cancel (Section 15.3), and annealing turned the sampler into an optimizer by cooling (Chapter 16). The previous lecture showed that SGD is, to leading order, such a sampler, with temperature \(\eta/|B|\) (Equation 17.2), relaxing toward \(e^{-L/T}\) for isotropic noise (Equation 17.4). Today that temperature weights basins by free energy (Equation 18.4), favoring the flat, high-entropy minima that are robust to the train→test shift, and entropy-SGD (Equation 18.5) turns the preference into an explicit objective.

Module 4 carries this machinery further. Week 10 removes the partition function outright — score matching works with \(\nabla_x \log p\), where \(Z\) differentiates to zero (Hyvärinen 2005) — moving the course’s dynamics from parameter space to configuration space at last. Week 11 turns nonequilibrium physics into an estimator: the Jarzynski and Crooks fluctuation theorems (Jarzynski 1997; Crooks 1999) extract free-energy differences from driven, out-of-equilibrium trajectories. Week 10 also runs the previous lecture’s Langevin/Fokker–Planck equations backwards in time, and Week 12 builds the diffusion model on that reversal to generate data.

At finite temperature SGD weights basins by free energy, and a flat minimum is a low-free-energy minimum.