4 Storage capacity: how memory dies – From Boltzmann to Diffusion

4.1 Is \(0.15\,N\) the answer?

The last lecture established three facts. We design the energy: Hebb’s rule Equation 3.4 writes \(P\) patterns into the couplings in one shot. Descent is a theorem: for symmetric couplings and asynchronous updates, the dynamics rolls monotonically downhill and halts in a local minimum (Section 3.4). And at a stored pattern, every neuron’s aligned local field splits as

\[ h_i\, \xi_i^\mu \;=\; 1 \;+\; C_i , \]

a clean unit signal plus a crosstalk term \(C_i\) whose variance we estimated but whose consequences we did not follow (Section 3.5). What remains is the computation of \(C_i\) — and of what the network does about it.

There is also a number on the table, and it came from your own reading. Hopfield (1982) reports, from simulations of networks of \(N = 100\) neurons, that recall degrades severely once the number of stored patterns exceeds about \(0.15\,N\). The obvious question is: is \(0.15\) the answer? It cannot be accepted as it stands, for two reasons that turn out to carry real physics. First, we do not yet know which question it answers. “How many patterns fit?” splits, on inspection, into two demands with no obligation to agree: strict stability — every bit of every stored pattern sits aligned with its field, no exceptions, so the memories are exact fixed points — and retrieval — the pattern is recalled up to a small dressing of wrong bits, and cues in its basin still flow to it. The first is a worst-case demand, governed by the largest of \(NP\) random crosstalk pulls; the second is a typical-case demand, governed by the bulk of the crosstalk distribution. You have met this split before in other fields: worst-case versus average-case in the analysis of algorithms, minimum-distance versus Shannon capacity in coding theory (MacKay 2003). The two answers here differ by a factor of order \(\ln N\).

Second, we do not know how memory fails when it fails. The natural guess — more patterns, gradually more wrong bits, memory dimming like an old photograph — is wrong. Retrieval instead survives, nearly intact, up to a sharp critical load, and then collapses discontinuously. Why the collapse is discontinuous is the substance of the analysis below; it is the course’s first phase transition found inside a learning machine.

The capacity of a Hebbian memory depends on what we demand of it: exact recall allows a sublinear number of patterns, tolerant recall an extensive one, and beyond the tolerant limit memory fails all at once.

Everything until the final section happens at \(T = 0\), so the partition function stays out of sight; the difficulty is interference, and the central limit theorem quantifies it.

Take-home 1

“How many patterns fit?” is two questions. Demanding every stored bit be exactly stable gives \(P_\text{max} \approx N / (2 \ln N)\), sublinear by a logarithm. Tolerating a percent of wrong bits gives extensive capacity \(P_\text{max} = \alpha_c N\) with \(\alpha_c \approx 0.138\). Capacity is a property of the network together with the demand placed on it.

4.2 The statistics of interference

The first task is the distribution of the crosstalk. Recall its definition — at stored pattern \(\xi^\mu\), the aligned field on site \(i\) carries, besides the unit signal,

\[ C_i \;=\; \frac{1}{N} \sum_{\nu \neq \mu} \sum_{j \neq i} \xi_i^\mu\, \xi_i^\nu\, \xi_j^\nu\, \xi_j^\mu \;\equiv\; \frac{1}{N} \sum_{\nu \neq \mu} \sum_{j \neq i} t_{\nu j}, \]

a sum of \((P-1)(N-1)\) terms \(t_{\nu j} = \pm 1\) (the tested pattern \(\mu\) and site \(i\) are held fixed throughout, so only the summation indices label the terms), each a product of four independent, unbiased signs. The claim that these signs are “pseudo-random” needs justification. Each term has mean zero: for \(\nu \neq \mu\) and \(j \neq i\) the four factors involve at least one sign that appears exactly once, and its average kills the product. Better, the terms are pairwise uncorrelated: for two distinct index pairs \((\nu, j) \neq (\nu', j')\), the expectation \(\langle t_{\nu j}\, t_{\nu' j'} \rangle\) always contains some pattern entry to an odd power — if \(\nu \neq \nu'\) the two patterns are independent and each factor averages to zero separately; if \(\nu = \nu'\) but \(j \neq j'\) the product contains \(\xi_j^\mu \xi_{j'}^\mu\) times signs of pattern \(\nu \ne \mu\), and again averages to zero. So the variance of the sum is exactly the sum of the variances — no approximation yet:

\[ \operatorname{Var}(C_i) \;=\; \frac{(P-1)(N-1)}{N^2} \;\approx\; \frac{P}{N} \;=\; \alpha . \]

(At a single site the terms are in fact fully independent: conditional on \(\xi^\mu\), the products \(\xi_i^\nu \xi_j^\nu\) are independent uniform signs. Correlations appear only between the crosstalk at different sites, which share the patterns \(\xi^\nu\), and they become important once errors feed back.) The central limit theorem then does the rest: for large \(N\) at fixed load \(\alpha\),

\[ C_i \;\sim\; \mathcal{N}(0,\, \alpha), \qquad \alpha = \frac{P}{N}. \tag{4.1}\]

crosstalk statistics

The physical reading: the signal is \(1\) no matter how many patterns are stored; the noise grows as \(\sqrt{P/N}\) — and interference depends on \(P\) and \(N\) only through their ratio. The load \(\alpha\) is the model’s control parameter, and all quantities below are functions of it.

A stored bit comes out wrong on the first update sweep when the crosstalk overwhelms its signal, \(1 + C_i < 0\) — a fluctuation of \(1/\sqrt{\alpha}\) standard deviations. The Gaussian tail integral (the Gaussian integral of Section 2.4) gives the first-sweep bit-error probability already tabulated in Section 3.6:

\[ P_\text{err} \;=\; \Pr[\,C_i < -1\,] \;=\; \tfrac{1}{2} \operatorname{erfc}\!\Big( \frac{1}{\sqrt{2\alpha}} \Big). \tag{4.2}\]

first-sweep error rate

Figure 4.1 plots it. The first feature is the steepness of a Gaussian tail: between \(\alpha = 0.01\) and \(\alpha = 0.2\) — a factor of twenty in load — the error probability climbs by twenty-one orders of magnitude, from \(10^{-23}\) to \(10^{-2}\). Steepness like this is why the capacity question has a sharp answer at all: whatever criterion we impose, the crossover from “satisfied absurdly well” to “violated badly” is compressed into a narrow window of \(\alpha\). The second feature is a warning: the curve is smooth through \(\alpha_c \approx 0.138\), the load where retrieval actually collapses. Nothing in the first-sweep error rate marks the edge, and explaining this mismatch is the task of Section 4.4.

Show code

fig, ax = plt.subplots(figsize=(8.4, 4.6))
al = np.linspace(0.004, 0.30, 500)
Perr = 0.5 * erfc(1 / np.sqrt(2 * al))
ax.semilogy(al, Perr, color=NAVY, lw=2.5)
ax.axvline(0.138, color=ORANGE, ls="--", lw=2)
labels = {0.01: r"$10^{-23}$: interference invisible", 0.05: r"$4\times 10^{-6}$",
          0.10: r"$8\times 10^{-4}$: first wrong bits", 0.20: r"$1.3\times 10^{-2}$"}
for a in labels:
    ax.plot([a], [0.5 * erfc(1 / np.sqrt(2 * a))], "o", color=NAVY, ms=6)
for a, xy, xt in [(0.01, (0.012, 2e-24), (0.03, 6e-23)), (0.05, (0.055, 4e-6), (0.075, 2e-7)),
                  (0.10, (0.10, 9e-4), (0.055, 8e-2)), (0.20, (0.205, 1.3e-2), (0.225, 2e-3))]:
    ax.annotate(labels[a], xy=xy, xytext=xt, color=GRAY, fontsize=10.5,
                arrowprops=dict(arrowstyle="-", color=GRAY, lw=0.8))
ax.annotate("the true edge $\\alpha_c\\approx 0.138$:\nretrieval collapses here, and this\n"
            "curve is smooth through it", xy=(0.138, 1e-7), xytext=(0.15, 3e-8),
            color=ORANGE, fontsize=11, ha="left",
            arrowprops=dict(arrowstyle="->", color=ORANGE, lw=1))
ax.set_xlabel(r"storage load $\alpha = P/N$")
ax.set_ylabel(r"first-sweep bit-error probability $P_\mathrm{err}$")
ax.set_ylim(1e-24, 1); ax.set_xlim(0, 0.30)
fig.tight_layout(); plt.show()

Figure 4.1: The first-sweep bit-error probability \(P_\text{err} = \frac{1}{2}\operatorname{erfc}(1/\sqrt{2\alpha})\) of Equation 4.2, on a logarithmic scale. The Gaussian tail is remarkably steep — twenty-one orders of magnitude between \(\alpha = 0.01\) and \(\alpha = 0.2\) — which is why capacity questions have sharp answers. The dashed line marks the true storage limit \(\alpha_c \approx 0.138\), where retrieval collapses; it leaves no trace on this curve, because the transition is caused by feedback, which a first-sweep error rate does not contain.

4.3 Perfection costs a logarithm

Now impose the strong demand: strict stability. Every stored pattern should be an exact fixed point of the dynamics — all \(N\) bits of all \(P\) patterns aligned with their fields, so that a memory, once retrieved, is letter-perfect. How many patterns does that permit?

Consider first a single stored pattern and ask that all \(N\) of its bits hold. Bit failures are (to leading order) independent rare events with probability \(P_\text{err}\) each, so the expected number of unstable bits is \(N P_\text{err}\), and the pattern is exactly stable, with high probability, as long as

\[ N\, P_\text{err} \;\lesssim\; 1 . \]

Insert the Gaussian tail asymptotic — for \(x \gg 1\), \(\tfrac12\operatorname{erfc}(x/\sqrt{2}) \approx e^{-x^2/2} / (x \sqrt{2\pi})\), here with \(x = 1/\sqrt{\alpha}\) — and keep only the exponentially dominant balance:

\[ N \sqrt{\frac{\alpha}{2\pi}}\; e^{-1/2\alpha} \;\approx\; 1 \qquad \Longrightarrow \qquad \frac{1}{2\alpha} \;\approx\; \ln N + \mathcal{O}(\ln \ln N), \]

where the prefactor’s logarithm is subleading and swept into the correction. Solving for the load,

\[ \boxed{\; \alpha_\text{strict} \approx \frac{1}{2 \ln N}, \qquad P_\text{max} \approx \frac{N}{2 \ln N} \;} \tag{4.3}\]

strict-stability capacity

— sublinear in \(N\). If instead we demand that every pattern be letter-perfect simultaneously, the union runs over all \(NP \approx \alpha N^2\) bits, the balance becomes \(\frac{1}{2\alpha} \approx \ln(N P) \approx 2 \ln N\), and the capacity halves again to \(P_\text{max} \approx N/(4 \ln N)\). Both statements — with the constants, the basin-of-attraction refinements, and full rigor in place of our union-bound physics — are the content of a celebrated information-theory paper (McEliece et al. 1987); our two-line estimate lands exactly on their scalings.

⇒ Demanding zero errors makes capacity sublinear; the logarithm counts the trials that must all succeed. The mechanism recurs across physics and statistics: the maximum of \(M\) independent Gaussian pulls grows like \(\sqrt{2 \ln M}\), so the worst bit in the network, not the typical one, sets the limit. At \(N = 1000\) the asymptotic formulas give about \(72\) patterns (\(36\) if every pattern must be perfect); solving \(N P_\text{err} = 1\) and \(NP\,P_\text{err} = 1\) exactly with Equation 4.2 gives about \(105\) and \(58\), since the \(\ln \ln N\) correction is still large at this size. Either way, Hopfield’s \(150\) lies beyond the strict answer — so either his simulations were wrong, or exact fixed points are the wrong demand. They were not wrong.

Notice what strict stability does not say. A pattern that fails it has, at \(\alpha = 0.1\), roughly one misaligned bit per thousand: the dynamics flips that bit, the state comes to rest one or two spin flips away from the stored pattern, and any user of the memory would call the recall a success. Strict stability is a bookkeeping demand, not a statement about whether the memory works. The working question — does a fixed point near \(\xi^\mu\) survive, with a basin around it? — is a question about the typical crosstalk and about what the first few errors do next. That question has more physics in it.

Trap

A small error rate is not safety. \(P_\text{err} = 10^{-4}\) sounds negligible, but there are \(NP\) stored bits: rare events at a fixed rate become certainties at scale, which is why the strict criterion collapses from “essentially never fails” to “fails somewhere, always” over a narrow range of \(\alpha\). Extreme-value reasoning — the worst of many — obeys different arithmetic from typical-value reasoning.

4.4 The collapse

So relax the demand and follow the dynamics. Start the network at a stored pattern with a few wrong bits — or far from it, at overlap \(m_0\) — and ask what one sweep of updates does. For a state with overlap \(m_0\) with pattern \(\xi^\mu\) and no other special correlations, the same central-limit argument as before gives an aligned field \(m_0 + \mathcal{N}(0, \alpha)\) on every site, so the overlap after one parallel sweep is the probability of ending aligned minus the probability of ending misaligned:

\[ m_1 \;=\; \operatorname{erf}\!\Big( \frac{m_0}{\sqrt{2\alpha}} \Big). \tag{4.4}\]

first-sweep retrieval map — legitimate once

This formula is valid for the first sweep only, a restriction that matters below. As a first-sweep statement it is already useful: it reproduces Equation 4.2 at \(m_0 = 1\) (since \(m_1 = 1 - 2P_\text{err}\)), and at small \(\alpha\) it becomes a step function of \(m_0\), recovering the one-pattern result of Section 3.5 that any positive initial overlap retrieves in a single sweep.

The natural next step is to iterate the map, treating each sweep as a fresh draw of Gaussian noise. Figure 4.2 cobwebs the map. The retrieval fixed point \(m^* = \operatorname{erf}(m^*/\sqrt{2\alpha})\) sits near \(1\) at small load, slides gracefully downward as \(\alpha\) grows, and survives — check the slope at the origin — all the way to \(\alpha = 2/\pi \approx 0.64\), where it vanishes continuously. The naive theory thus predicts that memory fades, with the overlap degrading smoothly and useful recall extending out to \(\alpha \approx 0.64\).

Both the number and the kind are wrong. The truth is \(\alpha_c \approx 0.138\) — five times smaller — and the failure is discontinuous. The calculation was clean and yet wrong by a factor of five: what did we throw away?

Show code

def fixed_point(alpha):                              # nonzero retrieval fixed point
    m = 0.99
    for _ in range(500):
        m = erf(m / np.sqrt(2 * alpha))
    return m
fig, axes = plt.subplots(1, 3, figsize=(12.6, 4.2))
configs = [(0.1, 0.3, 7, "fixed point $m^*\\approx 1$:\nretrieval", "near1"),
           (0.4, 0.3, 10, "$m^*\\approx 0.79$: retrieval predicted\n(the true network retrieves nothing)", "dot"),
           (0.7, 0.95, 10, "only $m^*=0$ survives:\nnaive loss of memory at\n"
            "$\\alpha=2/\\pi\\approx 0.64$:\ncontinuous, and\n$5\\times$ too optimistic", "zero")]
for ax, (alpha, m0, nst, note, kind) in zip(axes, configs):
    mm = np.linspace(0, 1, 300)
    ax.plot(mm, erf(mm / np.sqrt(2 * alpha)), color=NAVY, lw=2.5)     # the map
    ax.plot([0, 1], [0, 1], color=GRAY, ls="--", lw=1)               # diagonal
    x = m0; xs_, ys_ = [x], [0.0]
    for _ in range(nst):                                             # cobweb
        y = erf(x / np.sqrt(2 * alpha)); xs_ += [x, y]; ys_ += [y, y]; x = y
    ax.plot(xs_, ys_, color=ORANGE, lw=1.3)
    if kind == "dot":
        mstar = fixed_point(alpha); ax.plot([mstar], [mstar], "o", color=ORANGE, ms=9)
        ax.annotate(note, xy=(mstar, mstar), xytext=(0.12, 0.9), color=GRAY, fontsize=10.5,
                    arrowprops=dict(arrowstyle="->", color=GRAY, lw=1))
    elif kind == "zero":
        ax.plot([0], [0], "o", color=ORANGE, ms=9)
        ax.text(0.06, 0.9, note, color=GRAY, fontsize=10.5, va="top")
    else:
        ax.plot([fixed_point(alpha)], [fixed_point(alpha)], "o", color=ORANGE, ms=7)
        ax.annotate(note, xy=(0.97, 0.97), xytext=(0.5, 0.3), color=GRAY, fontsize=10.5,
                    arrowprops=dict(arrowstyle="->", color=GRAY, lw=1))
    ax.set_title(rf"$\alpha = {alpha}$", fontsize=12)
    ax.set_xlabel(r"$m_t$"); ax.set_xlim(0, 1); ax.set_ylim(0, 1.02)
axes[0].set_ylabel(r"$m_{t+1} = \mathrm{erf}(m_t/\sqrt{2\alpha})$")
fig.tight_layout(); plt.show()

Figure 4.2: The naive theory and its instructive failure. The first-sweep retrieval map \(m_{t+1} = \operatorname{erf}(m_t/\sqrt{2\alpha})\) of Equation 4.4, cobwebbed at three loads. Left, \(\alpha = 0.1\): rapid convergence to a retrieval fixed point near \(m^* = 1\). Center, \(\alpha = 0.4\): the naive fixed point survives at \(m^* \approx 0.79\) — although the true network retrieves nothing at this load. Right, \(\alpha = 0.7\): past \(\alpha = 2/\pi \approx 0.64\) only \(m^* = 0\) remains, and the naive theory predicts memory *fading* continuously to nothing. Both the predicted capacity and the predicted kind of failure are wrong, because iterating the map re-randomizes the crosstalk each sweep — precisely what the network does not do.

We threw away the feedback. Iterating Equation 4.4 assumes the crosstalk is re-randomized at every sweep — that the noise a spin feels now is statistically independent of the errors made earlier. It is not. A bit that flipped did so because the crosstalk pushed it; the flipped bit is therefore correlated with the noise, and through the couplings it pollutes the fields of every other site in a way that does not average out. Errors amplify the very noise that produced them. In the terms of Section 4.2, the first sweep was safe because the cue was built without reference to the other patterns; after it, every spin \(s_j\) depends on the \(\xi^\nu\), the terms of the crosstalk are no longer pairwise uncorrelated, and its variance is no longer the count of terms that gave \(\alpha\). The correct self-consistent treatment — set up by Amit, Gutfreund, and Sompolinsky (1985) and carried out in full in Amit, Gutfreund, and Sompolinsky (1987) — replaces the bare noise variance \(\alpha\) by an amplified one, and at zero temperature its equations read (stated here; Week 6 derives the reaction mechanism behind \(r\), while the full equations require the replica method, which this course does not develop):

\[ m = \operatorname{erf}\!\Big( \frac{m}{\sqrt{2 \alpha r}} \Big), \qquad \chi = \sqrt{\frac{2}{\pi \alpha r}}\; e^{-m^2 / 2\alpha r}, \qquad r = \frac{1}{(1 - \chi)^2}, \tag{4.5}\]

AGS self-consistency at \(T=0\) (stated)

where \(\chi\) measures the network’s integrated response to its own errors — a susceptibility, distinct from the crosstalk \(C_i\) above — and \(r \ge 1\) is the factor by which that response inflates the crosstalk. Solving the three equations numerically — an afternoon’s exercise once you trust them, and Figure 4.3 is exactly that computation — produces the picture the naive map could not. As \(\alpha\) grows, the retrieval solution (overlap near one) is approached from below by an unstable partner solution; the two branches bend toward each other, meet, and annihilate at

\[ \boxed{\; \alpha_c \;\approx\; 0.138, \qquad m(\alpha_c) \approx 0.97. \;} \tag{4.6}\]

storage capacity (Amit–Gutfreund–Sompolinsky)

This is a saddle-node bifurcation: beyond \(\alpha_c\) the retrieval state does not degrade — it ceases to exist, and the dynamics falls into the spin-glass states, which carry no macroscopic overlap with any stored pattern. (The equilibrium spin-glass solution has \(m = 0\); the zero-temperature dynamics started at the pattern halts in a nearby spin-glass state with a residual overlap, between about \(0.2\) and \(0.6\) in simulations at \(N = 1000\), far from retrieval.) The overlap therefore jumps discontinuously, and the number \(0.97\) is itself the signature of the jump: even at the edge, retrieval is still \(98.4\%\) correct per bit. Memory near capacity remains nearly perfect until the retrieval state disappears.

In the language of phase transitions, \(\alpha_c\) is a spinodal — the limit of existence of a metastable branch — rather than the thermodynamic transition. Comparing free energies, the retrieval states are the global minima only for \(\alpha < 0.051\); between \(0.051\) and \(0.138\) they lie above the spin-glass states and are metastable, yet a cue near a pattern still flows to them, so the memory works. The thermodynamic first-order transition sits at \(\alpha \approx 0.051\), and \(\alpha_c \approx 0.138\) marks where the retrieval states vanish altogether (Amit, Gutfreund, and Sompolinsky 1987). Module 2 returns to the point.

Show code

def branch(yv):                                      # AGS T=0 solution, parametrized by y
    m = erf(yv)
    chi = (2 * yv * np.exp(-yv**2)) / (np.sqrt(np.pi) * m)   # AGS susceptibility χ
    alpha = m**2 * (1 - chi)**2 / (2 * yv**2)
    return alpha, m
y_fold = 1.511                                       # maximum of alpha(y)
a_st, m_st = branch(np.linspace(y_fold, 6, 400))     # stable branch  (y > y_fold)
a_un, m_un = branch(np.linspace(0.33, y_fold, 400))  # unstable branch (y < y_fold)
ac, mc = branch(np.array([y_fold])); ac, mc = ac[0], mc[0]   # the fold
fig, ax = plt.subplots(figsize=(8.6, 4.8))
ax.axhline(1.0, color=GRAY, ls="--", lw=0.9); ax.axhline(0.0, color=GRAY, lw=3)
ax.plot(a_st, m_st, color=NAVY, lw=2.5)
ax.plot(a_un, m_un, color=NAVY, ls=":", lw=2.5)
ax.plot([ac], [mc], "o", color=ORANGE, ms=10)
ax.annotate("", xy=(ac, 0.02), xytext=(ac, mc - 0.02),
            arrowprops=dict(arrowstyle="->", color=ORANGE, lw=2))
ax.text(ac + 0.004, 0.5, "collapse —\ndiscontinuous", color=ORANGE, fontsize=11)
ax.annotate("$m(\\alpha_c)\\approx 0.967$: even at the edge, memory\n"
            "is 98% right, until the state disappears", xy=(ac, mc),
            xytext=(0.03, 1.12), color=GRAY, fontsize=10.5, ha="left",
            arrowprops=dict(arrowstyle="->", color=GRAY, lw=1))
ax.text(0.03, 0.93, "retrieval state (stable branch)", color=NAVY, fontsize=11)
ax.text(0.055, 0.66, "unstable partner", color=NAVY, fontsize=11, rotation=18)
ax.annotate("saddle-node at $\\alpha_c=0.138$:\nthe two branches merge, and the\n"
            "retrieval solution ceases to exist", xy=(ac, mc), xytext=(0.022, 0.26),
            color=NAVY, fontsize=10.5, ha="left",
            arrowprops=dict(arrowstyle="->", color=NAVY, lw=1))
ax.text(0.145, 0.05, "spin glass, $m=0$\n(nothing retrieved)", color=GRAY, fontsize=11)
ax.set_xlabel(r"storage load $\alpha = P/N$"); ax.set_ylabel(r"retrieval overlap $m$")
ax.set_xlim(0, 0.21); ax.set_ylim(-0.03, 1.2)
fig.tight_layout(); plt.show()

Figure 4.3: Retrieval collapse: the retrieval overlap from the self-consistent equations Equation 4.5. The stable retrieval branch (solid) descends only slightly — from \(m = 1\) to \(m \approx 0.967\) — while its unstable partner (dotted) rises to meet it; at \(\alpha_c \approx 0.138\) the two annihilate in a saddle-node bifurcation and no retrieval solution exists beyond. The overlap drops discontinuously to the spin-glass solution (\(m = 0\) in equilibrium): \(\alpha_c\) is the spinodal of the retrieval branch. Compare Figure 4.2 — feedback moved the edge from \(0.64\) to \(0.138\) *and* turned a continuous decline into a discontinuous collapse.

Below \(\alpha_c\), each vertical line in Figure 4.3 crosses three solutions of Equation 4.5: the spin-glass state at \(m = 0\), the retrieval state near \(m = 1\), and the unstable partner between them. In a free-energy landscape along \(m\), the two stable solutions are valleys and the unstable partner is the ridge that separates them. A state placed exactly on the ridge satisfies the self-consistency equations, but any perturbation sends it into one of the two valleys. The ridge therefore acts as a threshold on the cue: an initial overlap above it flows to retrieval, an overlap below it flows to the spin glass. Read this way, the figure describes how the basin of attraction of a memory shrinks with load. The ridge sits at \(m \approx 0.49\) for \(\alpha = 0.01\) and rises to \(0.72\) at \(\alpha = 0.05\) and \(0.86\) at \(\alpha = 0.1\), while the valley floor descends only from \(1.00\) to \(0.998\) over the same range. At \(\alpha_c\) the ridge reaches the valley floor and both disappear, so that the stored pattern itself, and every cue near it, falls to \(m = 0\). The threshold reading is a heuristic. The landscape is an equilibrium object, whereas the basin of attraction is a property of the update dynamics, and the two need not coincide: in Figure 3.3, at \(\alpha = 0.02\), a cue with \(m_0 = 0.4\) is retrieved exactly, although the ridge lies at \(0.58\). What carries over is the trend, namely that the cue must improve as the load grows and that no cue suffices beyond \(\alpha_c\).

Two refinements are worth noting. First, dynamics and equilibrium do not agree perfectly here: tracking the transient — overlap and noise variance sweep by sweep, with the long-term correlations handled carefully — is the statistical neurodynamics of Amari and Maginu (1988), which yields a discontinuous transition at \(\alpha \approx 0.16\); the residual gap to \(0.138\) largely reflects that theory’s approximate treatment of long-time error correlations, and a dynamical threshold need not coincide with an equilibrium boundary in any case. Second, finite size rounds the cliff: at \(N \sim 10^3\) the collapse is a steep sigmoid spread over a window of a few times \(10^{-2}\) in \(\alpha\), drifting toward \(0.138\) from above as \(N\) grows. At \(N = 1000\), a network started at a stored pattern still retrieves it at \(\alpha = 0.15\) and fails from about \(\alpha = 0.17\) — which is why Hopfield (1982) saw trouble “near \(0.15\,N\)” at \(N = 100\), and why a rounded curve in a simulation does not falsify a sharp theory.

Trap

First-sweep estimates cannot locate a transition. Equation 4.4 is valid for one sweep and invalid when iterated: each reuse assumes fresh noise, i.e. no correlation between the errors and the crosstalk that caused them. This assumption changes the order of the transition and misses the capacity by a factor of five. When an iterated mean-field estimate fails this badly, the discarded correlations carry the physics.

Take-home 2

Retrieval errors feed back: flipped bits amplify the crosstalk that flipped them, inflating the noise variance from \(\alpha\) to \(\alpha r\) with \(r = (1-\chi)^{-2}\). The self-consistent retrieval state then disappears in a saddle-node, and the collapse is discontinuous: retrieval is still \(98\%\) correct per bit just before the state vanishes.

4.5 Memory is a phase

The equations Equation 4.5 are the zero-temperature edge of something larger. Heat the network — we will see in Week 3 that the natural stochastic dynamics at temperature \(T\) exists and is well defined — and the same self-consistent analysis runs at every \((\alpha, T)\). The result is Figure 4.4: a genuine phase diagram for a machine-learning system, computed by Amit, Gutfreund, and Sompolinsky (1985, 1987) with the replica method of spin-glass theory, in the direct lineage of Sherrington and Kirkpatrick (1975). There are three phases. At high temperature, the paramagnet has noise overwhelming any structure, and no memory survives. Below the line \(T_g = 1 + \sqrt{\alpha}\), a spin-glass phase has exponentially many minima, none of them correlated with anything stored — a rugged landscape of pure crosstalk. And in the region under the curve \(T_M(\alpha)\), the retrieval phase has stored patterns as attractors, and the network is a working memory. A further line \(T_c(\alpha)\), not drawn in Figure 4.4, runs inside this region: below it the retrieval states are the global free-energy minima, above it they are metastable. It meets the cold axis at \(\alpha \approx 0.051\) and is the thermodynamic first-order transition; \(T_M\) bounds the existence of the retrieval states. The retrieval boundary runs from \(T = 1\) at vanishing load down to \(\alpha_c \approx 0.138\) at \(T = 0\) — the capacity \(\alpha_c \approx 0.138\) is the point where the memory phase pinches off against the load axis.

The diagram has two natural paths. The horizontal one — loading a cold network with ever more patterns — exits the retrieval phase at \(\alpha_c\). The vertical one — heating a lightly loaded network — shows retrieval surviving to high temperature before melting near \(T \approx 1\). Week 3 introduces the finite-temperature dynamics this path requires, Glauber dynamics, but uses it as the starting point of the Boltzmann machine; the retrieval side of the vertical path is quoted here and not revisited. The phase diagram also settles a point from the last lecture. Within the retrieval region, a little noise is useful: the spurious attractors — the mixture states of Chapter 3 — are shallower than the retrieval states and melt at lower temperature, so a warm network loses its spurious states while keeping its memories. The same temperature that restores the Boltzmann distribution also cleans the landscape.

Figure 4.4: The phase diagram of the Hopfield model with Hebbian couplings, computed from the finite-temperature replica-symmetric theory of Amit, Gutfreund, and Sompolinsky (1985, 1987) (the boundary \(T_M(\alpha)\) is solved numerically from the self-consistent equations; \(T_g = 1 + \sqrt{\alpha}\)). Below \(T_M(\alpha)\), stored patterns are attractors and the network is a working memory; the retrieval phase ends at \(\alpha_c \approx 0.138\) on the cold axis. Between \(T_M\) and \(T_g\), only spin-glass states survive — many minima, none correlated with a stored pattern. The two gray arrows mark the horizontal path across \(\alpha_c\), followed in this week’s tutorial at \(T = 0\), and the vertical path of heating, whose dynamics Week 3 introduces.

What the replica method computes, and our tools cannot, is the average of \(\log Z\) over the stored patterns themselves: the couplings \(J_{ij}\) are random variables built from the \(\xi^\mu\), the free energy must be averaged over that quenched disorder, and for an extensive number of patterns (\(P = \alpha N\)) this average is a nontrivial computation. The technology that computes it — the replica trick, and order parameters beyond \(m\) (the Edwards–Anderson overlap \(q\), and the noise amplification \(r\) you have already met) — is spin-glass theory proper. Module 2 acquires its modern descendants — mean-field theory, its systematic corrections, and the cavity ideas that became inference algorithms — and from that vantage the logic of these equations (feedback, reaction, self-consistency) becomes available; the replica computation itself remains, in this course, a named landmark rather than a technique. So Equation 4.5 and Figure 4.4 stand as quoted results, with the physics behind them developed in Module 2’s own language.

An information-theoretic comparison is instructive. At capacity, the network holds \(P \cdot N = 0.138\, N^2\) pattern bits in \(N^2\) real-valued couplings — about \(0.14\) bits per synapse, and fewer once the \(\sim 1.6\%\) wrong bits at capacity are discounted. The optimal capacity of a network permitted to choose its couplings freely — not restricted to Hebb’s one-shot rule — is \(2\) bits per synapse, a bound for error-free storage computed by Elizabeth Gardner with replica methods (Gardner 1988), which this course cites rather than derives. The Hebb rule thus gives up more than an order of magnitude in storage efficiency in exchange for learning in a single pass, with no optimization at all.

Take-home 3

The Hopfield model has a phase diagram: retrieval, spin glass, and paramagnet in the \((\alpha, T)\) plane. Memory is a phase of matter; capacity is its boundary at \(T = 0\). Within the phase, mild noise cleans the landscape — spurious states melt before memories do. The full computation runs on replica machinery (quenched average of \(\log Z\)) that this course quotes rather than builds; Module 2 acquires its modern mean-field descendants.

4.6 Example: the capacity ledger

We evaluate every capacity claim of the lecture on one network, the \(N = 1000\) neurons of Section 3.6.

question asked	formula	\(P_\text{max}\)	reading
every pattern letter-perfect	\(N / (4 \ln N)\)	\(\approx 36\) (\(58\) solved exactly)	perfection, all at once
a typical pattern letter-perfect	\(N / (2 \ln N)\)	\(\approx 72\) (\(105\) solved exactly)	perfection, one at a time
retrieval with \(\sim\!2\%\) tolerated error	\(\alpha_c N\), \(\alpha_c \approx 0.138\)	\(\approx 138\)	the AGS answer
Hopfield’s simulation criterion	\(\approx 0.15\, N\) (empirical, \(N=100\))	\(\approx 150\)	finite-size, tolerant
naive iterated map	\((2/\pi)\, N\)	\(\approx 637\)	wrong value, wrong kind

The five numbers answer three different questions; the naive value is retained as a record of what the theory without feedback misses. Hopfield’s \(0.15\) was the right physics, seen at \(N = 100\) through finite-size rounding, for the tolerant question; sharpened, it is \(0.138\). The strict question was never the one his simulations asked, and its answer is smaller by a logarithm.

The second table makes the same point numerically. For each load: the first-sweep expectation — how many of the \(1000\) bits flip on the first sweep, starting at a stored pattern, from Equation 4.2 — against the fate of the memory as \(N \to \infty\), from Figure 4.3. (At \(N = 1000\) itself, finite-size rounding moves the edge upward, as Section 4.4 notes.)

\(\alpha = P/N\)	\(P\)	\(P_\text{err}\)	first-sweep flips (of 1000)	fate as \(N \to \infty\)
\(0.05\)	\(50\)	\(4 \times 10^{-6}\)	\(\sim 0\)	retrieval, \(m \approx 1\)
\(0.10\)	\(100\)	\(8 \times 10^{-4}\)	\(\sim 1\)	retrieval, \(m\) high
\(0.138\)	\(138\)	\(3.6 \times 10^{-3}\)	\(\sim 4\)	the edge: \(m \approx 0.97\)
\(0.15\)	\(150\)	\(4.9 \times 10^{-3}\)	\(\sim 5\)	dead: no retrieval state
\(0.20\)	\(200\)	\(1.3 \times 10^{-2}\)	\(\sim 13\)	dead

In the \(\alpha = 0.15\) row, the first sweep gets \(99.5\%\) of the bits right, yet in the large-\(N\) limit the memory is already lost. Between \(\alpha = 0.138\) and \(0.15\), nothing in the per-bit error rate changes appreciably; what changes is that the self-consistent retrieval state stops existing, and the almost-correct first sweep begins a cascade of errors instead of a repair.

Capacity is a phase boundary: beyond \(\alpha_c\) the retrieval state ceases to exist, and memory collapses discontinuously.

4.7 Outlook: the partition function returns

In Week 2 we built a machine out of an Ising model, proved it computes (the Lyapunov theorem), taught it memories in one shot (Hebb), quantified interference (today), and found that the machine has a phase diagram in which memory is a phase and forgetting is a phase transition. All of it happened at zero temperature, with the partition function absent.

The step to Week 3 is a single modification. Replace the deterministic update of Equation 3.1 by a stochastic one — flip spin \(k\) with probability

\[ p(s_k \to -s_k) \;=\; \frac{1}{1 + e^{\beta\, \Delta E}}, \]

Glauber dynamics (Week 3)

which needs exactly the \(\Delta E = 2 s_k h_k\) we computed in the Lyapunov derivation, and which reduces to today’s greedy rule as \(\beta \to \infty\). Week 3 proves two statements about it. First, this dynamics has a stationary distribution, and it is \(p(s) \propto e^{-\beta E(s)}\): Week 1’s Boltzmann law returns, now over an energy we designed. Second, the moment we ask the network to learn its couplings from data by maximum likelihood — rather than have Hebb write them by fiat — the gradient comes out as \(\langle s_i s_j \rangle_\text{data} - \langle s_i s_j \rangle_\text{model}\), and the second average requires sampling from a distribution normalized by \(Z\). That machine is the Boltzmann machine, and the obstruction is Week 1’s partition function, intractable for all but the smallest networks.