Foreword

These notes are the written companion to a course taught at the blackboard. They exist for the reasons any script does: to correct the errors I will inevitably make while writing at speed, to carry the material for which ninety minutes leaves no room, to give the longer derivations the coherent form a lecture cannot, and to be usable in the week before the exam. They document one path through this material, not the completeness of a textbook, and where the path was a choice I have tried to say so.

The course they accompany is a graduate course in the physics department, and I have written for that reader: someone who has had a master’s course in statistical mechanics and is comfortable with ensembles, entropy, the free energy, and Gaussian integrals. In practice a course like this also draws bachelor students who are ahead of themselves, and people from the machine-learning side who arrive with the algorithms but without the thermodynamics. To the first group: the statistical mechanics is genuinely assumed, and Weeks 2 and 11 will be uncomfortable without it. To the second: the derivations here are self-contained, and what you lose by arriving without the companion course Machine Learning and Physics is the experience of having already trained the models we take apart. That is a real loss, since much of the pedagogical force of this course comes from explaining why something you have run actually works, but it is not a barrier to following the argument.

The books behind this one

Five books and one review shaped these notes, and it is worth saying what each contributes, since no single one of them covers the course.

Mehta and coauthors’ A high-bias, low-variance introduction to machine learning for physicists (Mehta et al. 2019) supplies the vocabulary the whole course speaks — the correspondence between energy and loss, between the partition function and the normalizer of a likelihood — and it is the reference text for the tutorials. MacKay’s Information Theory, Inference, and Learning Algorithms (MacKay 2003) is where I go for entropy and inference approached from the information-theoretic side, and its treatment of the relationship between coding, inference, and free energy is closer to this course’s spirit than any physics text. Sethna’s Statistical Mechanics (Sethna 2021) is the source of the habit of thinking in probabilities rather than in thermodynamic potentials, which is what makes the translation to machine learning read as translation and not as metaphor. Mézard and Montanari’s Information, Physics, and Computation (Mézard and Montanari 2009) carries the disordered-systems spine of Modules 1 and 2 — mean field, the cavity method, belief propagation — and no machine-learning text I know covers that material at all; the second module would not exist without it. Welling, Lu, and Holdijk’s Generative AI and Stochastic Thermodynamics (Welling, Lu, and Holdijk 2026) is the closest published match to the back half of the course, and I recommend it as parallel reading from Module 3 onward: it organizes the same territory around the free energy where I organize it around the partition function, which is the same choice made from the other side of \(F = -T \log Z\). Finally, Ulrich Schwarz’s Heidelberg script Theoretical Statistical Physics (Schwarz 2023) is the structural model for this document, and readers who want their statistical mechanics refreshed to the level assumed here should start with its early chapters.

Guiding principles

Four decisions shaped the course, and knowing them in advance will make the choices in the chapters that follow easier to read.

First, derivation rather than analogy. That statistical mechanics and machine learning resemble each other is remarked upon constantly and demonstrated rarely. This course makes a stronger and narrower claim: in a specific and enumerable set of cases the two fields are manipulating the same equation, and the identity can be proven. Where I can prove it, I do. Where the connection is genuinely a suggestive parallel rather than an identity, I say that instead.

Second, one object tracked through the whole semester. The partition function \(Z\) runs through all fourteen weeks. It is raised as the obstruction that blocks training a Boltzmann machine, approximated by variational and mean-field methods, sampled around by Monte Carlo and Langevin dynamics, and finally eliminated by score matching, where \(\nabla_x \log Z = 0\). A course organized as a tour of techniques would cover more ground; this organization gives the reader a thread to hold, and it means that Week 10 resolves a difficulty the reader met personally in Week 3.

Third, every method arrives as an explanation of something already run. This course is the structural-derivation complement to Machine Learning and Physics: that course teaches how to train a variational autoencoder, a restricted Boltzmann machine, a diffusion model; this one derives why those methods work. Week 12 accordingly opens on the mean-squared error you already minimized and takes it apart into objects the preceding three weeks built.

Fourth, complete derivations in writing and one derivation at the board. The script shows the algebra in full, with annotated multi-line displays. The lecture does one derivation properly — the engine of the day — and states the rest with a pointer to the chapter. The two documents are therefore not interchangeable, and the details live here.

The arc

We begin with the maximum-entropy argument, following Jaynes, and derive the Boltzmann distribution and its normalizer from it; the same lecture proves that the softmax layer terminating essentially every classifier trained today is that distribution, and the first week closes on the \(2^N\) wall that makes \(Z\) intractable. Module 1 then builds the energy-based view: Hopfield networks and the capacity transition, Boltzmann machines and the intractable gradient, contrastive divergence as the standard evasion, and modern Hopfield networks, whose update rule turns out to be transformer attention. Module 2 approximates what Module 1 could not compute, moving from mean-field theory and the variational free energy through the TAP equations and belief propagation to the Hubbard–Stratonovich transformation and the variational autoencoder. Module 3 stops approximating and starts sampling: Markov-chain Monte Carlo, Langevin dynamics, simulated annealing, and then the observation that stochastic gradient descent is itself a Langevin process, with the learning rate setting a temperature and flat minima favored for entropic reasons. Module 4 removes the obstruction rather than working around it, deriving score matching, the reverse-time stochastic differential equation, the Jarzynski and Crooks work relations and annealed importance sampling, and finally the denoising diffusion model, whose training loss we identify as a nonequilibrium free-energy estimator. Module 5 turns the machinery back on learning itself, through the information bottleneck and the phase transitions of inference, and the last week is a synthesis.

What is not here

A single semester leaves a great deal out, and the omissions below are deliberate.

I do not develop rigorous large-deviation theory, and I do not give the renormalization group its due; Week 5 computes mean-field exponents and states where they fail, but computing them correctly below the critical dimension is a different course. The replica method is used but not derived: Week 2 quotes the Amit–Gutfreund–Sompolinsky capacity \(\alpha_c \approx 0.138\) and states precisely what the replica calculation supplies that our tools cannot, which is the average of \(\log Z\) over quenched disorder. On the generative side, flow matching appears only as a pointer in Week 12, and Schrödinger bridges, the optimal-transport and JKO view of diffusion, and the thermodynamics of computation are absent entirely; for all of these the Welling, Lu, and Holdijk text is the natural continuation, and I have said so where each would have gone. The hardware and systems side of machine learning appears nowhere. Neither does any serious treatment of transformers beyond the attention identity of Week 4, which is a statement about one architecture’s update rule and not a theory of language models.

The tutorial problems are not reproduced here. They are revealed and worked in the room, a decision about the tutorials and not about these notes, whose reasoning is given in the course description.

The tutorials are run by Sander Hummerich and Gerrit Gerhartz. Errors that survive in these notes are mine, and I would be glad to hear about them.

Heidelberg, Winter Semester 2026/27

Tristan Bereau

MacKay, David J. C. 2003. Information Theory, Inference, and Learning Algorithms. Cambridge: Cambridge University Press.

Mehta, Pankaj, Marin Bukov, Ching-Hao Wang, Alexandre G. R. Day, Clint Richardson, Charles K. Fisher, and David J. Schwab. 2019. “A High-Bias, Low-Variance Introduction to Machine Learning for Physicists.” Physics Reports 810: 1–124. https://doi.org/10.1016/j.physrep.2019.03.001.

Mézard, Marc, and Andrea Montanari. 2009. Information, Physics, and Computation. Oxford: Oxford University Press.

Schwarz, Ulrich S. 2023. “Theoretical Statistical Physics.” Lecture-notes script, Heidelberg University.

Sethna, James P. 2021. Statistical Mechanics: Entropy, Order Parameters, and Complexity. 2nd ed. Oxford: Oxford University Press.

Welling, Max, Sirui Lu, and Lars Holdijk. 2026. Generative AI and Stochastic Thermodynamics: A Tale of Free Energies. Cambridge: Cambridge University Press. https://doi.org/10.1017/9781009709071.