From Boltzmann to Diffusion

Statistical Physics of Machine Learning

Author

Tristan Bereau

Published

September 17, 2026

Course information

Winter Semester 2026/27. Lecture period: October 12, 2026 to February 6, 2027 (no lectures December 21, 2026 to January 6, 2027)

Lectures: Mon & Wed 9-11; Tutorial: Wed 16-18

Lectures: Ph12 gHS; Tutorial: Ph12 106

Lecturer: Prof. Dr. Tristan Bereau, Institute for Theoretical Physics, Heidelberg University

Tutors: Sander Hummerich, Institute for Theoretical Physics & Interdisciplinary Center for Scientific Computing (IWR), Heidelberg University; Gerrit Gerhartz, Institute for Theoretical Physics, Heidelberg University

8 credit points

Course description

Modern generative machine learning is not merely inspired by statistical physics — it is, structurally, nineteenth- and twentieth-century statistical mechanics rediscovered. The Boltzmann distribution \(p \propto e^{-\beta E}\), the softmax layer \(p_i = e^{z_i}/\sum_j e^{z_j}\), and the score \(\nabla_x \log p\) that a diffusion model learns are the same equation in three costumes; the third is the first differentiated, which is exactly how the partition function disappears. This course traces that pedigree — the derivation, not the analogy.

A single object runs through the whole course: the partition function \(Z\), the normalizer of an energy-defined probability distribution. You can almost always write a model down; you can almost never normalize it. \(Z\) is raised as the obstruction that blocks training a Boltzmann machine (Module 1), approximated by variational and mean-field methods (Module 2), sampled around by Monte Carlo and Langevin dynamics (Module 3), and finally eliminated by score matching, where \(\nabla_x \log Z = 0\) (Module 4). Along the way, learning becomes the shaping of an energy landscape, training dynamics become a stochastic process with their own Fokker–Planck equation, and the generative process of a diffusion model is revealed as nonequilibrium thermodynamics run in reverse.

This course is the structural-derivation complement to the companion Machine Learning and Physics course: that course teaches the working methods (how to run DDPM, VAEs, MCMC); this one derives why they work from statistical mechanics.

Prerequisites

  • Required. Theoretical Statistical Physics or equivalent — ensembles, entropy, the free energy, the Boltzmann distribution, Gaussian integrals, Laplace’s method. Every module rests on it continuously.
  • Assumed familiarity. Machine Learning and Physics or equivalent. The course derives why methods you have already run work, and is written for readers who have trained a variational autoencoder and a diffusion model and run a Markov-chain sampler; Week 12 opens on the diffusion loss you minimized there. The derivations are self-contained regardless, so a reader arriving without that background meets these models here for the first time. Mehta et al. (2019) §I–II, the Week 1 tutorial reading, covers the machine-learning vocabulary the early lectures assume.

Assessment

  • Written final exam (100% of the grade). The exam is built directly from the tutorial problems and tests computational reasoning, not live coding. It is closed-book; where a result is needed that is not worth memorizing, the formula is provided on the exam paper.
  • Exam admission (ungraded Studienleistung, Zulassungsvoraussetzung). Submit the weekly stuck-point card for at least 8 of the 12 tutorial warm-ups (Weeks 2–13; the Week 1 tutorial has no warm-up).

Tutorials run every week (weeks 1–13, plus a week-14 review) and use a productive-failure, peer-instruction, just-in-time format: a light home warm-up whose stuck-point card feeds the in-class triage, then the hard problem is revealed and cracked in class — on paper first, and on the computation-led weeks with one laptop per group for a bounded stretch after the prediction has been written down. There is no separate graded problem-set track.

Course outline

The course spans 14 weeks and five modules, organized around the \(Z\) spine.

  • Week 1 — Introduction. Maximum entropy ⇒ the Boltzmann/Gibbs distribution ⇒ the partition function \(Z\); the softmax layer is the Boltzmann distribution; the \(2^N\) normalization wall.
  • Module 1 (Weeks 2–4) — The energy-based view. Hopfield networks → Boltzmann machines / RBMs and contrastive divergence → modern energy-based models → modern Hopfield networks ≡ attention. \(Z\) is raised as the training obstruction.
  • Module 2 (Weeks 5–7) — Variational & mean-field methods. Mean-field theory and the variational free energy → TAP / belief propagation / approximate message passing → Hubbard–Stratonovich transformations and variational autoencoders. \(Z\) is approximated.
  • Module 3 (Weeks 8–9) — Sampling & the Fokker–Planck of training. MCMC and Langevin dynamics, simulated annealing → SGD-as-Langevin, flat minima, and entropy-SGD. \(Z\) is sampled around.
  • Module 4 (Weeks 10–12) — Nonequilibrium thermodynamics of generative modeling. Score matching → Jarzynski / Crooks work relations and annealed importance sampling → DDPM and score-based SDEs. \(Z\) is eliminated (\(\nabla_x \log Z = 0\)).
  • Module 5 (Weeks 13–14) — Information, free energy & bridges. The information bottleneck → phase transitions in inference → synthesis and exam preparation. \(Z\) becomes the object of study rather than the obstruction.

Main references

  • Mehta, P., Bukov, M., Wang, C.-H., Day, A. G. R., Richardson, C., Fisher, C. K., & Schwab, D. J. (2019). A high-bias, low-variance introduction to machine learning for physicists. Physics Reports, 810, 1–124. https://doi.org/10.1016/j.physrep.2019.03.001
  • MacKay, D. J. C. (2003). Information Theory, Inference, and Learning Algorithms. Cambridge University Press. https://www.inference.org.uk/mackay/itila/
  • Sethna, J. P. (2021). Statistical Mechanics: Entropy, Order Parameters, and Complexity (2nd ed.). Oxford University Press.
  • Welling, M., Lu, S., & Holdijk, L. (2026). Generative AI and Stochastic Thermodynamics: A Tale of Free Energies. Cambridge University Press (forthcoming). https://www.cambridge.org/9781009709064 — companion reading for the back half of the course (Modules 3–5): the free-energy view of generative modeling and stochastic thermodynamics.
  • Jaynes, E. T. (1957). Information theory and statistical mechanics. Physical Review, 106(4), 620–630. https://doi.org/10.1103/PhysRev.106.620
  • Hopfield, J. J. (1982). Neural networks and physical systems with emergent collective computational abilities. PNAS, 79(8), 2554–2558. https://doi.org/10.1073/pnas.79.8.2554
  • Hyvärinen, A. (2005). Estimation of non-normalized statistical models by score matching. JMLR, 6, 695–709.
  • Ho, J., Jain, A., & Abbeel, P. (2020). Denoising diffusion probabilistic models. NeurIPS 33. https://arxiv.org/abs/2006.11239
  • Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., & Poole, B. (2021). Score-based generative modeling through stochastic differential equations. ICLR. https://arxiv.org/abs/2011.13456

Acknowledgments

These lecture notes, exercises, and supporting code were prepared with the assistance of Claude (Anthropic). All material has been reviewed, edited, and validated by the instructor.