This paper takes the reader on a journey through the history of Bayesian computation, from the 18th century to the present day. Beginning with the one-dimensional integral first confronted by Bayes in 1763, we highlight the key contributions of: Laplace, Metropolis (and, importantly, his coauthors), Hammersley and Handscomb, and Hastings, all of which set the foundations for the computational revolution in the late 20th century—led, primarily, by Markov chain Monte Carlo (MCMC) algorithms. A very short outline of 21st century computational methods—including pseudo-marginal MCMC, Hamiltonian Monte Carlo, sequential Monte Carlo and the various “approximate” methods—completes the paper.

The 21st century has seen an enormous growth in the development and use of approximate Bayesian methods. Such methods produce computational solutions to certain “intractable” statistical problems that challenge exact methods like Markov chain Monte Carlo: for instance, models with unavailable likelihoods, high-dimensional models and models featuring large data sets. These approximate methods are the subject of this review. The aim is to help new researchers in particular—and more generally those interested in adopting a Bayesian approach to empirical work—distinguish between different approximate techniques, understand the sense in which they are approximate, appreciate when and why particular methods are useful and see the ways in which they can can be combined.

Software tools for Bayesian inference have undergone rapid evolution in the past three decades, following popularisation of the first generation MCMC-sampler implementations. More recently, exponential growth in the number of users has been stimulated both by the active development of new packages by the machine learning community and popularity of specialist software for particular applications. This review aims to summarize the most popular software and provide a useful map for a reader to navigate the world of Bayesian computation. We anticipate a vigorous continued development of algorithms and corresponding software in multiple research fields, such as probabilistic programming, likelihood-free inference and Bayesian neural networks, which will further broaden the possibilities for employing the Bayesian paradigm in exciting applications.

Bayesian models are powerful tools for studying complex data, allowing the analyst to encode rich hierarchical dependencies and leverage prior information. Most importantly, they facilitate a complete characterization of uncertainty through the posterior distribution. Practical posterior computation is commonly performed via MCMC, which can be computationally infeasible for high-dimensional models with many observations. In this article, we discuss the potential to improve posterior computation using ideas from machine learning. Concrete directions are explored in vignettes on normalizing flows, statistical properties of variational approximations, Bayesian coresets and distributed Bayesian inference.

Denoising diffusion models are a novel class of generative models that have recently become extremely popular in machine learning. In this paper, we describe how such ideas can also be used to sample from posterior distributions and, more generally, any target distribution whose density is known up to a normalizing constant. The key idea is to consider a forward “noising” diffusion initialized at the target distribution, which “transports” this latter to a normal distribution for long diffusion times. The time reversal of this process, the “denoising” diffusion, thus “transports” the normal distribution to the target distribution and can be approximated so as to sample from the target. To accelerate simulation, we show how one can introduce and approximate a Schrödinger bridge between these two distributions, that is, a diffusion which transports the normal to the target in finite time.

Bayesian experimental design (BED) provides a powerful and general framework for optimizing the design of experiments. However, its deployment often poses substantial computational challenges that can undermine its practical use. In this review, we outline how recent advances have transformed our ability to overcome these challenges and thus utilize BED effectively, before discussing some areas for future development in the field.

We identify recurrent ingredients in the antithetic sampling literature leading to a unified sampling framework. We introduce a new class of antithetic schemes that includes the most used antithetic proposals. This perspective enables the derivation of new properties of the sampling schemes: (i) optimality in the Kullback–Leibler sense; (ii) closed-form multivariate Kendall’s τ and Spearman’s ρ; (iii) ranking in concordance order and (iv) a central limit theorem that characterizes stochastic behaviour of Monte Carlo estimators when the sample size tends to infinity. The proposed simulation framework inherits the simplicity of the standard antithetic sampling method, requiring the definition of a set of reference points in the sampling space and the generation of uniform numbers on the segments joining the points. We provide applications to Monte Carlo integration and Markov Chain Monte Carlo Bayesian estimation.

We discuss several algorithms for sampling from unnormalized probability distributions in statistical physics, but using the language of statistics and machine learning. We provide a self-contained introduction to some key ideas and concepts of the field, before discussing three well-known problems: phase transitions in the Ising model, the melting transition on a two-dimensional plane and simulation of an all-atom model for liquid water. We review the classical Metropolis, Glauber and molecular dynamics sampling algorithms before discussing several more recent approaches, including cluster algorithms, novel variations of hybrid Monte Carlo and Langevin dynamics and piece-wise deterministic processes such as event chain Monte Carlo. We highlight cross-over with statistics and machine learning throughout and present some results on event chain Monte Carlo and sampling from the Ising model using tools from the statistics literature. We provide a simulation study on the Ising and XY models, with reproducible code freely available online, and following this we discuss several open areas for interaction between the disciplines that have not yet been explored and suggest avenues for doing so.

With increasing data availability, causal effects can be evaluated across different data sets, both randomized controlled trials (RCTs) and observational studies. RCTs isolate the effect of the treatment from that of unwanted (confounding) co-occurring effects but they may suffer from unrepresentativeness, and thus lack external validity. On the other hand, large observational samples are often more representative of the target population but can conflate confounding effects with the treatment of interest. In this paper, we review the growing literature on methods for causal inference on combined RCTs and observational studies, striving for the best of both worlds. We first discuss identification and estimation methods that improve generalizability of RCTs using the representativeness of observational data. Classical estimators include weighting, difference between conditional outcome models and doubly robust estimators. We then discuss methods that combine RCTs and observational data to either ensure unconfoundedness of the observational analysis or to improve (conditional) average treatment effect estimation. We also connect and contrast works developed in both the potential outcomes literature and the structural causal model literature. Finally, we compare the main methods using a simulation study and real world data to analyze the effect of tranexamic acid on the mortality rate in major trauma patients. A review of available codes and new implementations is also provided.

Stephen M. Stigler received his Ph.D. in Statistics from the University of California, Berkeley, with a dissertation on the asymptotic distribution of linear functions of order statistics. Starting in 1967, he taught at the University of Wisconsin, Madison, then in 1979 moved to the University of Chicago where he taught from 1979 to 2021. Stigler has worked on a variety of topics in mathematical statistics, ranging from asymptotic theory to the theory of experimental design, and on applications of statistics including in anthropology, forensic science, paleontology, psychology, information transfer and sports. In recent years, he has concentrated on the history of statistics, with inquiries ranging from the development of statistical methods in astronomy and geodesy and their spread to biological and social sciences, to lotteries, to the modern development of statistical theory. He has published four books, The History of Statistics (1986), Statistics on the Table (1999), The Seven Pillars of Statistical Wisdom (2016) and Casanova’s Lottery (2022). A recent research focus has been upon the way the work of Francis Galton on the statistics of inheritance led to the creation of modern multivariate analysis and made a true Bayesian inference possible, and on how R. A. Fisher’s transformation of Karl Pearson’s path breaking research led to a modern period of statistical enlightenment.

Stigler is an elected member of the American Academy of Arts and Sciences and of the American Philosophical Society; he has served as President of the Institute of Mathematical Statistics and of the International Statistical Institute. In 2005, he received the Humboldt Foundation Research Award; in 2010, he was elected Membre Associé of the Académie royale de Belgique, Classe des Sciences. Stigler served as Theory and Methods Editor for the Journal of the American Statistical Association 1979–1982. He was a Guggenheim Fellow in 1977, and received awards for undergraduate teaching at the University of Wisconsin (1971) and University of Chicago (1998).

This interview with Stigler was conducted remotely in July 2021.

Sir David Spiegelhalter and Professor Sylvia Richardson are two eminent statisticians of our time who have made pioneering contributions to Statistics and Data Science with careers spanning over more than four decades. They have a long and celebrated legacy built through foundational research in Bayesian statistics, impactful collaborations, steadfast professional service and superb scientific communications. They have won many prestigious awards and recognitions throughout their distinguished careers. During my sabbatical in 2022 at the University of Cambridge I had the honor of sitting down for a conversation with these two remarkable individuals. We discussed early career influences and digressions, research philosophy, role of mentors and advice for the future generation. We hope this conversation with David and Sylvia will inspire many future statisticians.