Bayesian Inference

Introduction

Bayesian inference is a statistical methodology that quantifies uncertainty by treating unknown parameters as random variables governed by probability distributions. Unlike frequentist approaches, which interpret probability as long-run frequencies of events, Bayesian inference adopts a subjective interpretation: probability represents a degree of belief that is updated as new evidence accumulates.

This paradigm, rooted in the work of Thomas Bayes and later formalized by Pierre-Simon Laplace, has experienced a renaissance in the 21st century. Driven by advances in computational power and algorithmic innovation, Bayesian methods now underpin everything from recommendation systems and medical diagnostics to robotic perception and climate modeling.

"Probability is not a property of the physical world, but a measure of human ignorance about it." — E.T. Jaynes, Probability Theory: The Logic of Science

Bayes' Theorem

The mathematical foundation of Bayesian inference is Bayes' theorem, which relates the conditional and marginal probabilities of random variables. For a hypothesis \( H \) and observed data \( D \), the theorem states:

\( P(H \mid D) = \frac{P(D \mid H) \cdot P(H)}{P(D)} \) Equation 1: Bayes' Theorem

The elegance of this equation lies in its reversibility: it allows us to compute the probability of a hypothesis given the data, from quantities that are often easier to specify—namely, the probability of the data given the hypothesis, and the prior probability of the hypothesis.

Prior, Likelihood & Posterior

Bayesian analysis decomposes inference into three fundamental components:

Prior Distribution \( P(\theta) \): Represents existing knowledge or assumptions about the parameter \( \theta \) before observing data. Priors can be informative (encoding specific domain knowledge) or non-informative (designed to exert minimal influence).
Likelihood \( P(D \mid \theta) \): The probability of observing the data given a specific parameter value. It is typically derived from the data-generating process (e.g., Bernoulli for coin flips, Poisson for counting events).
Posterior Distribution \( P(\theta \mid D) \): The updated belief about \( \theta \) after incorporating the data. It combines prior beliefs with empirical evidence through Bayes' rule.

The marginal likelihood \( P(D) \), also called the evidence, acts as a normalizing constant ensuring the posterior integrates to one. It is computed by integrating the likelihood over all possible parameter values: \( P(D) = \int P(D \mid \theta)P(\theta)d\theta \).

Conjugate Priors

A conjugate prior is a family of distributions that, when combined with a given likelihood, yields a posterior from the same family. This property enables closed-form analytical solutions, bypassing the need for numerical integration.

[Visual: Prior → Likelihood → Posterior Evolution]

Figure 1: Iterative updating of belief distributions as sequential data is observed.

Common conjugate pairs include:

Beta-Binomial: Beta prior for a Bernoulli/binomial likelihood (e.g., estimating coin bias).
Normal-Normal: Normal prior for a Normal likelihood with known variance (e.g., estimating a population mean).
Gamma-Poisson: Gamma prior for a Poisson likelihood (e.g., modeling event rates).
Dirichlet-Multinomial: Dirichlet prior for categorical outcomes (e.g., topic modeling).

While conjugacy simplifies computation, modern Bayesian practice often favors flexible, non-conjugate priors paired with computational sampling techniques.

Computational Methods

When analytical solutions are intractable, Bayesian inference relies on numerical approximation. Two dominant paradigms have emerged:

Markov Chain Monte Carlo (MCMC)

MCMC algorithms construct a Markov chain whose stationary distribution matches the target posterior. By simulating dependent samples, they approximate integrals, expectations, and credible intervals. Notable methods include:

Metropolis-Hastings: Proposes new states and accepts/rejects based on a ratio test.
Hamiltonian Monte Carlo (HMC): Uses gradient information to navigate high-dimensional parameter spaces efficiently (implemented in Stan and PyMC).

Variational Inference (VI)

VI reframes inference as an optimization problem. It approximates the true posterior \( P(\theta \mid D) \) with a simpler distribution \( Q(\theta) \) by minimizing the Kullback-Leibler divergence \( \text{KL}(Q \| P) \). VI is computationally faster than MCMC but may underestimate posterior uncertainty, particularly in multimodal distributions.

Applications

Bayesian inference has permeated nearly every quantitative discipline:

Machine Learning: Bayesian neural networks, Gaussian processes, and hierarchical models provide uncertainty quantification crucial for safety-critical systems.
Medical Diagnosis: Updating disease probability based on sequential test results, incorporating base rates and test sensitivity/specificity.
Econometrics & A/B Testing: Bayesian experimental design enables continuous monitoring, early stopping, and direct probability statements about treatment effects.
Ecology & Conservation: Hierarchical models account for detection bias, spatial autocorrelation, and multi-source data integration.
Philosophy of Science: Provides a formal framework for hypothesis testing, model comparison (via Bayes factors), and scientific belief updating.

Advantages & Limitations

Strengths: Bayesian methods naturally incorporate prior knowledge, handle small sample sizes gracefully, produce full posterior distributions (not just point estimates), and excel in sequential updating and hierarchical modeling.

Challenges: Computationally intensive for high-dimensional models, sensitive to prior specification (especially with limited data), and requires careful diagnostic validation (e.g., checking chain convergence, posterior predictive checks). The philosophical debate between subjective vs. objective probability interpretations also remains active among statisticians.

"Bayesian inference is not a single method, but a coherent framework for reasoning under uncertainty." — Andrew Gelman et al., Bayesian Data Analysis

References

Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., & Rubin, D. B. (2020). Bayesian Data Analysis (3rd ed.). CRC Press.
McElreath, R. (2020). Statistical Rethinking: A Bayesian Course with Examples in R and Stan. CRC Press.
Jaynes, E. T. (2003). Probability Theory: The Logic of Science. Cambridge University Press.
Blei, D. M., Kucukelbir, A., & McAuliffe, J. D. (2017). Variational Inference: A Review for Statisticians. Journal of the American Statistical Association, 112(518), 859-877.
Laplace, P.-S. (1812). Théorie analytique des probabilités. Vve Courcier.