Marginal Likelihood

The marginal likelihood, also known as the model evidence, integrated likelihood, or simply the evidence, is a fundamental quantity in Bayesian statistics and probabilistic modeling. It represents the probability of the observed data under a given statistical model, averaged over all possible values of the model's parameters according to their prior distribution.

Unlike the standard likelihood function, which conditions on specific parameter values, the marginal likelihood integrates out the parameters, thereby providing a direct measure of how well a model explains the data while inherently penalizing unnecessary complexity. This property makes it the cornerstone of Bayesian model comparison and selection.

Mathematical Definition

Let D denote the observed data, M represent a candidate statistical model, and θ be the parameter vector associated with M. The marginal likelihood is defined as:

P(D | M) = \int P(D | θ, M) \cdot P(θ | M) dθ Where P(D | θ, M) is the likelihood function and P(θ | M) is the prior distribution over the parameters.

In the discrete parameter case, the integral is replaced by a sum:

P(D | M) = \sum θ P(D | θ, M) \cdot P(θ | M)

By the law of total probability, this quantity marginalizes the joint distribution of data and parameters. It is strictly a function of the data and the model specification, not of any particular parameter instantiation.

Role in Bayesian Inference

The marginal likelihood plays several critical roles in Bayesian methodology:

Posterior Normalization

In Bayes' theorem, the marginal likelihood serves as the normalizing constant that ensures the posterior distribution integrates to one:

P(θ | D, M) = P(D | θ, M) \cdot P(θ | M) / P(D | M)

While often treated as a computational nuisance in posterior sampling (since it cancels out in ratios), it is indispensable for proper probabilistic normalization.

Model Comparison & Bayes Factors

When comparing two models M₁ and M₂, the ratio of their marginal likelihoods forms the Bayes factor:

BF₁₂ = P(D | M₁) / P(D | M₂) A BF₁₂ > 1 indicates evidence in favor of M₁. Values are typically interpreted using Jeffreys' scale (e.g., >3: substantial, >10: strong).

Unlike information criteria (AIC, BIC), the Bayes factor naturally accounts for model complexity through the prior, embodying an information-theoretic Occam's razor.

Why complexity is penalized: As a model's parameter space grows, the prior probability mass spreads thinner. Unless the likelihood concentrates sharply around the true data-generating process, the average evidence decreases. This automatic penalty prevents overfitting without ad-hoc regularization terms.

Computation & Challenges

Despite its theoretical elegance, computing the marginal likelihood is notoriously difficult in high-dimensional spaces. The integral is a high-dimensional expectation that lacks a closed-form solution for most practical models.

Common computational approaches include:

Laplace Approximation: Uses a Gaussian approximation around the posterior mode. Fast but inaccurate for skewed or multi-modal posteriors.
Thermodynamic Integration (TI): Computes evidence by integrating log-likelihood over a temperature schedule. Highly accurate but computationally expensive.
Bridge Sampling / Harmonic Mean: Ratio estimators using samples from the posterior. Harmonic mean is generally discouraged due to high variance.
Variational Evidence Lower Bound (ELBO): Used in variational inference. Provides a tractable lower bound rather than the exact evidence.
Particle Marginal Likelihood: Leverages Sequential Monte Carlo (SMC) methods to estimate evidence while tracking model complexity over time.

Recent advances in normalizing flows and diffusion-based samplers have improved evidence estimation, but it remains an active area of research in computational statistics.

Practical Applications

Hyperparameter Tuning: Optimizing the marginal likelihood with respect to hyperparameters is equivalent to empirical Bayes or Type-II maximum likelihood.
Model Selection: Choosing among competing architectures in machine learning, econometrics, and bioinformatics.
Evidence Accumulation: In sequential decision-making and hypothesis testing, updating odds via Bayes factors as new data arrives.
Genomic & Phylogenetic Inference: Comparing evolutionary models or regulatory network structures where parameter counts vary widely.

Key References

Kass, R. E., & Raftery, A. E. (1995). Bayes Factors. Journal of the American Statistical Association, 90(430), 773–795.
Jeffreys, H. (1961). Theory of Probability (3rd ed.). Oxford University Press.
Gelman, A., & Meng, X.-L. (1998). Simulating Normalizing Constants: From Importance Sampling to Bridge Sampling to Path Sampling. Statistical Science, 13(2), 163–185.
Liddle, A. R. (2007). Information Constraints and the Occam Factor. Monthly Notices of the Royal Astronomical Society, 377(3), 1288–1296.