Bayes Theorem

Introduction

Bayes' Theorem, named after 18th-century English statistician and philosopher Thomas Bayes, provides a mathematical framework for revising beliefs in light of new evidence. Unlike classical frequentist probability, which treats probability as the long-run frequency of events, Bayesian probability interprets probability as a degree of belief. This paradigm shift has made Bayes' Theorem indispensable in fields ranging from machine learning and artificial intelligence to medical diagnostics and legal reasoning.

At its core, the theorem answers a simple but profound question: If I observe an outcome, how should I adjust my confidence in a particular hypothesis? The answer depends on three factors: the initial belief in the hypothesis (the prior), the likelihood of the observed outcome if the hypothesis is true, and the overall probability of observing that outcome regardless of the hypothesis.

Mathematical Formulation

For two events \(A\) and \(B\), where \(P(B) > 0\), Bayes' Theorem states:

P(A | B) = [P(B | A) · P(A)] / P(B)

P(A | B) Posterior probability

P(B | A) Likelihood

P(A) Prior probability

P(B) Marginal likelihood

In words, the probability of hypothesis \(A\) given evidence \(B\) equals the probability of evidence \(B\) given hypothesis \(A\), multiplied by the prior probability of \(A\), all divided by the total probability of observing \(B\>.

Derivation

Bayes' Theorem follows directly from the definition of conditional probability. By definition:

\[ P(A | B) = \frac{P(A \cap B)}{P(B)} \]

\[ P(B | A) = \frac{P(A \cap B)}{P(A)} \]

Since \( P(A \cap B) = P(B \cap A) \), we can equate the numerators and solve for \( P(A | B) \):

\[ P(A | B) = \frac{P(B | A) \cdot P(A)}{P(B)} \]

When multiple mutually exclusive hypotheses \(A_1, A_2, ..., A_n\) exist, the denominator \(P(B)\) is expanded using the Law of Total Probability:

P(B) = Σ P(B | Aᵢ) · P(Aᵢ)

Step-by-Step Example

🧪 Medical Diagnostic Testing

A disease affects 1% of the population (\(P(D) = 0.01\)). A test for the disease is 95% accurate for those who have it (\(P(T+|D) = 0.95\)) and has a 5% false positive rate for healthy individuals (\(P(T+|\neg D) = 0.05\)). If a person tests positive, what is the probability they actually have the disease?

Identify priors: \(P(D) = 0.01\), \(P(\neg D) = 0.99\)
Identify likelihoods: \(P(T+|D) = 0.95\), \(P(T+|\neg D) = 0.05\)
Calculate marginal probability: \(P(T+) = (0.95 × 0.01) + (0.05 × 0.99) = 0.0095 + 0.0495 = 0.059\)
Apply Bayes' Theorem: \(P(D|T+) = 0.0095 / 0.059 ≈ 0.161\)

Result: Only ~16.1% of positive test results are true positives. This counterintuitive result highlights the critical importance of base rates.

Applications

Machine Learning: Naïve Bayes classifiers for spam detection, sentiment analysis, and document classification.
Medical Diagnostics: Interpreting test results, updating disease risk assessments, and personalized treatment planning.
Legal Reasoning: Evaluating forensic evidence and updating juror beliefs based on new testimony.
Scientific Inference: Bayesian statistics for experimental design, A/B testing, and parameter estimation.
Finance & Risk: Credit scoring, fraud detection, and dynamic portfolio optimization.

Common Misconceptions

1. Confusing P(A|B) with P(B|A) — The most frequent error. Just because a test is highly accurate doesn't mean a positive result implies a high probability of the condition. Base rates matter.

2. Ignoring the prior — Dismissing prior evidence as "subjective" overlooks that all scientific inference starts with existing knowledge. Priors can be updated iteratively.

3. Assuming independence where none exists — Naïve Bayes classifiers assume feature independence, which is rarely true in reality. Despite this, they often perform surprisingly well due to robustness to dependency violations.

Bayes' Theorem