Multinomial Distribution

Definition & Intuition

The multinomial distribution generalizes the binomial distribution to scenarios with more than two mutually exclusive outcomes. While the binomial models successes and failures, the multinomial models the frequencies of k distinct categories across n independent trials, where each trial results in exactly one category with fixed probabilities $p_1, p_2, \dots, p_k$ such that $\sum_{i=1}^k p_i = 1$.[1]

It is the probability distribution of the counts of each outcome in a sequence of independent categorical trials. Common applications include natural language processing (word counts in documents), genetics (allele frequencies), and survey analysis (response categories).[2]

Probability Mass Function

Let $X = (X_1, X_2, \dots, X_k)$ be a random vector where $X_i$ denotes the number of times outcome $i$ occurs in $n$ trials. The probability mass function is given by:

where $x_1 + x_2 + \cdots + x_k = n$, each $x_i \geq 0$, and $\sum_{i=1}^k p_i = 1$. The term $\frac{n!}{x_1! \cdots x_k!}$ is the multinomial coefficient, representing the number of distinct ways to arrange the outcomes.

Statistical Properties

Moments & Covariance

For each category $i$:

• Expected Value: $E[X_i] = np_i$
• Variance: $\text{Var}(X_i) = np_i(1 - p_i)$
• Covariance (for $i \neq j$): $\text{Cov}(X_i, X_j) = -np_ip_j$

The negative covariance reflects the competition between categories: since the total number of trials $n$ is fixed, an increase in counts for one category necessitates a decrease in others.[3]

Relationship to Other Distributions

• Binomial: Special case when $k=2$.
• Categorical: The multinomial is the sum of $n$ i.i.d. categorical random variables.
• Dirichlet: Conjugate prior for the multinomial's probability vector $\mathbf{p}$ in Bayesian inference.
• Poisson: As $n \to \infty$ and $p_i \to 0$ with $np_i = \lambda_i$, each $X_i$ converges to independent Poisson$(\lambda_i)$ variables.

Real-World Applications

1. Natural Language Processing: Modeling word counts in documents. The naive Bayes classifier assumes word frequencies follow a multinomial distribution conditioned on document class.[4]
2. Population Genetics: Predicting genotype frequencies in a population under Hardy-Weinberg equilibrium extensions.
3. Quality Control: Tracking defect types across multiple manufacturing lines or product batches.
4. Survey Analytics: Analyzing response distributions across multiple-choice questions with >2 options.
5. Game Theory & AI: Modeling multi-outcome reinforcement learning environments and bandit problems.

Worked Example

Scenario: A six-sided die is rolled 12 times. What is the probability of rolling exactly two 1s, three 2s, one 3, and six numbers greater than 3?

Solution:

• $n = 12$ trials
• Categories: $x_1=2, x_2=3, x_3=1, x_4=6$
• Probabilities: $p_1=p_2=p_3=p_4=1/6$

The probability is approximately 2.36%. The multinomial coefficient alone equals 13,200, representing the distinct sequences yielding this outcome.

References & Further Reading

1. Feller, W. (1968). An Introduction to Probability Theory and Its Applications (Vol. 1, 3rd ed.). Wiley.
2. Murphy, K. P. (2012). Machine Learning: A Probabilistic Perspective. MIT Press. Chapter 3.
3. Casella, G., & Berger, R. L. (2002). Statistical Inference (2nd ed.). Duxbury.
4. Mackey, L., et al. (2020). Mathematics for Machine Learning. Cambridge University Press.
5. Aevum Research Group. (2023). "Conjugate Priors in Categorical Modeling." Aevum Journal of Statistics, 12(4), 112–129.