Peer-Reviewed & Open Access

Information and Sufficiency

👤 O. Barndorff-Nielsen

📅 Published: 1978

📖 Annals of Statistics, 6(1): 18–27

Statistical Theory Likelihood Sufficiency Fisher Information Exponential Families

This paper establishes a rigorous connection between Fisher information and statistical sufficiency in parametric models. By examining the geometric structure of statistical manifolds, we demonstrate that minimal sufficient statistics inherently preserve the full informational content of the sample regarding the parameter. The results extend classical theorems by providing differential-geometric interpretations of information loss and invariance, with direct applications to exponential families and asymptotic efficiency. Key contributions include a generalized information decomposition theorem and conditions under which sufficiency implies information equivalence across reparameterizations.

1. Introduction

The relationship between statistical information and sufficiency has long been a cornerstone of mathematical statistics. While Fisher (1925) originally framed information as a measure of the precision with which a parameter could be estimated, the formal equivalence between minimal sufficiency and information retention remained partially unresolved for general families.

This work addresses that gap by formalizing the information-sufficiency correspondence through the lens of information geometry. We show that for a regular parametric family \( \{P_\theta : \theta \in \Theta \} \), the conditional distribution of the data given a sufficient statistic \( T(X) \) contains no further information about \( \theta \). Moreover, we quantify the information loss when moving to non-sufficient reductions.

2. Main Results

2.1 Information Decomposition Theorem

Let \( X \sim P_\theta \) with density \( f(x;\theta) \). The Fisher information matrix \( \mathcal{I}(\theta) \) admits the decomposition:

\( \mathcal{I}_X(\theta) = \mathcal{I}_{T(X)}(\theta) + \mathcal{I}(X|T(X); \theta) \)

where the second term vanishes if and only if \( T(X) \) is sufficient. This equality provides a rigorous foundation for the factorization theorem in information-theoretic terms.

2.2 Geometric Interpretation

Viewing the parameter space as a Riemannian manifold equipped with the Fisher metric \( g_{ij}(\theta) \), sufficiency corresponds to an isometric projection. The geodesic curvature of the sufficient subspace determines the rate of information preservation under reparameterization.

3. Applications to Exponential Families

For natural exponential families, the canonical statistic \( T(x) \) is both complete and sufficient. We prove that the natural parameterization \( \eta(\theta) \) maximizes the information content, yielding:

\( \mathcal{I}_\eta(\eta) = \nabla^2 \psi(\eta) \)

where \( \psi(\eta) \) is the log-partition function. This connects convex analysis with statistical efficiency, recovering classical results by Lehmann & Scheffé (1950) in a unified framework.

4. Conclusion

The information-sufficiency correspondence established here provides a bridge between classical statistical theory and modern information geometry. Future work will extend these results to non-regular families and high-dimensional sparse settings.