Statistical inference §
Statistical inference is the reverse direction of probability: given observed data (realizations of random variables), it infers the underlying model, parameters, or latent states. It encompasses estimation, hypothesis testing, Bayesian updating, and causal inference. It is the bridge between probability theory and data science, engineering, and the sciences.
Brief history §
- Fisher (1920s): maximum likelihood, sufficiency, and the theory of estimation.
- Neyman and Pearson (1933): hypothesis testing and the power function.
- Bayesian revival (1950s–): conjugate priors, decision theory, and computation (MCMC).
- Modern era: high-dimensional inference, nonparametric methods, and causal inference.
Authoritative sources
Classes §
The four classes are organized by the inferential question they answer. Each targets a fundamentally different type of quantity: an associational parameter, a decision about a parameter set, a full posterior distribution, or an interventional effect.
| Page | Class | Question it answers | Key objects |
|---|---|---|---|
| Estimation | Estimation | What is the parameter value? | MLE, method of moments, Cramér–Rao bound, confidence intervals |
| Hypothesis testing | Hypothesis testing | Is the parameter in a specified set? | p-values, power, likelihood ratio tests, multiple comparisons |
| Bayesian inference | Bayesian inference | What is the full posterior distribution? | Priors, posteriors, MCMC, Bayes factors |
| Causal inference | Causal inference | What is the effect of an intervention? | Structural causal models, do-calculus, instrumental variables, counterfactuals |
Details
The four classes are organized by the inferential task, each targeting a distinct type of quantity:
- Estimation asks: what is the parameter? It produces a number (point estimate) or a range (confidence interval) with known coverage.
- Hypothesis testing asks: is the parameter in a certain set? It produces a rejection decision with controlled error rates.
- Bayesian inference asks: given the data, what is the posterior distribution of the parameter? It combines a prior with a likelihood via Bayes’ theorem (see Conditional probability).
- Causal inference asks: what would happen under intervention? It targets interventional and counterfactual quantities that are not identifiable from associational data alone without additional assumptions.
All four build on Random variables and expectation (the data are random variables) and Distributions (the sampling model). The asymptotic theory of estimators and tests relies on Limit theorems (CLT for the sampling distribution of the MLE).
Relationship to information theory. Estimation and model selection admit a unifying information-theoretic lens: the Kullback–Leibler divergence measures the loss from using an approximate model, mutual information quantifies the reduction in uncertainty about a parameter given the data, and minimum description length (MDL) provides a coding-theoretic criterion for model selection that parallels Bayesian model evidence. These connections make information theory a natural companion to statistical inference.