Estimation §

The MLE has several key properties:

• Consistency: $\hat{\theta}_{\text{MLE}} \xrightarrow{P} \theta_0$ as $n \to \infty$ under regularity conditions.
• Asymptotic normality: $\sqrt{n}(\hat{\theta}_{\text{MLE}} - \theta_0) \xrightarrow{d} N(0, I(\theta_0)^{-1})$ where $I(\theta)$ is the Fisher information. This result rests on the Limit theorems (in particular the CLT applied to the score).
• Efficiency: the MLE achieves the Cramér–Rao lower bound asymptotically.
• Invariance: if $\hat{\theta}$ is the MLE of $\theta$, then $g(\hat{\theta})$ is the MLE of $g(\theta)$.

The score function is $S(\theta) = \partial \ell / \partial \theta$. Under regularity conditions its expectation is zero, $E[S(\theta)] = 0$, and its variance is the Fisher information: $I(\theta) = \text{Var}(S(\theta)) = -E[\partial^2 \ell / \partial \theta^2]$. The MLE is the value of $\theta$ at which the score vanishes, $S(\hat{\theta}) = 0$. The Fisher information thus measures the curvature of the log-likelihood and controls the asymptotic variance of the MLE.

Example. For $X_i \sim N(\mu, \sigma^2)$ with known $\sigma^2$, the MLE is $\hat{\mu} = \bar{X}$ and $\text{Var}(\hat{\mu}) = \sigma^2/n$.

The MLE is a special case of the more general framework of Distributions: the likelihood is the density of the data viewed as a function of the parameter.

Connection to Bayesian inference. The MLE coincides with the Bayesian MAP (maximum a posteriori) estimate when the prior is flat (uniform). See Bayesian inference for the full posterior-based approach.

The method of moments is simpler than MLE but generally less efficient. It works by matching the first $k$ sample moments to the first $k$ theoretical moments, where $k$ is the number of parameters.

Example. For $X_i \sim \text{Gamma}(\alpha, \beta)$ (shape–rate), set $\bar{X} = \alpha/\beta$ and $s^2 = \alpha/\beta^2$ to get $\hat{\alpha} = \bar{X}^2/s^2$ and $\hat{\beta} = \bar{X}/s^2$.

The Cramér–Rao bound gives a lower bound on the variance of any unbiased estimator. An estimator that achieves the bound is called efficient. The MLE is asymptotically efficient: its variance approaches the bound as $n \to \infty$.

The bound can be violated for finite samples (e.g., the sample mean for a uniform distribution), and it does not apply to biased estimators without modification.

Example. For $X_i \sim N(\mu, \sigma^2)$ with known $\sigma^2$, $I(\mu) = 1/\sigma^2$ and the bound is $\sigma^2/n$, which is achieved by $\bar{X}$.

The decomposition shows that an estimator’s total error has two independent sources: systematic error (bias) and random fluctuation (variance). The Cramér–Rao bound constrains only the variance of unbiased estimators; a biased estimator can have lower MSE by trading some bias for reduced variance.

Example. For $X_i \sim N(\mu, \sigma^2)$ with known $\sigma^2$, the sample mean $\bar{X}$ is unbiased and achieves the CRLB. The estimator $c\,\bar{X}$ with $0 < c < 1$ is biased (bias $= (1-c)\mu$), but its MSE is $(1-c)^2\mu^2 + c^2\sigma^2/n$, which can be smaller than $\sigma^2/n$ for large $|\mu|/(\sigma/\sqrt{n})$.

This tradeoff is the key lens for comparing estimators beyond the Cramér–Rao bound and connects directly to the bias–variance decomposition in Bayesian inference (posterior mean vs. MAP).

Fisher’s factorization theorem. $T(X)$ is sufficient for $\theta$ if and only if the likelihood factors as

for some functions $g$ and $h$ (where $h$ does not depend on $\theta$).

Rao–Blackwell theorem. If $T$ is any unbiased estimator of $g(\theta)$ and $S$ is a sufficient statistic, then $T^* = E[T \mid S]$ is also unbiased and has variance no greater than that of $T$. Conditioning on a sufficient statistic can only reduce variance.

Example. For $X_i \sim N(\mu, \sigma^2)$ with both parameters unknown, $(\bar{X}, \sum_i (X_i - \bar{X})^2)$ is sufficient. For $X_i \sim \text{Bernoulli}(p)$ the sufficient statistic is $\sum_i X_i$.

Sufficiency is the bridge between the raw data and the minimal information needed for estimation; it underlies the construction of UMVU (uniformly minimum-variance unbiased) estimators.

Construction methods:

• Pivotal quantity: find a statistic $Q(\hat{\theta}, \theta)$ whose distribution does not depend on $\theta$ (a pivot). Invert the pivot’s quantiles.
• Asymptotic (Wald): use $\hat{\theta} \pm z_{\alpha/2} \cdot \text{SE}(\hat{\theta})$.
• Likelihood ratio: $\{\theta : 2[\ell(\hat{\theta}) - \ell(\theta)] \leq \chi^2_{1,1-\alpha}\}$.

The coverage probability is a property of the procedure, not of the realized interval. A 95% CI means that 95% of intervals generated by repeated sampling contain the true parameter.

Example. For $\bar{X} \sim N(\mu, \sigma^2/n)$ with known $\sigma$, the pivot is $Z = (\bar{X} - \mu)/(\sigma/\sqrt{n}) \sim N(0,1)$ and the 95% CI is $\bar{X} \pm 1.96\,\sigma/\sqrt{n}$.

(a) The MLE is $\hat{\mu} = \bar{x} = 1.004$ cm.

(b) Since $\sigma = 0.01$ is known, the 95% CI is

so the interval is $[1.0001, 1.0079]$ cm.

(c) The Fisher information for one observation is $I(\mu) = 1/\sigma^2 = 10000$ and for $n = 25$ observations is $nI = 250000$. The Cramér–Rao bound is

The sample mean achieves this bound, confirming it is efficient.