Hypothesis testing §

Every test trades off two kinds of mistakes, and the whole theory is about controlling them:

• Type I error (false positive): rejecting $H_0$ when it is true. Probability $\alpha = \sup_{\theta \in \Theta_0} E_\theta[\varphi]$ — the size of the test.
• Type II error (false negative): failing to reject $H_0$ when $H_1$ is true. Probability $\beta(\theta) = E_\theta[1 - \varphi]$ for $\theta \in \Theta_1$.
• Power: $1 - \beta(\theta)$ — the probability of correctly rejecting $H_0$ when $\theta \in \Theta_1$. It is a function of the true $\theta$, not a single number.

The asymmetry is deliberate: $H_0$ is the “status quo” hypothesis, and we require strong evidence to overturn it. This mirrors a criminal trial: $H_0$ = innocence, and a conviction requires proof beyond reasonable doubt (small $\alpha$), at the cost of letting some guilty defendants go free (large $\beta$).

Worked example. A factory tests whether a machine is calibrated ($H_0: \mu = 0$ mm offset) against $H_1: \mu \neq 0$. With $n = 25$ measurements of standard deviation $\sigma = 0.1$, the test rejects when $|\bar{X}| > 1.96 \times 0.1/5 = 0.0392$. Then $\alpha = 0.05$ exactly, and the power at $\mu = 0.05$ is $1 - \beta = P(|\bar{X}| > 0.0392 \mid \mu = 0.05) \approx 0.61$.

Example problem. In the factory test above, what is the power at $\mu = 0.02$?

Worked example. Let $X \sim N(\theta, 1)$ with $\theta_0 = 0$ and $\theta_1 = 1$. The likelihood ratio is

which is increasing in $x$. So the most powerful test rejects for large $x$: reject when $x > z_{1-\alpha}$. For $\alpha = 0.05$, reject when $x > 1.645$, and the power is $P_{\theta=1}(X > 1.645) = 1 - \Phi(0.645) \approx 0.26$. No other size-0.05 test can do better at $\theta = 1$.

The lemma generalizes to composite alternatives through the likelihood ratio test and to composite nulls by taking the supremum over $\Theta_0$. Every classical test (z-test, t-test, chi-squared test) is either a likelihood-ratio test or an approximation to one.

Example problem. For $X_1, \ldots, X_5 \sim \text{Exp}(\theta)$ with $\theta_0 = 1$ and $\theta_1 = 2$, what does the Neyman–Pearson most powerful test look like?

Authoritative sources

• J. Neyman & E. S. Pearson, On the Problem of the Most Efficient Tests of Statistical Hypotheses, Phil. Trans. R. Soc. A 231, 289–337 (1933)

A small p-value indicates that the observed data would be unlikely if $H_0$ were true. The p-value is not the probability that $H_0$ is true. It is a tail probability computed under the assumption that $H_0$ holds.

Common practice: reject $H_0$ if $p < \alpha$ (typically $\alpha = 0.05$). This is a decision rule, not a statement about the truth of $H_0$.

The figure below shows the anatomy of a one-sided test: the null distribution of the statistic, the rejection region (red, area $\alpha$), the critical value, the observed statistic, and the p-value (blue, the tail area beyond the observation).

Worked example. Testing $H_0: \mu = 0$ vs. $H_1: \mu \neq 0$ with $\bar{X} \sim N(\mu, \sigma^2/n)$. The test statistic is $Z = \bar{X}/(\sigma/\sqrt{n})$ and the two-sided p-value is $p = 2(1 - \Phi(|z|))$. With $n = 50$, $\sigma = 2$, $\bar{x} = 0.5$: $z = 0.5/(2/\sqrt{50}) = 1.77$, so $p = 2(1 - \Phi(1.77)) \approx 0.077$ — not significant at 5%, though a one-sided test would give $p = 0.038$ (see One-sided vs. two-sided tests).

Example problem. A test statistic $Z$ is standard normal under $H_0$ and the observed value is $z = 2.1$. What is the two-sided p-value, and would you reject at $\alpha = 0.05$?

When is each appropriate? A one-sided test is legitimate only when departures in the other direction would be practically meaningless or would lead to the same action (e.g., “is the new drug worse than placebo?” — if it is better, you would not switch). The direction must be chosen before seeing the data: choosing the side after observing the statistic doubles your effective Type I error rate, because a one-sided test at level $\alpha$ rejects half as often as a two-sided test at the same level.

How the rejection region shifts. For the same $\alpha = 0.05$ with a normal statistic:

• Two-sided: reject when $|z| > 1.96$ (two tails of 2.5% each).
• One-sided (upper): reject when $z > 1.645$ (one tail of 5%).

The one-sided critical value is smaller, so a one-sided test is more powerful in the declared direction — but completely blind to an effect in the opposite direction. In the worked example above, $z = 1.77$ is significant one-sided ($p = 0.038$) but not two-sided ($p = 0.077$).

Worked example. Testing whether a new bearing has lower friction than the old one ($H_0: \mu = \mu_0$ vs. $H_1: \mu < \mu_0$), with $z_{\text{obs}} = -1.8$: the one-sided (lower-tail) p-value is $p = \Phi(-1.8) \approx 0.036$, significant at 5%. The two-sided p-value would be $0.072$, not significant. The one-sided test is appropriate here because higher friction would simply mean “keep the old bearing” — the same action as no difference.

Example problem. You observe $z = 1.5$ in a test where $H_1: \mu > 0$ was pre-registered. Is the result significant at $\alpha = 0.05$? What if $H_1$ had been two-sided?

The likelihood ratio test (LRT) is a general-purpose test that works for composite hypotheses. It compares the best fit under the null to the best fit under the full model. It is the natural extension of the Neyman–Pearson lemma to composite hypotheses: when the alternative is composite, replacing $f_1$ by the supremum over $\Theta_1$ (equivalently over $\Theta$, since $\Lambda \leq 1$) yields a test that is asymptotically optimal.

Worked example. Testing $H_0: \mu = 0$ vs. $H_1: \mu \neq 0$ for $X_i \sim N(\mu, \sigma^2)$ with known $\sigma^2$. The LRT statistic is $-2\log\Lambda = n\bar{X}^2/\sigma^2 \sim \chi^2_1$ under $H_0$. With $n = 50$, $\sigma = 2$, $\bar{x} = 0.5$: $-2\log\Lambda = 50 \times 0.25/4 = 3.125$, compared to the $\chi^2_1$ critical value $3.84$ at $\alpha = 0.05$ — fail to reject, matching the two-sided z-test ($z = 1.77$, $z^2 = 3.125$).

The LRT is connected to Estimation: the numerator and denominator are both maximized likelihoods, i.e., MLEs under restricted and unrestricted parameter spaces.

Example problem. For $X_1, \ldots, X_n \sim \text{Exp}(\theta)$ (mean $1/\theta$), derive the LRT of $H_0: \theta = \theta_0$ vs. $H_1: \theta \neq \theta_0$ and give its large-sample form.

Worked example. With $\bar{x} = 0.5$, $\sigma = 2$, $n = 50$, the 95% confidence interval for $\mu$ is

The two-sided z-test of $H_0: \mu = 0$ does not reject ($0 \in (-0.054, 1.054)$), consistent with the p-value $0.077 > 0.05$ computed earlier. Testing $H_0: \mu = 1.2$ would reject, since $1.2$ is outside the interval — and indeed $z = (0.5 - 1.2)/0.283 = -2.47$, $p = 0.013 < 0.05$.

This duality means every confidence interval is a family of tests, and every test is a family of confidence sets. It also clarifies interpretation: “the 95% CI contains 0” and “the p-value exceeds 0.05” are the same statement, not two independent pieces of evidence.

Example problem. A 99% confidence interval for a proportion is $(0.31, 0.49)$. At which significance levels does the test of $H_0: p = 0.5$ reject?

The multiple comparisons problem arises in genomics (testing millions of genes), multiple regression, and any setting with many simultaneous tests. With $m$ independent tests each at level $\alpha$, the probability of at least one false positive is $1 - (1-\alpha)^m$, which is $0.40$ for $m = 10$ and $\alpha = 0.05$ — the Bonferroni correction restores the family-wise rate to $\alpha$ at the price of power.

The Benjamini–Hochberg procedure: order p-values $p_{(1)} \leq \cdots \leq p_{(m)}$, find the largest $k$ such that $p_{(k)} \leq k\alpha/m$, and reject all $H_{(1)}, \ldots, H_{(k)}$. It controls the false discovery rate $\text{FDR} = E[V/\max(R,1)]$ (false positives over rejections) rather than the probability of any false positive — a weaker but far more powerful guarantee, well suited to exploratory screening where some false leads are tolerable.

Worked example. Five p-values: 0.001, 0.008, 0.039, 0.041, 0.20, with $\alpha = 0.05$, $m = 5$. Compare thresholds $k\alpha/m = 0.01, 0.02, 0.03, 0.04, 0.05$. The largest $k$ with $p_{(k)} \leq k\alpha/m$ is $k = 2$ ($0.008 \leq 0.02$, but $0.039 > 0.03$). BH rejects the two smallest p-values. Bonferroni would reject only one ($0.001 < 0.01$).

Example problem. With $m = 100$ independent tests at $\alpha = 0.05$ and no correction, what is the probability of at least one Type I error? What does Bonferroni require?

Problem. A pharmaceutical company tests a new drug against a placebo. The response variable is $N(\mu, 4)$ (known variance). The null is $H_0: \mu = 0$ (no effect) vs. $H_1: \mu > 0$. A sample of $n = 100$ patients gives $\bar{x} = 0.6$. (a) Compute the z-statistic and p-value. (b) Is the result significant at $\alpha = 0.05$? (c) What is the power at $\mu = 1$?