Absolutely continuous distributions §

Step 1 — From mass to density. For an atomic distribution, probability lives on points: $P(X = x) = p(x)$. For an absolutely continuous distribution, every single point has probability zero, and probability lives on intervals: it is the area under the density curve. The density $f$ is a “probability per unit length”: $f(x)\,\Delta x$ is (approximately) the probability of falling in a small interval of width $\Delta x$ around $x$.

Step 2 — The value $f(x)$ is not a probability. Since $P(X = x) = \int_{\{x\}} f = 0$, the density value at a point can be larger than 1 (the uniform density on $[0, 1/2]$ equals 2). What matters is the integral: $P(a < X \leq b) = F(b) - F(a)$. This is why the CDF, not the density, is the universal object — see singular and mixed distributions for laws that have no density at all.

Step 3 — CDF and density are two views of the same law. The CDF is the running integral of the density, and the density is the derivative of the CDF (wherever the derivative exists — “almost everywhere” is enough, since changing $f$ on a null set does not change any integral).

Step 4 — Uniqueness is only almost-everywhere. If $f$ and $g$ are both densities of the same distribution, then $f = g$ except on a set of Lebesgue measure zero. Densities are equivalence classes, not functions; any version works for computing probabilities.

Sources: Grinstead & Snell ch. 2; Papoulis & Pillai ch. 4.

Step 1 — Mass conservation. A small interval $[x, x + dx]$ of probability mass $f_X(x)\, dx$ is carried by $g$ onto an interval of length $|g'(x)|\, dx$ around $y = g(x)$. The same mass, spread over a longer interval, means a smaller density: $f_Y(y) = f_X(x) / |g'(x)|$. The density is stretched wherever $g$ stretches, and compressed wherever $g$ compresses.

Step 2 — Worked example (location-scale). Let $X \sim \mathcal{N}(0,1)$ and $Y = 2X + 1$. Then $g^{-1}(y) = (y-1)/2$ and $|(g^{-1})'(y)| = 1/2$, so

which is exactly $\mathcal{N}(1, 4)$: the location-scale operation of Distributional structure and operations is a change of variables.

Step 3 — Non-injective maps: sum over preimages. If several points map to the same $y$, each contributes its own mass. For $Y = X^2$ with $X \sim \mathcal{N}(0,1)$, the preimages of $y > 0$ are $\pm\sqrt{y}$ and $|d(x^2)/dx| = 2|x| = 2\sqrt{y}$, giving

The simulation below lets you explore both ideas interactively: the normal density with adjustable $\mu, \sigma$, and the pushforward of a standard normal through $Y = X^2$.

Sources: Papoulis & Pillai ch. 5; Grinstead & Snell ch. 2, exercise on functions of random variables.

Step 1 — Marginals are shadows. Integrating the joint density over all other coordinates projects the law onto one axis, exactly as a shadow projects a 3D object onto a plane. Unlike the joint, marginals do not determine the joint: many different dependence structures share the same marginals (this is the subject of the copula section below).

Step 2 — Conditional densities slice the joint. Fix $x_{-j}$ and renormalize the slice of the joint density at that point so it integrates to 1: divide by the marginal $f_{-j}(x_{-j})$, which is exactly the mass of the slice. Under independence every slice has the same shape $f_j$, so conditioning changes nothing.

Step 3 — Worked example. Let $f(x, y) = x + y$ on the unit square $[0,1]^2$ (it integrates to 1). The marginals are $f_X(x) = x + \tfrac{1}{2}$ and $f_Y(y) = y + \tfrac{1}{2}$ — not proportional to the joint, so $X$ and $Y$ are dependent. The conditional density of $Y$ given $X = x$ is

and the conditional mean is $E[Y \mid X = x] = \frac{x/2 + 1/3}{x + 1/2}$, which increases in $x$: larger $X$ shifts probability mass of $Y$ upward.

Sources: Papoulis & Pillai ch. 6; Grinstead & Snell ch. 2, joint distributions.

Step 1 — Why copulas exist. Apply the probability-integral transform $U_i = F_i(X_i)$: each $U_i$ is uniform on $[0,1]$ (see the example problem below). The joint law of $(U_1, \ldots, U_d)$ is the copula — it contains only the dependence, since the marginals are all uniform. Sklar’s theorem says this construction is reversible.

Step 2 — The catalog of standard copulas.

• Independent copula: $C(u_1, \ldots, u_d) = \prod_i u_i$. It is the copula of independent coordinates, whatever the marginals.
• Gaussian copula: $C(u) = \Phi_d\bigl(\Phi_1^{-1}(u_1), \ldots, \Phi_1^{-1}(u_d); \Sigma\bigr)$, built from a multivariate normal with correlation matrix $\Sigma$ pushed through the univariate normal quantile function $\Phi_1^{-1}$. It captures dependence by correlation but has tail independence: extreme events decouple.
• Clayton copula (bivariate): $C(u, v) = (u^{-\theta} + v^{-\theta} - 1)^{-1/\theta}$ for $\theta > 0$. It has lower tail dependence: joint crashes are more likely than a Gaussian copula would predict — one reason copulas are used in credit-risk modeling.

Step 3 — Worked example. Take exponential marginals $F_1, F_2$ (rates 1 and 2) joined by the Clayton copula with $\theta = 1$. Sklar gives the joint CDF $F(x,y) = \bigl(F_1(x)^{-1} + F_2(y)^{-1} - 1\bigr)^{-1}$. The marginals are still exponential — the copula changed only the dependence. This modularity (marginals ⊕ copula = joint) is why copulas are the standard tool for building multivariate models with prescribed marginals.

Sources: Sklar (1959); R. B. Nelsen, An Introduction to Copulas (2006).

Step 1 — Read the shapes. The plot below shows the standard member of each family: normal $\mathcal{N}(0,1)$, exponential $\mathrm{Exp}(1)$, gamma $\mathrm{Gamma}(2,1)$, beta $\mathrm{Beta}(2,2)$, uniform $\mathrm{Unif}(0,1)$. Note how the gamma with $\alpha = 2$ starts at zero (one factor of $x$), while the exponential starts at its maximum; the beta is confined to $(0,1)$.

Step 2 — The families interlock. With the rate parameterization used above (density $\propto x^{\alpha-1} e^{-\beta x}$):

• The exponential is $\mathrm{Gamma}(1, \lambda)$.
• The chi-squared with $k$ degrees of freedom is $\mathrm{Gamma}(k/2,\, 1/2)$ — the density $\frac{1}{2^{k/2}\Gamma(k/2)} x^{k/2-1} e^{-x/2}$ matches $\beta = 1/2$, not $\beta = 2$. (The scale is 2; the rate is 1/2.)
• $\mathrm{Beta}(1,1)$ is $\mathrm{Unif}(0,1)$.
• If $X \sim \mathrm{Gamma}(\alpha, 1)$ and $Y \sim \mathrm{Gamma}(\beta, 1)$ are independent, then $X/(X+Y) \sim \mathrm{Beta}(\alpha, \beta)$.

Step 3 — Shared algebra. All five belong to the exponential family, which gives them shared algebraic properties (sufficient statistics, conjugate priors). See Distributional structure and operations for closure properties (convolution, mixtures, order statistics) and Statistical inference for the role of conjugacy.

Sources: Grinstead & Snell ch. 2, 5; Wikipedia: List of probability distributions.

Step 1 — Why CDFs, not densities. Convergence of densities is too strong a requirement: an atomic distribution can be a limit of absolutely continuous ones (a narrow spike of width $1/n$ and height $n$ converges in distribution to a point mass), and densities may fail to converge even when distributions do. Convergence at continuity points of $F$ is exactly the right weakening.

Step 2 — When densities do converge, everything follows (Scheffé’s lemma). If $f_n \to f$ almost everywhere and each $f_n$ is a density with $\int f_n = \int f = 1$, then $\int |f_n - f| \to 0$, and consequently $\sup_x |F_n(x) - F(x)| \to 0$: convergence is uniform over all intervals at once. This is why pointwise density convergence is such a useful certificate.

Step 3 — The CLT as density convergence. If $X_1, X_2, \ldots$ are i.i.d. with mean 0 and variance 1, the density of the normalized sum $S_n = (X_1 + \cdots + X_n)/\sqrt{n}$ converges pointwise to the standard normal density $\phi$ (the local central limit theorem, under mild conditions). The picture below shows the same phenomenon in a related family: the Student-$t$ density with $\nu$ degrees of freedom converges to $\mathcal{N}(0,1)$ as $\nu \to \infty$ — heavy tails drain into the center.

Sources: Wikipedia: Convergence of random variables; Limit theorems.

The map $g(x) = x^2$ is not injective on $\mathbb{R}$. For $y > 0$, the preimages are $\pm\sqrt{y}$ with $|g'(x)| = 2|x| = 2\sqrt{y}$. Applying the transformation formula:

This is a $\mathrm{Gamma}(1/2,\, 1/2)$ distribution — the chi-squared distribution with 1 degree of freedom (rate $\beta = 1/2$, matching the density $\frac{\beta^\alpha}{\Gamma(\alpha)} y^{\alpha-1} e^{-\beta y}$ with $\alpha = 1/2$).

$E[Y] = E[X^2] = \mathrm{Var}(X) + (E[X])^2 = 1 + 0 = 1$, which agrees with the gamma mean $\alpha/\beta = (1/2)/(1/2) = 1$.