Distributional structure and operations

This topic studies the algebraic and structural relationships between distributions: the fundamental operations that generate new distributions from old ones (convolution, mixture, transformation), the parametric families defined by their closure under those operations (exponential, location-scale, stable), and order statistics. It provides the unifying framework that explains why certain distributions recur across applications and how new distributions can be generated from old ones.

Authoritative sources

B. Efron, Exponential Families in Theory and Practice (Cambridge University Press)

R. D. Durrett, Probability: Theory and Examples (5th ed.)

The operations come first because the families are defined by how they behave under them. We build the page in that order: the two basic operations (convolution, mixture), how they interact, then the three families that are characterized by their closure properties, and finally order statistics.

Convolution

The convolution of two distributions $\mu$ and $\nu$ is the distribution of $X + Y$ where $X \sim \mu$ and $Y \sim \nu$ independently:

(\mu * \nu)(A) = \int \mu(A - y)\, \nu(dy).

For densities: $(f * g)(x) = \int f(x - y)\, g(y)\, dy$ .
For PMFs: $(p * q)(k) = \sum_j p(j)\, q(k - j)$ .
For PGFs: $G_{X+Y}(z) = G_X(z) \cdot G_Y(z)$ .
For characteristic functions: $\varphi_{X+Y}(t) = \varphi_X(t) \cdot \varphi_Y(t)$ .

Convolution is associative and commutative. The delta measure $\delta_0$ is the identity. The relationship to Atomic distributions: the PGF converts convolution to multiplication, making it the algebraic tool for discrete sums.

Details

Convolution is the distributional analogue of multiplication in the algebra of functions: it is the operation that answers “what is the law of a sum of independent quantities?” The defining integral has a concrete geometric reading — slide one density past the other, multiply pointwise, and integrate — which is the single idea behind every closure result below.

The sliding picture. Fix $x$ . The integrand $f(x-y)\,g(y)$ is the product of $g$ at $y$ and $f$ reflected and shifted to be centered at $x$ . As $x$ increases, that reflected copy of $f$ slides to the right, and the area under the product traces out the density of the sum.

Worked example (discrete). Let $X \sim \mathrm{Poisson}(\lambda_1)$ and $Y \sim \mathrm{Poisson}(\lambda_2)$ be independent. The PMF of the sum is

P(X+Y=k) = \sum_{j=0}^{k} \frac{e^{-\lambda_1}\lambda_1^j}{j!}\,\frac{e^{-\lambda_2}\lambda_2^{k-j}}{(k-j)!} = e^{-(\lambda_1+\lambda_2)} \frac{1}{k!}\sum_{j=0}^{k}\binom{k}{j}\lambda_1^j \lambda_2^{k-j} = \frac{e^{-(\lambda_1+\lambda_2)}(\lambda_1+\lambda_2)^k}{k!}.

The inner sum is the binomial expansion, so $X+Y \sim \mathrm{Poisson}(\lambda_1+\lambda_2)$ . Equivalently, in one line via PGFs: $G_{X+Y}(z)=e^{\lambda_1(z-1)}e^{\lambda_2(z-1)}=e^{(\lambda_1+\lambda_2)(z-1)}$ .

Closure results. Convolution is the operation that the families below are closed under:

The sum of independent normals is normal (closure of the normal family).
The sum of independent gammas with the same rate is gamma (closure of the gamma family).
The sum of independent Poissons is Poisson (closure of the Poisson family).
The sum of independent stable( $\alpha, \beta$ ) variables is stable( $\alpha, \beta'$ ) (closure of the stable family).

Note that closure is a property of specific families, not of the exponential family as a whole: the sum of i.i.d. $\mathrm{Exp}(\lambda)$ is $\mathrm{Gamma}(n,\lambda)$ , not exponential, and the sum of i.i.d. $\mathrm{Bernoulli}(p)$ is $\mathrm{Binomial}(n,p)$ , not Bernoulli.

The irreducibility of convolution: if $\mu * \nu = \delta_0$ , then $\mu = \nu = \delta_0$ . This means convolution has no nontrivial inverses, unlike multiplication of functions.

Authoritative sources

R. D. Durrett, Probability: Theory and Examples (5th ed.)

Example problem

Problem. If $X \sim \mathrm{Poisson}(3)$ and $Y \sim \mathrm{Poisson}(5)$ are independent, find the distribution of $X+Y$ and of $X-Y$ .

Solution

$X+Y \sim \mathrm{Poisson}(3+5) = \mathrm{Poisson}(8)$ by the closure result above.

For $X-Y$ , use characteristic functions: $\varphi_{X-Y}(t)=\varphi_X(t)\varphi_Y(-t)=e^{3(e^{it}-1)}e^{5(e^{-it}-1)}$ . This is the characteristic function of the skewed Poisson (a difference of independent Poissons), which is not a Poisson law — a reminder that convolution (sums) is closed for Poissons, but differences are not.

Mixtures

A mixture of distributions $\{\mu_i$ with weights $\{w_i$ (where $w_i \geq 0$ , $\sum_i w_i = 1$ ) is

\mu = \sum_i w_i \, \mu_i.

The CDF is $F(x) = \sum_i w_i F_i(x)$ .
If the $\mu_i$ are absolutely continuous, the mixture density is $f(x) = \sum_i w_i f_i(x)$ .
Mixtures are the natural model for latent class structures: the component index is a hidden variable.

Details

Mixtures differ from convolutions in the order of operations: a mixture is a weighted average of distributions (the component is chosen first, then a single sample is drawn from it), while a convolution is the distribution of a sum (both components contribute to the same sample). A mixture is a convex combination of measures; a convolution is a product in the characteristic-function algebra.

Worked example (Poisson–gamma → negative binomial). Let $N \mid \Lambda \sim \mathrm{Poisson}(\Lambda)$ and $\Lambda \sim \mathrm{Gamma}(r, \beta)$ (shape $r$ , rate $\beta$ ). Marginalizing over the latent rate:

P(N=k) = \int_0^\infty \frac{e^{-\lambda}\lambda^k}{k!}\,\frac{\beta^r}{\Gamma(r)}\lambda^{r-1}e^{-\beta\lambda}\,d\lambda = \frac{\Gamma(r+k)}{k!\,\Gamma(r)}\left(\frac{\beta}{\beta+1}\right)^r\left(\frac{1}{\beta+1}\right)^k,

which is the negative binomial PMF. The mixture “smears” the Poisson rate and produces the heavier-tailed negative binomial — a mixture is generally not in the same family as its components.

Key examples.

Gaussian mixture: $f(x) = \sum_i w_i \, \mathcal{N}(x; \mu_i, \sigma_i^2)$ . Used in clustering and density estimation.
Poisson mixture: a Poisson–gamma mixture gives the negative binomial distribution (worked above).
Scale mixtures of normals: $X = \sigma Z$ where $Z \sim \mathcal{N}(0,1)$ and $\sigma^2$ has some distribution. The t-distribution is a scale mixture of normals with an inverse-gamma mixing distribution.

The relationship to Singular and mixed distributions: a mixture of an atomic and a continuous distribution is a mixed distribution in the Lebesgue-decomposition sense.

Authoritative sources

R. D. Durrett, Probability: Theory and Examples (5th ed.)

Example problem

Problem. A coin is chosen at random: with probability $1/3$ it is fair, with probability $2/3$ it is biased with $P(H)=3/4$ . Find the mixture probability of heads, and the probability that the coin is fair given that it landed heads.

Solution

Mixture probability of heads: $P(H) = \tfrac{1}{3}\cdot\tfrac{1}{2} + \tfrac{2}{3}\cdot\tfrac{3}{4} = \tfrac{1}{6}+\tfrac{1}{2} = \tfrac{2}{3}$ .

By Bayes’ rule, $P(\text{fair}\mid H) = \frac{P(H\mid\text{fair})P(\text{fair})}{P(H)} = \frac{(1/2)(1/3)}{2/3} = \tfrac{1}{4}$ . The latent-class (mixture) structure is exactly what makes the posterior computable.

How the operations interact

Convolution and mixture are not interchangeable, and the distinction is the key to a clean mental map of the algebra.

Mixture of convolutions vs. convolution of mixtures. A mixture of convolutions, $\sum_i w_i(\mu_i * \nu_i)$ , first forms each sum and then averages the resulting laws. A convolution of mixtures, $(\sum_i w_i \mu_i) * (\sum_j v_j \nu_j)$ , expands by bilinearity into $\sum_{i,j} w_i v_j\,(\mu_i * \nu_j)$ — a double sum over all cross-pairs. The two are equal only in special cases (e.g. when the mixing is degenerate). Intuitively, a mixture of convolutions keeps the two summands of each pair tied together, while a convolution of mixtures lets every component of one sum pair with every component of the other.
Exponential family under convolution vs. mixture. For a closed subfamily (normal, Poisson, gamma with fixed shape, stable laws), convolution of two members stays in the family — the natural parameters add. A mixture of two members of the same exponential family is, in general, not a member of that family: a two-component Gaussian mixture is not Gaussian, and a Poisson–gamma mixture is negative binomial, not Poisson. So convolution tends to preserve the exponential-family structure (for closed subfamilies), while mixture tends to destroy it.
Both are linear in the measure. Mixture is literally linear ( $\sum w_i \mu_i$ ); convolution is bilinear ( $(\mu_1+\mu_2)*\nu = \mu_1*\nu + \mu_2*\nu$ ). This linearity is why characteristic functions turn both operations into algebra: mixture becomes a weighted sum and convolution becomes a product of $\varphi$ .

Exponential families

A one-parameter exponential family has density (or PMF) of the form

f_\theta(x) = h(x) \, \exp\bigl(\theta\, T(x) - A(\theta)\bigr),

where $h$ is the base measure, $T$ is the sufficient statistic, $\theta$ is the natural parameter, and $A(\theta) = \log \int h(x) e^{\theta T(x)}\, dx$ is the log-partition function (normalizer).

Key properties:

$E_\theta[T(X)] = A'(\theta)$ (mean of the sufficient statistic).
$\mathrm{Var}_\theta(T(X)) = A''(\theta)$ (variance).
The natural parameter space is $\{\theta : A(\theta) < \infty\}$ .
Closure under convolution holds only for specific subfamilies (normal, Poisson, gamma with fixed shape, stable laws), not for the exponential family as a whole.

Details

The exponential family is the natural class for statistical inference: it is the class of distributions for which the sufficient statistic has fixed dimension regardless of sample size. The key members:

Distribution	T(x)	θ	A(θ)
Normal (known variance)	x	μ/σ²	μ²/(2σ²)
Poisson	x	log λ	λ
Exponential	x	−λ	−log λ
Bernoulli	x	log(p/(1−p))	−log(1−p)
Gamma (known shape)	x	−β	α log β

Worked example (recovering the mean from $A$ ). For the Poisson family, $h(x)=1/x!$ , $T(x)=x$ , $\theta=\log\lambda$ , so $A(\theta)=\lambda=e^{\theta}$ . Then $E[X]=A'(\theta)=e^{\theta}=\lambda$ and $\mathrm{Var}(X)=A''(\theta)=e^{\theta}=\lambda$ , recovering the familiar Poisson mean and variance purely from the log-partition function.

The conjugate prior for an exponential family is itself a member of the exponential family (in the parameter $\theta$ ). This is the algebraic reason Bayesian updating is tractable for these families. See Statistical inference for the full treatment.

The relationship to Absolutely continuous distributions: the normal, exponential, gamma, and beta are all absolutely continuous members of the exponential family. The relationship to Atomic distributions: the Poisson, binomial, and geometric are atomic members.

Authoritative sources

B. Efron, Exponential Families in Theory and Practice (Cambridge University Press)

Example problem

Problem. Write the Bernoulli( $p$ ) PMF in exponential-family form and identify $h$ , $T$ , $\theta$ , and $A$ .

Solution

$p^x(1-p)^{1-x} = \exp\bigl(x\log\frac{p}{1-p} + (1-x)\log(1-p)\bigr)$ . So $h(x)=1$ , $T(x)=x$ , $\theta=\log\frac{p}{1-p}$ , and $A(\theta)=-\log(1-p)=-\log(1+e^{-\theta})$ . Check: $A'(\theta)=\frac{e^{-\theta}}{1+e^{-\theta}}=\frac{1}{1+e^{\theta}}=p$ , so $E[X]=p$ as expected.

Location-scale families

A location-scale family is generated from a base distribution by affine transformations: if $X_0$ has CDF $F_0$ , then $X = \mu + \sigma X_0$ has CDF

F(x) = F_0\bigl((x - \mu)/\sigma\bigr), \quad \sigma > 0.

The location parameter $\mu$ shifts the distribution.
The scale parameter $\sigma$ stretches it.
The base distribution $F_0$ is the standard member ( $\mu = 0$ , $\sigma = 1$ ).

Examples: normal, logistic, uniform, Gumbel.

Details

Location-scale families are closed under affine transformations: the sum of independent location-scale variables from the same family is again in the family (if the base is closed under convolution). The normal family is the unique location-scale family that is also closed under convolution for all parameter values (the stable property, see below).

Worked example (standardizing). If $X \sim \mathcal{N}(\mu, \sigma^2)$ , then $Z=(X-\mu)/\sigma \sim \mathcal{N}(0,1)$ . Conversely, any $\mathcal{N}(\mu,\sigma^2)$ is an affine image of the standard normal, so the whole normal family is a location-scale family with base $\mathcal{N}(0,1)$ . The density transforms by the change-of-variables rule $f_X(x)=\frac{1}{\sigma}f_0((x-\mu)/\sigma)$ .

The relationship to Absolutely continuous distributions: the transformation formula $f_Y(y) = \frac{1}{\sigma} f_0((y-\mu)/\sigma)$ is the change-of-variables applied to the location-scale map.

Example problem

Problem. If $X \sim \mathrm{Uniform}(0,1)$ , find the CDF and density of $Y = 3 + 2X$ .

Solution

$F_Y(y) = P(3+2X \le y) = P(X \le (y-3)/2) = (y-3)/2$ for $3 \le y \le 5$ , and $0$ /1 outside. So $Y \sim \mathrm{Uniform}(3,5)$ , with density $f_Y(y)=1/2$ on $[3,5]$ . The location-scale map sends the standard uniform to a uniform on any interval — the uniform family is a location-scale family.

Stable distributions

A distribution is stable if the sum of independent copies, after affine rescaling, has the same distribution: for i.i.d. $X_1, X_2 \sim F$ ,

X_1 + X_2 \stackrel{d}{=} a X + b, \quad a > 0.

Stable laws are parameterized by:

Stability index $\alpha \in (0, 2]$ : controls tail heaviness ( $P(|X| > t) \sim t^{-\alpha}$ ).
Skewness $\beta \in [-1, 1]$ .
Location $\mu$ and scale $\sigma > 0$ .

Key facts:

$\alpha = 2$ : the normal distribution (the only stable law with finite variance).
$\alpha = 1$ : the Cauchy distribution ( $\beta = 0$ ) or logistic-type ( $\beta \neq 0$ ).
$\alpha = 1/2$ : the positive stable law (first-passage times of Brownian motion).
The characteristic function is $\varphi(t) = \exp(-\sigma^\alpha |t|^\alpha [1 - i\beta \, \mathrm{sgn}(t) \, \chi(\alpha, t)])$ .
Stable laws are the attractors of the generalized central limit theorem: sums of i.i.d. variables with $\alpha$ -stable tails converge to a stable law.

Details

The stable distributions are the universal limits for sums of heavy-tailed variables, just as the normal is the universal limit for sums of light-tailed (finite-variance) variables. The relationship to Limit theorems is direct: the generalized CLT characterizes which stable law is the attractor based on the tail index of the summands.

Reading the plot. The figure below overlays three stable densities on a common axis. The normal ( $\alpha=2$ ) has light tails and finite variance. The Cauchy ( $\alpha=1$ ) has heavy tails and no finite mean. The positive stable / Lévy law ( $\alpha=1/2$ ) is supported on $x>0$ and has an extremely heavy right tail. As $\alpha$ decreases the tails get heavier and the peak sharper.

The Cauchy distribution ( $\alpha = 1, \beta = 0$ ) has no finite moments: $E|X| = \infty$ . Its density is $f(x) = \frac{1}{\pi(1+x^2)}$ and its CDF is $F(x) = \frac{1}{\pi}\arctan(x) + \frac{1}{2}$ .

Authoritative sources

R. D. Durrett, Probability: Theory and Examples (5th ed.)

Example problem

Problem. If $X_1, X_2$ are i.i.d. Cauchy, show that $(X_1+X_2)/2$ is also Cauchy, and explain why the sample mean of Cauchy variables does not converge to a constant.

Solution

The Cauchy characteristic function is $\varphi(t)=e^{-|t|}$ . For the average, $\varphi_{(X_1+X_2)/2}(t)=\varphi(t/2)^2=e^{-|t/2|}e^{-|t/2|}=e^{-|t|}=\varphi(t)$ , so the average is again standard Cauchy. Because the mean is not a finite constant (the Cauchy has no finite mean), the law of large numbers does not apply: the sample mean has the same Cauchy law for every $n$ , so it does not concentrate.

Transformations

A transformation (change of variables) pushes a distribution forward through a function: if $Y = g(X)$ and $g$ is a one-to-one differentiable map with inverse $g^{-1}$ , then the density of $Y$ is

f_Y(y) = f_X\bigl(g^{-1}(y)\bigr)\,\left|\frac{d}{dy}g^{-1}(y)\right|.

The Jacobian factor $|d\,g^{-1}/dy|$ accounts for how $g$ stretches or compresses intervals.
Monotone maps are the clean case; for non-injective $g$ one sums over all preimages.
Location-scale families are exactly the transformations $g(x)=\mu+\sigma x$ of a base law.

Details

Transformations are the third basic operation: where convolution adds independent quantities and mixture averages over latent classes, a transformation relabels the outcomes of a single quantity. The Jacobian is what keeps total probability equal to one: the density is rescaled by the local stretch factor so that $\int f_Y(y)\,dy = \int f_X(x)\,dx = 1$ .

Worked example (log transform). If $X \sim \mathcal{N}(0,1)$ and $Y=e^X$ , then $g^{-1}(y)=\log y$ with derivative $1/y$ , so $f_Y(y)=\frac{1}{y}\frac{1}{\sqrt{2\pi}}e^{-(\log y)^2/2}$ for $y>0$ — the log-normal density. The full treatment, including the multivariate Jacobian and copulas, is in Absolutely continuous distributions.

Example problem

Problem. If $X \sim \mathrm{Uniform}(0,1)$ , find the density of $Y = -\ln X$ .

Solution

$g^{-1}(y)=e^{-y}$ with derivative $-e^{-y}$ , so $|d\,g^{-1}/dy|=e^{-y}$ . Since $f_X=1$ on $(0,1)$ , $f_Y(y)=e^{-y}$ for $y>0$ . Thus $Y \sim \mathrm{Exp}(1)$ : the exponential is the log-transform of the uniform, a transformation that turns a bounded law into an unbounded one.

Order statistics

Given i.i.d. $X_1, \ldots, X_n$ with CDF $F$ , the order statistics are $X_{(1)} \leq X_{(2)} \leq \cdots \leq X_{(n)}$ . The density of the $k$ -th order statistic is

f_{X_{(k)}}(x) = \frac{n!}{(k-1)!(n-k)!} \, [F(x)]^{k-1} \, f(x) \, [1 - F(x)]^{n-k}.

The minimum $X_{(1)}$ has CDF $1 - [1-F(x)]^n$ .
The maximum $X_{(n)}$ has CDF $[F(x)]^n$ .
The range is $X_{(n)} - X_{(1)}$ .

Details

Order statistics are fundamental in extreme value theory, robust statistics, and reliability. The relationship to Limit theorems: the maximum of i.i.d. samples, after appropriate normalization, converges to one of three extreme value distributions (Gumbel, Fréchet, or Weibull), depending on the tail of $F$ .

Reading the spacings. For a uniform sample the ordered points split $(0,1)$ into $n+1$ gaps. The figure shows a sample of $n=5$ with the spacings highlighted; these spacings are exchangeable and, in the uniform case, jointly Dirichlet.

For the uniform(0, 1) distribution, the order statistics have a simple Dirichlet structure: the spacings $(X_{(1)}, X_{(2)} - X_{(1)}, \ldots, 1 - X_{(n)})$ are distributed as a Dirichlet(1, 1, …, 1) random vector.

Authoritative sources

R. D. Durrett, Probability: Theory and Examples (5th ed.)

Example problem

Problem. For $n=3$ i.i.d. $\mathrm{Uniform}(0,1)$ variables, find the density of the maximum $X_{(3)}$ and its expected value.

Solution

The CDF of the maximum is $F_{X_{(3)}}(x)=x^3$ for $x\in[0,1]$ , so the density is $f_{X_{(3)}}(x)=3x^2$ . The expectation is $E[X_{(3)}]=\int_0^1 x\cdot 3x^2\,dx=\int_0^1 3x^3\,dx=3/4$ . In general the maximum of $n$ uniforms has mean $n/(n+1)$ , approaching $1$ as $n$ grows.

Example problem

Problem. Let $X_1, \ldots, X_n$ be i.i.d. $\mathrm{Exp}(\lambda)$ . Show that $S_n = X_1 + \cdots + X_n$ has a gamma distribution, and find its parameters. Then show that the minimum $X_{(1)}$ is also exponential, and find its rate.

Solution

Sum (convolution): The MGF of $\mathrm{Exp}(\lambda)$ is $M(t) = \frac{\lambda}{\lambda - t}$ for $t < \lambda$ . The MGF of $S_n$ is

M_{S_n}(t) = \bigl(M(t)\bigr)^n = \frac{\lambda^n}{(\lambda - t)^n}.

This is the MGF of $\mathrm{Gamma}(n, \lambda)$ (shape $n$ , rate $\lambda$ ). So $S_n \sim \mathrm{Gamma}(n, \lambda)$ .

Minimum (order statistic): The CDF of $X_{(1)}$ is

P(X_{(1)} > x) = P(\forall i: X_i > x) = e^{-n\lambda x}.

Therefore $X_{(1)} \sim \mathrm{Exp}(n\lambda)$ .

The rate is $n\lambda$ : the minimum of $n$ exponentials is exponential with rate equal to the sum of the individual rates. This is the memoryless property in action.

Distributional structure and operations §

Convolution §

Mixtures §

How the operations interact §

Exponential families §

Location-scale families §

Stable distributions §

Transformations §

Order statistics §

Example problem §

Distributional structure and operations

Convolution

Mixtures

How the operations interact

Exponential families

Location-scale families

Stable distributions

Transformations

Order statistics

Example problem