Random variables and expectation §

The measurability requirement ensures that events of the form $\{\omega : X(\omega) \in B\}$ are in the sigma-algebra, so they have a well-defined probability. For a finite or countable sample space every function is measurable; the condition only becomes restrictive for uncountable spaces. The general notion of a measurable map between measure spaces is developed in Measurable functions.

The sigma-algebra generated by $X$ is $\sigma(X) = \{X^{-1}(B) : B \in \mathcal{B}(\mathbb{R})\}$. Two random variables $X, Y$ are independent iff $\sigma(X)$ and $\sigma(Y)$ are independent sigma-algebras (see Conditional probability).

Worked example (pushforward of a finite space). Flip two fair coins: $\Omega = \{HH, HT, TH, TT\}$ with $P(\omega) = 1/4$ each, and let $X(\omega)$ = number of heads. The pushforward assigns $P(X=0) = 1/4$ (only $TT$), $P(X=1) = 1/2$ ($HT, TH$), $P(X=2) = 1/4$ (only $HH$). All the probability questions about $X$ — its mean, its tails — now live on the real line, where they can be computed with the tools below. This is exactly the distribution the Distributions page calls Binomial(2, 1/2).

Example problem. Let $X$ be the constant random variable $X(\omega) = c$ on any sample space. What is $\sigma(X)$, and what is the pushforward measure $X_*P$?

The expectation is the Lebesgue integral of $X$ with respect to the probability measure $P$. It exists (is finite) iff $\int |X|\, dP < \infty$. The integral is defined on the measure space $(\Omega, \mathcal{F}, P)$; see Measure theory and the Lebesgue integral for the general framework.

The transformation rule (law of the unconscious statistician). Expectations are almost never computed on $\Omega$; one pushes forward to the real line first. For any measurable $g$:

depending on whether the law of $X$ is atomic or has density $f$. This is the bridge to the Distributions page: every practical expectation is an integral against the distribution, not against $P$. When $g$ is differentiable and one only needs $\mathbb{E}[g(X)]$ approximately, the delta method replaces $g$ by its first-order Taylor expansion around $\mathbb{E}[X]$: $\mathbb{E}[g(X)] \approx g(\mathbb{E}[X]) + \tfrac{1}{2} g''(\mathbb{E}[X]) \operatorname{Var}(X)$.

Key properties:

• Monotonicity: $X \leq Y$ a.s. $\implies \mathbb{E}[X] \leq \mathbb{E}[Y]$
• Jensen’s inequality: for convex $\varphi$, $\varphi(\mathbb{E}[X]) \leq \mathbb{E}[\varphi(X)]$
• Tower property: $\mathbb{E}[\mathbb{E}[X \mid \mathcal{G}]] = \mathbb{E}[X]$

Worked example. For a fair die, $\mathbb{E}[X] = (1+2+3+4+5+6)/6 = 3.5$. Using the transformation rule with $g(x) = x^2$: $\mathbb{E}[X^2] = (1+4+9+16+25+36)/6 = 91/6 \approx 15.17$. Jensen is visible here: $\mathbb{E}[X^2] = 15.17 \geq (\mathbb{E}[X])^2 = 12.25$, with the gap being exactly the variance. The transformation rule also carries a warning: for $X \sim \text{Unif}[0,1]$ and $g(x) = 1/x$, the integral $\int_0^1 \frac{dx}{x}$ diverges, so $\mathbb{E}[1/X] = \infty$ even though $g$ is smooth — finiteness must always be checked.

Authoritative sources

• P. Billingsley, Probability and Measure, §19–21 (expectation as Lebesgue integral)

Example problem. Let $X \sim \text{Unif}[0,1]$. Compute $\mathbb{E}[X^2]$ and $\mathbb{E}[e^X]$ using the transformation rule.

Variance measures spread around the mean. For independent $X, Y$: $\operatorname{Var}(X+Y) = \operatorname{Var}(X) + \operatorname{Var}(Y)$ (the cross term vanishes). Without independence, $\operatorname{Var}(X+Y) = \operatorname{Var}(X) + \operatorname{Var}(Y) + 2\operatorname{Cov}(X,Y)$. Variance is homogeneous of degree 2: $\operatorname{Var}(aX + b) = a^2 \operatorname{Var}(X)$.

Zero covariance does not imply independence. Discrete example: let $X$ be uniform on $\{-1, 0, 1\}$ and $Y = X^2$. Then $\operatorname{Cov}(X, Y) = \mathbb{E}[X^3] - \mathbb{E}[X]\,\mathbb{E}[X^2] = 0 - 0 \cdot \frac{2}{3} = 0$, yet $Y$ is a deterministic function of $X$, so the two are maximally dependent. Independence implies zero covariance, but not conversely — covariance only detects linear dependence.

Worked example. For the fair die: $\mathbb{E}[X^2] = 91/6$ and $(\mathbb{E}[X])^2 = 12.25 = 49/4$, so

Example problem. Using the die’s variance $35/12$, compute $\operatorname{Var}(2X + 3)$ without recomputing any expectation.

Why the MGF and not the moment sequence? Moments do not always determine a distribution: this is the Stieltjes/Hamburger moment problem. Classic example: the log-normal distributions with density $f(x) \propto x^{-1} \exp(-(\ln x - \mu)^2/2)$ (for suitable parameter pairs) share all their moments yet are different laws. So knowing every moment is not enough in general. What saves the day is analyticity: when $M_X$ is finite on an open interval around 0, it is a holomorphic function there, the identity theorem for holomorphic functions propagates its values from that interval to the whole complex plane, and the distribution is pinned down. When even that fails (heavy tails), one falls back on the characteristic function, for which the inversion theorem always holds.

Cumulants. The cumulant generating function is $K_X(t) = \log M_X(t)$; its derivatives at 0 give the cumulants. The first two cumulants are the mean and variance. The third and fourth cumulants, suitably standardized, are the skewness $\kappa_3/\sigma^3$ (asymmetry of the distribution) and the excess kurtosis $\kappa_4/\sigma^4$ (tail heaviness relative to a normal) — the standard descriptors beyond mean and variance.

Sums. For independent $X_1, \ldots, X_n$: $M_{\sum X_i}(t) = \prod_i M_{X_i}(t)$. This multiplicative property is the workhorse behind the Central Limit Theorem (see Limit theorems).

Worked example. Let $X \sim \text{Exp}(\lambda)$ (density $\lambda e^{-\lambda x}$ on $x \geq 0$). Then

Differentiating: $M_X'(0) = 1/\lambda$ (the mean) and $M_X''(0) = 2/\lambda^2$, so $\operatorname{Var}(X) = 2/\lambda^2 - 1/\lambda^2 = 1/\lambda^2$. Note the MGF only exists for $t < \lambda$ — a neighborhood of 0, which is all the uniqueness argument needs.

For variables where the MGF does not exist at all (e.g. Cauchy), the characteristic function $\varphi_X(t) = \mathbb{E}[e^{itX}]$ always exists (it is bounded by 1) and serves the same role; the inversion theorem applies to it.

Authoritative sources Authoritative sources

• The moment problem (Stieltjes/Hamburger): moments need not determine a law
• Characteristic function and the inversion theorem

These inequalities bound tail probabilities using only moments or boundedness, without knowing the full distribution. Each stronger hypothesis buys a faster decay, as the plot above shows.

• Markov: uses only the first moment. Proof: $\mathbb{E}[X] \geq \int_{\{X \geq a\}} X\, dP \geq a\, P(X \geq a)$.
• Chebyshev: applies Markov to $(X - \mu)^2$. Gives polynomial decay $O(1/k^2)$.
• Hoeffding: uses the boundedness and independence to get exponential decay. It is the key tool in learning theory and the analysis of randomized algorithms.

Worked example. Roll a fair die $n = 100$ times; $\mu = 3.5$, $\sigma^2 = 35/12$, support width $b - a = 5$. Bound $P(|\bar{X} - 3.5| \geq 1)$:

• Chebyshev: $\dfrac{\sigma^2}{n \varepsilon^2} = \dfrac{35/12}{100 \cdot 1} \approx 0.029$.
• Hoeffding (two-sided version): $2\exp\!\left(-\dfrac{2 \cdot 100 \cdot 1^2}{25}\right) = 2e^{-8} \approx 0.00067$.

Hoeffding is about 40× sharper here. The comparison flips near the mean (small $\varepsilon$), where Chebyshev’s $1/\varepsilon^2$ can beat Hoeffding’s $e^{-c\varepsilon^2}$ — each bound wins in its own regime, and the plot above shows why: polynomial vs exponential decay.

For sub-Gaussian variables (e.g. normal), Hoeffding-type bounds hold with a variance proxy. For sub-exponential variables, the tail is $\exp(-ct)$ rather than $\exp(-ct^2)$.

These inequalities are the probabilistic backbone of the law of large numbers (see Limit theorems) and of statistical estimation (see Statistical inference).

Authoritative sources

• W. Hoeffding (1963), original paper — see the summary and references

Example problem. A sensor reads a value in $[0, 10]$ with unknown mean $\mu$. You average $n = 400$ independent readings. Give a bound (valid with no distributional assumptions beyond boundedness) on $P(|\bar{X} - \mu| \geq 0.5)$.

Let $X_1, \ldots, X_n$ be i.i.d. with $\mathbb{E}[X_i] = \mu$ and $\operatorname{Var}(X_i) = \sigma^2$, and let $\bar{X} = \frac{1}{n}\sum_i X_i$.

1. Compute $\mathbb{E}[\bar{X}]$ and $\operatorname{Var}(\bar{X})$.
2. Use Chebyshev to bound $P(|\bar{X} - \mu| \geq \varepsilon)$.
3. What does the bound say as $n \to \infty$, and which theorem sharpens it?