Conditional probability §

Step 1: The restriction idea.

Suppose you draw a card from a standard 52-card deck. Let $A$ = “the card is a King” and $B$ = “the card is a heart”. Without any information, $P(A) = 4/52 = 1/13$. Now you are told the card is a heart. The sample space shrinks to the 13 hearts. Among those, exactly one is a King, so $P(A \mid B) = 1/13$.

Formally: $P(A \cap B) = 1/52$ (the King of hearts), $P(B) = 13/52 = 1/4$, and indeed $P(A \mid B) = (1/52)/(1/4) = 1/13$.

Step 2: Renormalization.

The division by $P(B)$ is a renormalization. Before conditioning, the 13 hearts carry total mass $13/52$. After conditioning, they must carry total mass 1, so every probability inside $B$ is scaled up by $1/P(B) = 4$.

Step 3: When $P(B) = 0$.

The formula breaks down when $P(B) = 0$ (e.g. “the temperature is exactly 23.456 °C”). The remedy is the regular conditional probability, developed in the last section of this page. There, one replaces the quotient by a limit or by a measure-theoretic construction (Radon–Nikodým derivative).

Example problem.

Step 1: Independence as a structural property.

Independence is a statement about the joint measure, not about moments. Two variables can have zero covariance yet be strongly dependent.

Worked example. Let $X \sim \text{Unif}[0, 2\pi]$ and $Y = \sin(X)$. Then:

• $\mathbb{E}[X] = \pi$, $\mathbb{E}[Y] = 0$.
• $\mathbb{E}[XY] = \mathbb{E}[X \sin X] = \int_0^{2\pi} x \sin x \, \frac{dx}{2\pi} = 0$ (by symmetry: the integrand is odd about $\pi$).
• So $\operatorname{Cov}(X, Y) = 0 - \pi \cdot 0 = 0$.

Yet $Y$ is a deterministic function of $X$, so knowing $X$ determines $Y$ exactly. The variables are maximally dependent despite zero covariance.

Step 2: Conditional independence.

$X \perp\!\!\perp Y \mid Z$ means $P(X \in A, Y \in B \mid Z) = P(X \in A \mid Z)\, P(Y \in B \mid Z)$ almost surely. This is the key relation in Bayesian networks and hidden Markov models (see Stochastic processes).

Step 3: Mutual vs. pairwise independence.

Pairwise independence (every pair is independent) does not imply mutual independence. Classic counterexample: let $X_1, X_2$ be independent Rademacher (±1 with equal probability) and set $X_3 = X_1 X_2$. Then every pair is independent, but $X_1 X_2 X_3 = 1$ always, so the triple is not mutually independent.

Example problem.

Step 1: The algebraic inversion.

Bayes’ theorem converts $P(B \mid A)$ (how the data looks given a hypothesis) into $P(A \mid B)$ (how plausible the hypothesis is given the data). The numerator is the joint $P(A \cap B) = P(B \mid A)\,P(A) = P(A \mid B)\,P(B)$, and the theorem simply solves for $P(A \mid B)$.

Step 2: The base-rate fallacy.

A common error is to ignore the prior $P(A)$ and treat the likelihood $P(B \mid A)$ as the posterior. When the prior is very small, the posterior can be far smaller than the likelihood suggests. The interactive simulation above makes this tangible: set the prior to 0.001 and the sensitivity to 0.99, and watch the posterior stay near 2% even with a 99% sensitive test.

Step 3: The law of total probability as a partition identity.

The denominator $P(B) = \sum_i P(B \mid A_i)\,P(A_i)$ is not a separate theorem; it is the definition of $P(B)$ expanded over a partition. For a binary partition $\{A, A^c\}$: $P(B) = P(B \mid A)\,P(A) + P(B \mid A^c)\,P(A^c)$.

Worked example (disease screening).

A disease affects 1 in 10 000 people. A test has sensitivity 99% and specificity 99.5%. A patient tests positive. What is the probability the patient actually has the disease?

Let $D$ = has disease, $T$ = tests positive.

• Prior: $P(D) = 10^{-4}$
• Sensitivity: $P(T \mid D) = 0.99$
• False-positive rate: $P(T \mid D^c) = 0.005$

Despite a positive test, the probability of disease is only about 2%. The low prior (base rate) dominates.

Example problem.

Step 1: Derivation from the definition.

The chain rule follows by repeated application of $P(A \mid B) = P(A \cap B)/P(B)$. For three variables:

Each step peels off one variable by conditioning on all previous ones.

Step 2: The continuous case.

For joint densities: $f(x_1, \ldots, x_n) = \prod_i f(x_i \mid x_1, \ldots, x_{i-1})$. This is the form used in autoregressive models and in the sequential factorization of language models.

Step 3: Ordering matters for compactness.

The ordering is arbitrary; different orderings give different factorizations of the same joint. Choosing an ordering that matches the causal or temporal structure of the problem yields the most compact model. For example, in a Markov chain, the temporal ordering gives $P(x_1, \ldots, x_n) = P(x_1)\prod_i P(x_i \mid x_{i-1})$, a product of $n$ one-step transitions rather than $n$ full-history conditionals.

Example problem.

Step 1: The discrete case.

When $\mathcal{G}$ is generated by a discrete random variable $Y$ with values $y_1, \ldots, y_k$, the regular conditional probability reduces to the familiar formula:

The random variable $P(A \mid \mathcal{G})$ simply takes the value $P(A \mid Y = y_i)$ on the set $\{Y = y_i\}$.

Step 2: The continuous case.

In the continuous case, a regular conditional probability $P(A \mid Y = y)$ exists as a probability kernel $y \mapsto P(\cdot \mid Y = y)$ under mild conditions (standard Borel spaces). This is the object that makes Bayesian updating rigorous when the observation is continuous. The construction uses the Radon–Nikodým theorem: the measure $\mu(B) = P(A \cap B)$ is absolutely continuous with respect to $\nu(B) = P(B)$ on $\mathcal{G}$, and the RN derivative is $P(A \mid \mathcal{G})$.

Step 3: Connection to conditional expectation.

The conditional expectation $\mathbb{E}[X \mid \mathcal{G}]$ is defined identically, with $P(A \cap B)$ replaced by $\int_B X\, dP$. It is the “integral version” of conditional probability: where $P(A \mid \mathcal{G})$ answers “what is the probability of $A$ given the information in $\mathcal{G}$?”, $\mathbb{E}[X \mid \mathcal{G}]$ answers “what is the best $\mathcal{G}$-measurable guess of $X$?”. See Random variables and expectation for the full development.

Example problem.

A disease affects 1 in 10 000 people. A test has sensitivity 99% and specificity 99.5%. A patient tests positive. What is the probability the patient actually has the disease?