Bayesian inference §

Bayes’ theorem is an application of Conditional probability to the parameter. The key components:

• Prior $\pi(\theta)$: encodes beliefs before seeing the data. Can be informative (based on previous studies) or diffuse (weakly informative, e.g., $\pi(\theta) \propto 1$ for a location parameter).
• Likelihood $p(x \mid \theta)$: the sampling model, same as in Estimation.
• Posterior $\pi(\theta \mid x)$: the updated belief. All uncertainty about $\theta$ is captured by this distribution.
• Evidence $p(x) = \int p(x \mid \theta)\, \pi(\theta)\, d\theta$: the marginal probability of the data. Used for model comparison (Bayes factors).

The posterior mean, median, and mode are all valid point summaries. The posterior credible interval is the Bayesian analogue of the confidence interval: a 95% credible interval contains the true parameter with probability 0.95 (under the posterior).

Visual intuition. The posterior is proportional to the pointwise product of the prior and the likelihood. The diagram below shows a typical case: a broad prior, a narrow likelihood centred away from the prior mean, and the resulting posterior that is pulled toward the likelihood but retains some of the prior’s spread.

• Jeffreys prior. Derived from the Fisher information matrix $I(\theta)$: $\pi(\theta) \propto \sqrt{\det I(\theta)}$. It is invariant under reparameterization and is the most common “default” objective prior. For a Bernoulli model, $I(\theta) = 1/[\theta(1-\theta)]$, giving $\pi(\theta) \propto [\theta(1-\theta)]^{-1/2}$, i.e., $\text{Beta}(1/2, 1/2)$ (the Haldane–Jeffreys prior).

• Reference prior. Defined by Berger and Robert (1984) as the prior that maximizes the expected Kullback–Leibler divergence between the posterior and the prior. It is unique up to a constant and is designed so that the posterior is as “informative” as possible. For a Normal mean with known variance, the reference prior is flat: $\pi(\mu) \propto 1$.

• Haldane prior. The improper prior $\pi(\theta) \propto 1/[\theta(1-\theta)]$ for a Bernoulli parameter. It encodes complete indifference (zero prior pseudo-observations) and yields a posterior $\text{Beta}(s, n-s)$ that places zero mass at the boundaries $\theta = 0$ and $\theta = 1$.

• Flat (uniform) prior. $\pi(\theta) \propto 1$ on the parameter space. Simple but not invariant under reparameterization: a uniform prior on $\theta$ does not imply a uniform prior on $\log\theta$.

The choice of prior affects the posterior, especially with small samples. With large $n$, the likelihood dominates and the posterior becomes prior-insensitive (the Bernstein–von Mises theorem).

Authoritative sources

• H. Jeffreys, Theory of Probability (1939), Ch. 4
• J. O. Berger, R. E. Robert, The Open Reference Prior (1984)

Conjugacy gives closed-form posteriors, enabling exact computation without MCMC. It also makes sequential updating trivial: the posterior after batch 1 becomes the prior for batch 2.

Example. Flipping a coin: prior $\theta \sim \text{Beta}(1, 1)$ (uniform). After 7 heads and 3 tails, the posterior is $\text{Beta}(8, 4)$ with mean $8/12 = 0.667$.

Metropolis–Hastings. At each step, given the current state $\theta_t$, draw a proposal $\theta^* \sim q(\cdot \mid \theta_t)$ from a proposal distribution $q$ (often a symmetric random walk, e.g., $\theta^* = \theta_t + \epsilon$ with $\epsilon \sim \mathcal{N}(0, \sigma^2)$). Accept the move with probability

For a symmetric proposal ($q(\theta_t|\theta^*) = q(\theta^*|\theta_t)$), this reduces to the ratio of posterior densities. If rejected, the chain stays at $\theta_t$. The tuning parameter $\sigma$ controls the acceptance rate; an optimal acceptance rate for random-walk MH in one dimension is about 0.44.

Hamiltonian Monte Carlo (HMC) / NUTS. Augments the parameter with a momentum variable and simulates Hamiltonian dynamics using gradients of the log-posterior. This allows large, correlated moves that explore the posterior efficiently, especially in moderate dimensions. The No-U-Turn Sampler (NUTS) adapts the trajectory length automatically.

Variational inference. Approximates the posterior with a simpler family (e.g., a Gaussian) by minimizing the KL divergence $\text{KL}(q(\theta) \| \pi(\theta|x))$. Faster than MCMC but introduces approximation bias.

Convergence diagnostics. Because MCMC samples are autocorrelated, one must verify that the chain has reached stationarity and is mixing well:

• Gelman–Rubin $\hat{R}$ (R-hat): run multiple chains from different starting points; $\hat{R} \approx 1$ (typically < 1.01) indicates convergence.
• Effective sample size (ESS): the number of independent samples equivalent to the autocorrelated chain. A rule of thumb is ESS > 400 for reliable quantile estimates.
• Trace plots and autocorrelation plots provide visual confirmation.

The choice of method depends on dimensionality and smoothness:

• Low dimension, smooth: HMC/NUTS (as in Stan, PyMC).
• High dimension, sparse: coordinate-wise MCMC or variational methods.
• Very high dimension: variational inference or normalizing flows.

The posterior is a Distribution in its own right: it has a mean, variance, quantiles, and can be used for prediction via the posterior predictive distribution.

The posterior predictive distribution accounts for both the data-generating uncertainty (aleatoric) and the parameter uncertainty (epistemic). It is wider than the likelihood evaluated at the posterior mean because it averages over the spread of the posterior.

Uses:

• Prediction: sample $\theta^{(t)} \sim \pi(\theta|x)$ (e.g., from MCMC), then $x_{\text{new}}^{(t)} \sim p(x_{\text{new}}|\theta^{(t)})$. The collection $\{x_{\text{new}}^{(t)}\}$ approximates the predictive distribution.
• Model checking (posterior predictive checks): generate replicated datasets $x^{\text{rep}}$ from the posterior predictive and compare summary statistics (mean, variance, extremes) to the observed data. Large discrepancies indicate model misspecification.
• Bayesian forecasting: in time-series models, the posterior predictive gives a full predictive distribution with credible intervals that widen further into the future as parameter uncertainty accumulates.

For a conjugate model the posterior predictive often has a closed form. For example, with a Bernoulli likelihood and Beta(α, β) prior, after observing $s$ successes in $n$ trials, the posterior predictive for the next flip is a Beta-Binomial: $P(x_{\text{new}}=1 \mid x) = (\alpha+s)/(\alpha+\beta+n)$.

A Bayes factor $BF_{10} > 1$ favors $M_1$; $BF_{10} < 1$ favors $M_0$. It is the ratio of marginal likelihoods and automatically penalizes model complexity (Occam’s factor): a more complex model spreads its prior over more parameter values, reducing the marginal likelihood.

A common qualitative scale (Jeffreys, 1961):

This connects to Hypothesis testing: the Bayes factor is the Bayesian analogue of the likelihood ratio test, but it integrates over the parameter space rather than maximizing.

Example. Testing $H_0: \mu = 0$ vs. $H_1: \mu \neq 0$ with a Normal model. The Bayes factor depends on the prior width under $H_1$ and the sample size. For large $n$, it approximates the BIC (Bayesian Information Criterion) comparison.

(a) The posterior is $\text{Beta}(2 + 14,\; 2 + 6) = \text{Beta}(16, 8)$.

(b) The posterior mean is $16/(16+8) = 2/3 \approx 0.667$. The 95% credible interval is approximately $[0.47,\; 0.84]$ (from the Beta(16, 8) quantiles: 2.5th percentile ≈ 0.47, 97.5th percentile ≈ 0.836).

(c) The marginal likelihood under $H_1$ (Beta(2,2) prior) is

Under $H_0: \theta = 0.5$: $p(x \mid H_0) = \binom{20}{14}(0.5)^{20} \approx 0.037$.

The Bayes factor $BF_{10} = p(x \mid H_1)/p(x \mid H_0) \approx 1.6$, providing anecdotal evidence against $H_0$ (Jeffreys scale: $1 < BF < 3$ corresponds to $\log_{10} BF < 0.5$).