Gaussian processes §

Why PSD is the right condition. A covariance matrix $K$ of any random vector must satisfy $\mathrm{Var}\big(\sum_i c_i X_i\big) = \sum_{i,j} c_i c_j K_{ij} \geq 0$, because a variance is never negative. So PSD is not an extra assumption — it is precisely the requirement that $k$ can be the covariance of some Gaussian vector at every finite set of points. Conversely, if $k$ is PSD, the Kolmogorov extension theorem guarantees a Gaussian process with that kernel.

Worked check — a kernel that passes. Take $k(s,t) = \min(s,t)$ on $[0,\infty)$ (the Brownian kernel). For ordered points $0 \le t_1 < \cdots < t_n$, the matrix $K_{ij} = \min(t_i,t_j)$ is exactly the covariance matrix of the vector $(W_{t_1},\dots,W_{t_n})$ of a Brownian motion, so it is PSD by construction. Concretely, for two points $s<t$, $K = \begin{pmatrix} s & s \\ s & t \end{pmatrix}$ has determinant $st - s^2 = s(t-s) \geq 0$ and positive trace, hence PSD.

Worked check — a candidate that fails. Consider the tempting candidate $k(s,t) = s^2 + t^2$. At the two points $s=1, t=2$ it gives the matrix

whose determinant is $2\cdot 8 - 5^2 = 16 - 25 = -9 < 0$. A $2\times 2$ symmetric matrix with negative determinant has a negative eigenvalue, so it is not PSD. Indeed, with $c = (-1.77,\, 1)$ one gets $c^{\top} K c \approx -3.4 < 0$. The candidate $s^2+t^2$ is therefore not a valid kernel: it cannot be the covariance of any random process. (The fix is to use a product $s\,t$, which is rank-one and PSD, since $\sum_{i,j} c_i c_j s_i t_j = \big(\sum_i c_i s_i\big)^2 \geq 0$.)

The global idea. For an arbitrary process, knowing the law of $(X_{t_1}, X_{t_2})$ tells you nothing about the law of $(X_{t_1}, X_{t_2}, X_{t_3})$; you must specify each finite-dimensional distribution separately, and they must all be mutually consistent. For a Gaussian process this burden vanishes: the joint law of any finite subfamily is a multivariate normal, and a multivariate normal $\mathcal{N}(\mu, \Sigma)$ has no free parameters beyond $\mu$ and $\Sigma$. So once you fix $m$ and $k$, every finite-dimensional distribution is forced, and consistency is automatic.

Concrete example. Suppose $T = \{1, 2, 3\}$. To fully specify a general process on these three points you would need: three one-dimensional laws, three two-dimensional joint laws, and one three-dimensional joint law, all consistent with each other. To specify a Gaussian process on the same three points you need only $3$ numbers (the means $m(1),m(2),m(3)$) and a $3\times 3$ symmetric PSD matrix (the kernel values $k(i,j)$). That is the entire specification.

The payoff. Because the whole law is pinned down by $m$ and $k$, every question about the process — its conditional distributions, its sample-path smoothness, its predictions — reduces to manipulating the single function $k$. The next two sections are the two most important consequences: conditioning (GP regression) and regularity.

Reading smoothness off the decay. The figure below plots $k(0,t)$ (correlation with the origin) against distance $t$ for three kernels with the same $\ell = 0.5$. The squared-exponential curve leaves the origin flat (its derivative at $t=0$ is $0$), the Matérn curve leaves with a corner, and the Ornstein–Uhlenbeck curve falls off as a pure exponential. That difference in local shape is exactly what shows up in the sample paths: flat departure gives smooth paths, a corner gives continuous-but-rough paths.

The Matérn family as a dial. The Matérn kernel interpolates between the two extremes: for $\nu = 1/2$ it reduces exactly to the Ornstein–Uhlenbeck kernel $\sigma^2 e^{-|s-t|/\ell}$, for $\nu = 3/2$ to $\sigma^2(1 + |s-t|/\ell)\, e^{-|s-t|/\ell}$, and as $\nu \to \infty$ it approaches the squared-exponential kernel. The parameter $\nu$ controls the number of continuous derivatives of the sample paths (see the regularity section).

The global idea. Conditioning a multivariate normal is pure linear algebra: the conditional distribution of one component given the others is Gaussian, with mean and covariance given by block-matrix formulas. Because a GP is just a (possibly infinite) multivariate normal, conditioning the function $f$ on its values at the observed points is the same computation, and the answer is again a GP — the posterior GP — whose mean and kernel are the two formulas above. The posterior mean is a weighted average of the observations (the weights are $k_*^{\top}K^{-1}$), and the posterior variance is the prior variance $k(t_0,t_0)$ minus the information the observations carry about $t_0$.

Why the variance shrinks near the data. The term $k_*^{\top}K^{-1}k_*$ measures how correlated $t_0$ is with the observed points. If $t_0$ coincides with an observed point, $k_*$ is a row of $K$ and the variance becomes exactly $0$ (noiseless case): you already know the value there. Far from the data, $k_*$ is small and the variance returns to the prior $k(t_0,t_0)$. This “tighten around the data, relax away from it” behavior is the whole point of the method.

Numeric example (Brownian kernel). Let $m \equiv 0$ and $k(s,t) = \min(s,t)$. Observe $y_1 = 1$ at $t_1 = 1$ and $y_2 = 3$ at $t_2 = 4$, and predict at $t_0 = 2$.

Then $K^{-1}y = \tfrac{1}{3}(1, 2)^{\top}$, so

So the model predicts $f(2) \approx 1.67$ with standard deviation $\sqrt{1/3} \approx 0.58$. As a sanity check, predicting at the observed point $t_0 = 1$ gives $k_* = (1,1)^{\top}$ and $\sigma_*^2(1) = 1 - (1,1)\cdot\tfrac{1}{3}(3,0)^{\top} = 1 - 1 = 0$, exactly as it must.

The interactive simulation below shows this in action: a hidden smooth function is sampled from a GP, a few points are observed, and the posterior mean with its $\pm 2\sigma_*$ confidence band is drawn. Add observations to watch the band tighten around the data and relax between them.

The global idea. The criterion says: if the expected size of an increment over a small gap $|t-s|$ is smaller than the gap itself (raised to a power $> 1$), the paths must be continuous. For a Gaussian process the increment $X_t - X_s$ is Gaussian, so its typical size is its standard deviation $\sqrt{\mathrm{Var}(X_t-X_s)}$. The question “are the paths continuous?” becomes “does $\mathrm{Var}(X_t - X_s)$ go to zero faster than $|t-s|$?”

Concrete examples, from the kernel table.

• Brownian motion ($k(s,t)=\min(s,t)$): for $s < t$, $\mathrm{Var}(W_t - W_s) = t - s = |t-s|$. This is $|t-s|^{1+0}$, right at the borderline — the criterion gives continuity but no differentiability. Brownian paths are continuous yet nowhere differentiable.
• Squared-exponential ($k(s,t)=e^{-(s-t)^2/2\ell^2}$): near $s=t$, $1 - k(s,t) \approx (s-t)^2/2\ell^2$, so $\mathrm{Var}(X_t-X_s) \approx C\,|t-s|^{2}$ = $|t-s|^{1+1}$ with $\beta = 1$. The paths are not just continuous but infinitely differentiable.
• Matérn ($\nu$): near $s=t$, $1 - k(s,t) \sim C\,|t-s|^{2\nu}$. Continuity needs $2\nu > 1$ (i.e. $\nu > 1/2$); differentiability needs $2\nu > 2$ (i.e. $\nu > 1$). So the $\nu = 1/2$ Matérn (= Ornstein–Uhlenbeck) has continuous but nowhere-differentiable paths, while $\nu = 3/2$ has once-differentiable paths.

The figure below shows one sample path of a smooth GP (squared-exponential kernel): a single continuous, differentiable random curve.

A noiseless Gaussian process has zero mean and the squared-exponential kernel $k(s,t) = e^{-(s-t)^2/2}$ (so $\ell = 1$). You observe $f(0) = 1$ and $f(2) = 1$.

1. Write down the $2\times 2$ kernel matrix $K$ and the vector $k_*$ for predicting at $t_0 = 1$.
2. Explain, without computing, whether the posterior variance at $t_0 = 1$ is larger or smaller than the posterior variance at $t_0 = 10$, and why.