Learning Paradigms and Models §

Step 1 — What one layer computes. A transformer layer applies two sub-blocks, each wrapped in a residual connection: multi-head self-attention, where tokens exchange information with each other, followed by a position-wise feed-forward MLP, where each token is transformed independently. Layer normalization stabilizes the activations before each sub-block. Stacking $L$ such layers, with positional information added at the input, gives the full model.

Step 2 — A concrete trace: one token through one layer. Take a sequence of three tokens with embeddings $x_1 = [1, 0]$, $x_2 = [0, 1]$, $x_3 = [1, 1]$, and a single attention head with identity projections ($W_Q = W_K = W_V = I$, $d = 2$). For position 1:

1. Scores: $q_1 \cdot k_j = x_1 \cdot x_j$ gives $[1, 0, 1]$.
2. Softmax: $\text{softmax}([1, 0, 1]) \approx [0.422, 0.155, 0.422]$.
3. Attention output: $0.422\,x_1 + 0.155\,x_2 + 0.422\,x_3 = [0.845, 0.538]$ — a blend of all three value vectors, weighted by relevance.
4. Residual: $x_1 + \text{output} = [1.845, 0.538]$ — the original token enriched with information from the other tokens.

The feed-forward sub-block then transforms this vector independently, and the next layer repeats the process on the whole sequence.

Reference — the blocks at a glance.

Inputs. One-hot encoding: a categorical variable with $V$ classes as a $V$-dimensional vector with a single 1; wasteful for large vocabularies. Embeddings: a learned dense vector $e_c \in \mathbb{R}^d$ per token; the table is a $V \times d$ matrix and lookup is a row selection; co-occurring tokens end up with nearby vectors. Tokenization (BPE): iteratively merge the most frequent adjacent character pairs, balancing vocabulary size against sequence length. Positional encoding: transformers have no inherent order, so a position signal is added — sinusoidal $PE(pos, 2i) = \sin(pos / 10000^{2i/d})$ (fixed, extrapolates) or learned (flexible, limited to training length).

Outputs. Sigmoid: $\sigma(z) = 1/(1+e^{-z})$ maps a scalar to $(0,1)$; used for binary classification and LSTM gates. Softmax: $\text{softmax}(z)_i = e^{z_i} / \sum_j e^{z_j}$ maps a vector to a probability distribution; used for multiclass heads and language-model outputs. Pooling: max preserves salient features, min preserves the weakest signal, average smooths; used in CNNs to reduce spatial resolution and to produce fixed-size sentence representations.

Blocks. MLP: $f(x) = W_L \sigma(\cdots \sigma(W_1 x + b_1)\cdots)$; the transformer’s position-wise FFN is a 2-layer MLP with $d_{ff} \approx 4d$. ReLU: $\max(0, x)$; GELU and SiLU are smooth variants that train better. CNN: $y_i = \sum_j W_{i-j} x_j$ with shared weights — translation invariance plus local connectivity builds a hierarchy of spatial features. RNN / LSTM: recurrent state $h_t = f(h_{t-1}, x_t)$; the LSTM’s cell state $c_t = f_t \odot c_{t-1} + i_t \odot \tilde{c}_t$, $h_t = o_t \odot \tanh(c_t)$ with input/forget/output gates lets it remember over long distances. Residual connections: $y = F(x) + x$ let gradients reach early layers, enabling very deep stacks. Self-attention: $\text{Attn}(Q, K, V) = \text{softmax}(QK^T / \sqrt{d_k})\,V$; multi-head attention runs $h$ heads in parallel and concatenates them. KV cache: during autoregressive generation, cached keys/values cut per-token cost from $O(t^2)$ to $O(t)$.

Source: Attention Is All You Need introduces the layer traced above.

Step 1 — What a loss does. The loss compresses “how wrong the model is on this example” into one differentiable number. Gradient descent then follows the slope of the average loss downhill. Two consequences follow: the loss must be differentiable (or subdifferentiable) in the model’s outputs, and its shape matters — a convex loss has a single basin, while a deep network’s loss surface is non-convex, full of local minima and saddle points that SGD nevertheless navigates well in practice.

Step 2 — A concrete example: cross-entropy for a 3-class prediction. A classifier outputs logits $z = [1.0, 2.0, 0.5]$ for three classes; the true label is class 2. Softmax gives $\hat{y} = [0.186, 0.506, 0.309]$. The cross-entropy against the one-hot target $y = [0, 1, 0]$ is $\mathcal{L} = -\log(0.506) \approx 0.681$. The gradient with respect to the logits is $\hat{y} - y = [0.186, -0.494, 0.309]$: push class 2 up, push the others down, in proportion to how wrong each was.

Reference — the losses in common use.

Mean squared error (MSE): $\mathcal{L} = \frac{1}{n}\sum_i (\hat{y}_i - y_i)^2$. Used for regression. The gradient is proportional to the residual, so large errors dominate the update; sensitive to outliers.

Cross-entropy: $\mathcal{L} = -\sum_i \sum_c y_{i,c} \log \hat{y}_{i,c}$. Used for classification; the negative log-likelihood of the true label. With softmax the gradient is $\hat{y} - y$, which vanishes when correct and grows when wrong.

Perplexity: $PPL = \exp(\mathcal{L}_{CE})$. The geometric mean number of choices the model “considers” per token; used to compare language models.

KL divergence: $D_{KL}(P \| Q) = \sum_x P(x) \log \frac{P(x)}{Q(x)}$. The information lost when $Q$ approximates $P$; used in VAEs, distillation, and GAN variants.

Weight decay (L2 regularization): adds $\lambda \|\theta\|^2$ to the loss, shrinking weights toward zero. AdamW applies the decay directly to parameters (“decoupled”).

Knowledge distillation: the student $f_\theta$ matches both hard labels and the teacher’s softened outputs: $\mathcal{L} = \alpha \cdot \text{CE}(\hat{y}, y) + (1-\alpha) \cdot T^2 \cdot D_{KL}(p_\phi / T \| p_\theta / T)$. The temperature $T$ softens the teacher’s distribution so the student learns its “dark knowledge” (relative probabilities of incorrect classes).

Source: Distilling the Knowledge in a Neural Network introduces the distillation loss and the temperature scaling.

Step 1 — The paradigm is the signal. Before choosing an architecture or a loss, ask what the environment gives you. If you only have raw data, the model must invent its own supervision (structure: clusters, axes of variance, compressed codes, generative fidelity). If you have labels, the loss can compare the prediction to the truth directly. If an agent must act and only receives delayed rewards, the model must trade off exploration and long-term return. Everything downstream — loss, architecture, evaluation — follows from this choice.

Step 2 — Unsupervised learning. Each method targets a different notion of “structure”.

Clustering (k-means): partition $n$ points into $k$ groups by minimizing within-cluster variance:

The algorithm alternates between assigning each point to the nearest centroid and recomputing centroids as cluster means; it converges to a local optimum in finite steps. Try it interactively:

PCA: find the orthonormal basis that maximizes projected variance. Eigendecompose the covariance matrix $\Sigma = U \Lambda U^T$; the top-$k$ eigenvectors span the maximum-variance subspace. Used for dimensionality reduction, denoising, and visualization; it is the linear special case of an autoencoder.

Autoencoders: a network $x \to z \to \hat{x}$ with a bottleneck $z$ learns a compressed representation by minimizing reconstruction error $\|x - \hat{x}\|^2$. Being nonlinear, it captures manifold structure that PCA misses; variational autoencoders add a probabilistic prior on $z$ for generative modeling. (GANs and diffusion models push the same generative idea further: GANs train a generator against a discriminator in a minimax game, and diffusion models learn to reverse a gradual noising process.)

Step 3 — Supervised learning. Given a dataset $\{(x_i, y_i)\}_{i=1}^{n}$, the model $f_\theta$ is trained to minimize a loss $\mathcal{L}(f_\theta(x_i), y_i)$ by gradient descent.

Regression: the target $y$ is continuous; the standard loss is mean squared error (see the Loss section above). Applications: price prediction, demand forecasting, sensor calibration.

Classification: the target $y$ is a discrete label; the loss is categorical cross-entropy (see the Loss section above), with a sigmoid head for the binary case and a softmax head for the multiclass case. Applications: image labeling, spam detection, medical diagnosis.

Sequence labeling: one label per input token — named-entity recognition, part-of-speech tagging, speech recognition — typically an RNN or transformer with a per-position classification head.

Step 4 — Reinforcement learning. An agent in a Markov Decision Process $(S, A, P, R, \gamma)$ learns a policy $\pi(a|s)$ to maximize expected cumulative reward $V^\pi(s) = \mathbb{E}[\sum_t \gamma^t R(s_t, a_t)]$. The Bellman equation $V^\pi(s) = \sum_a \pi(a|s) \sum_{s'} P(s'|s,a)[R(s,a,s') + \gamma V^\pi(s')]$ links a state’s value to its successors. Policy gradient (REINFORCE) directly ascends $\nabla J(\theta) = \mathbb{E}[\nabla \log \pi_\theta(a|s) \cdot G_t]$; actor-critic adds a value function to reduce variance. When the agent must also plan, tree search over the state graph is used — see Graphs for the underlying search algorithms (best-first search, MCTS-style playouts, A*).

Source: Reinforcement Learning: An Introduction covers MDPs, Bellman equations, and policy gradients in depth.

Step 1 — The training loop. One iteration is: (1) draw a mini-batch of examples; (2) forward pass — apply the network to the batch and evaluate the loss; (3) backward pass — propagate the loss gradient back through the computation graph to every parameter; (4) update — move each parameter a small step against its gradient. Repeat until the loss stops improving. The backward pass is exactly the chain rule applied in reverse topological order over the graph recorded during the forward pass; the formulas and the automatic-differentiation machinery live in the Optimization and differentiation topic linked above.

Step 2 — Batching and stochastic gradient descent. Computing the gradient over the whole dataset (“full batch”) is exact but expensive; computing it over one example is noisy. The mini-batch (typically 32–4096 examples) sits in the sweet spot: gradients are averaged over enough examples to be informative, and the batch is large enough to use GPU matrix multiplication efficiently. The noise of small batches is not purely a cost — it helps escape sharp minima.

Step 3 — Learning-rate schedules. The step size is the single most important hyperparameter. Schedules shape it over time: step decay (drop by 10× at milestones), cosine decay (smoothly anneal to zero), warmup (ramp up over the first few hundred steps, essential for large transformers, whose early training is unstable). Adaptive optimizers (Adam, AdamW) give each parameter its own effective step size based on gradient statistics.

Step 4 — Initialization. The first forward pass must neither saturate nonlinearities nor explode activations. Xavier initialization scales weights by $1/\sqrt{n_{in}}$; He initialization by $\sqrt{2/n_{in}}$ for ReLU networks. Bad initialization shows up immediately: the loss does not decrease in the first hundred steps.

Step 5 — Training at scale. A single dense layer is a matrix product, and GPUs execute thousands of such products in parallel, which is why deep learning took off with GPU hardware. When a model no longer fits on one device, the training loop itself is distributed: data parallelism replicates the model and splits the batch, pipeline parallelism splits the layers across devices, and tensor parallelism splits individual weight matrices. These are distributed-systems problems (see the Distributed systems topic) layered on top of the same loop.

Source: Deep Learning, chapter 8 (Optimization for Training Deep Models) covers batching, learning rates, and initialization.

Step 1 — The split. Hold data out in three roles. The training set fits the parameters $\theta$. The validation set is untouched by gradient descent but is looked at: it selects hyperparameters, sets the learning-rate schedule, and decides when to stop. The test set is touched exactly once, at the end, to report the final number — every peek at it is a form of fitting.

Step 2 — Overfitting and underfitting. The generalization gap is validation performance minus training performance. Underfitting: both losses are high — the model lacks capacity or training time; add capacity, train longer, or improve features. Overfitting: training loss keeps dropping while validation loss flattens and rises — the model is memorizing noise; add data, add regularization (weight decay, dropout), or stop earlier. The learning-curve diagram above shows the canonical signature: the two curves diverge after the early-stopping point.

Step 3 — Model selection. Given the validation curve, choose the checkpoint with the lowest validation loss (early stopping). For hyperparameters (learning rate, depth, weight decay $\lambda$), run the same training with several values and pick by validation performance — never by training performance, which always favors the most overfit model. When data is scarce, k-fold cross-validation rotates the validation role across $k$ splits and averages, giving a lower-variance estimate at $k$ times the training cost.

Step 4 — Reading curves in practice. Noisy validation loss is normal; compare smoothed curves. A validation loss that plateaus early while training loss falls slowly suggests the learning rate is too small or the model too small. A validation loss that spikes suggests the learning rate is too large. Curves are the model’s vital signs: most debugging in deep learning is curve reading.

Source: Deep Learning, chapter 5 (Machine Learning Basics) formalizes the generalization gap, cross-validation, and the bias-variance trade-off.

(a) A single transformer layer contains:

• Multi-head self-attention: four $d \times d$ weight matrices ($W_Q, W_K, W_V, W_O$): $4 \times 512^2 = 1\,048\,576$ parameters.
• Position-wise feedforward: two matrices ($W_1 \in \mathbb{R}^{d \times d_{ff}}$, $W_2 \in \mathbb{R}^{d_{ff} \times d}$): $2 \times 512 \times 2048 = 2\,097\,152$ parameters.
• Two layer norms: each has a scale and a bias vector of length $d$: $2 \times 2 \times 512 = 2\,048$ parameters.

Total: $1\,048\,576 + 2\,097\,152 + 2\,048 = 3\,147\,776$ parameters per layer.

(b) Perplexity is defined as $PPL = \exp(\mathcal{L}_{CE})$, so the cross-entropy loss per token is $\mathcal{L}_{CE} = \ln(8) \approx 2.079$ nats.

(c) The combined distillation loss is:

Substituting $\alpha = 0.5$ and $T = 4$:

The coefficient $8$ multiplying the KL term is the combined product $(1-\alpha) \cdot T^2 = 0.5 \times 16 = 8$ — the mixing weight times the squared temperature, not $T^2$ alone. The temperature softens the teacher’s distribution (which shrinks the KL term’s magnitude), the $T^2$ factor compensates for that softening, and the $(1-\alpha)$ weight balances the hard-label and soft-label contributions equally.