Optimization and Differentiation §
Optimization and differentiation provide the computational machinery for finding minima of functions: closed-form solvers for linear objectives, iterative refinement for fixed points, gradient-based methods for continuous differentiable landscapes, and parametric models (neural networks) that learn representations from data. They are relevant whenever a loss must be minimized, a derivative computed, or a model fitted to observations.
Utility §
- It finds the parameters of a model that best explain observed data.
- It computes exact derivatives of arbitrary compositions of elementary operations.
- It provides convergence guarantees for iterative solvers.
- It scales to millions of parameters via stochastic and parallel methods.
Details
Optimization sits at the boundary between mathematics and computation: the objective function defines the problem, and the algorithm defines the procedure. The four subtopics below are ordered from the simplest (a closed-form linear solve) to the most complex (a parametric model with learned representations trained by stochastic gradient descent). Each builds on the previous: regression motivates the need for iterative methods, fixed-point iteration provides the convergence framework, gradient descent is the workhorse algorithm, and neural networks are the dominant application.
History §
- 1810: Gauss and Legendre independently develop the method of least squares for regression.
- 1952: Rosenbrock introduces a non-convex test function that remains the standard benchmark for gradient-based optimizers.
- 1986: Rumelhart, Hinton, and Williams popularize backpropagation for training multilayer networks.
- 2015: Kingma and Ba introduce Adam, the adaptive optimizer that becomes the default for deep learning.
Classes §
| Page | Core object | Key property | Boundary |
|---|---|---|---|
| Regression | Parametric model fitted to observed data by minimizing a loss | Closed-form solution for linear models; overfitting controlled by train/validation split | Does not specify the iterative solver when no closed form exists |
| Fixed-point iteration | Map g and its fixed point x* = g(x*) | Linear convergence under contraction; quadratic for Newton's method | Does not require the map to be a gradient; general framework for iterative refinement |
| Gradient descent | Differentiable loss f and its gradient ∇f | Convergence rate governed by smoothness L and strong convexity μ; stochastic variants scale to large data | Requires differentiability; does not handle discrete or non-smooth objectives directly |
| Neural networks | Parametric model with learned representations, trained by gradient descent | Architecture, loss, data paradigm, and compute are orthogonal design axes | Assumes a differentiable composition of elementary operations; does not cover combinatorial search |
Authoritative sources