Machine-learning foundations

Frontier discussion emphasizes scale, but the recurring engineering questions remain classical: what distribution produced the data, what loss is optimized, what inductive bias is introduced, what uncertainty is measured, and whether the evaluation distribution matches deployment. Open models and efficient training widen access, while energy, memory, data rights, contamination, and evaluation validity remain binding constraints.

This topic treats supervised learning, statistical learning theory, reinforcement learning, and representation learning as related attempts to improve behavior from experience under stated assumptions.

Given samples (x_i, y_i) ~ D, learning chooses f to minimize expected risk R(f) = E_D[L(f(x), y)]. Because D is unknown, training minimizes empirical risk R̂(f). Generalization is the gap R(f) - R̂(f).

For a linear predictor ŷ = w·x with squared loss L = 1/2(ŷ-y)^2, the gradient is ∇_w L = (ŷ-y)x. With learning rate η, gradient descent updates w ← w - η(ŷ-y)x.

Worked example: let x=2, y=1, initial w=0.2, and η=0.1. Then ŷ=0.4, error is -0.6, gradient is -1.2, and the update gives w=0.32. The prediction moves from 0.4 to 0.64. The rule learns because error changes the parameter in a direction that reduces the local loss.

In sequential problems, Q-learning updates Q(s,a) ← Q(s,a) + α[r + γ max_a' Q(s',a') - Q(s,a)]. The bracketed term is a temporal-difference error: observed reward plus a bootstrapped future estimate minus the previous estimate.

1. 1960 — ADALINE and Widrow–Hoff learning. Bernard Widrow and Marcian Hoff minimized continuous error in an adaptive linear unit, an early foundation for online gradient methods. Stanford paper (opens in a new tab).
2. 1967 — Nearest-neighbor classification. Thomas Cover and Peter Hart established a durable nonparametric baseline and a bound relative to Bayes error. IEEE (opens in a new tab).
3. 1975 — Genetic algorithms. John Holland formalized adaptation through selection, recombination, and mutation. MIT Press (opens in a new tab).
4. 1984 — PAC learning. Leslie Valiant made learnability depend explicitly on sample size, accuracy, confidence, and computational feasibility. CACM (opens in a new tab).
5. 1986 — Backpropagation. David Rumelhart, Geoffrey Hinton, and Ronald Williams demonstrated gradient learning of distributed internal representations. Nature (opens in a new tab).
6. 1988–1992 — TD learning and Q-learning. Richard Sutton formalized temporal-difference prediction; Christopher Watkins and Peter Dayan established convergence conditions for Q-learning. TD learning (opens in a new tab); Q-learning (opens in a new tab).
7. 1995–2001 — SVMs, boosting, and random forests. These methods showed how margin, ensembles, and randomized trees could deliver strong generalization with different biases. SVM (opens in a new tab); AdaBoost (opens in a new tab); Random forests (opens in a new tab).
8. 1998–2006 — LeNet and deep belief nets. Convolutional learning succeeded on documents, while layer-wise pretraining helped renew interest in deeper networks. LeNet-5 (opens in a new tab); deep belief nets (opens in a new tab).
9. 2007 — Greedy layer-wise training. Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle trained deep networks one layer at a time before supervised fine-tuning. Later initialization, normalization, data scale, and architectures reduced the need for this recipe, but the work remains a key bridge to trainable deep representation hierarchies. Primary source (opens in a new tab).
10. 2009 — Learning Deep Architectures for AI. Yoshua Bengio synthesized the optimization, representation, and inductive-bias case for deep learning. This is a research monograph and synthesis, not one new controlled experiment. Primary source (opens in a new tab).
11. 2010 — Autonomous helicopter aerobatics. Pieter Abbeel, Adam Coates, and Andrew Ng combined demonstrations, learned dynamics, and control to reproduce expert aerobatic maneuvers on a physical platform. The result remained vehicle- and maneuver-specific. Primary source (opens in a new tab).

1. 2011: Single-layer unsupervised feature learning. Adam Coates, Andrew Ng, and Honglak Lee showed how preprocessing, representation size, receptive fields, and extraction choices could dominate results in shallow feature-learning pipelines. Primary source (opens in a new tab).
2. 2012 | AlexNet. Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton trained a large convolutional network on ImageNet with GPUs, rectified linear units, dropout, and data augmentation. The result connected backpropagation and convolutional inductive bias to large labeled datasets and commodity parallel hardware. Its gains depended on benchmark data, compute, and vision-specific assumptions. NeurIPS (opens in a new tab).
3. 2013: Initialization and momentum. Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton showed that initialization and momentum schedules materially affect deep-network optimization. The reported configurations do not transfer unchanged to every architecture. Primary source (opens in a new tab).
4. 2015: Deep reinforcement learning from pixels. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Demis Hassabis, and collaborators combined convolutional perception with Q-learning across Atari games. This is strong bounded-simulator evidence, not open-world reliability. Primary source (opens in a new tab).
5. 2015: Deep Learning review. Yann LeCun, Yoshua Bengio, and Geoffrey Hinton organized the methods and evidence defining the modern deep-learning program. Treat it as a scholarly review rather than a standalone experiment. Primary source (opens in a new tab).
6. 2016 | Residual networks. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun used identity shortcuts to optimize networks far deeper than earlier convolutional systems. A 152-layer model won the ILSVRC 2015 classification task, but the evidence did not remove dependence on architecture, labeled data, or compute. CVPR (opens in a new tab).
7. 2020 | Denoising diffusion probabilistic models. Jonathan Ho, Ajay Jain, and Pieter Abbeel learned to reverse a gradual noising process, connecting generative modeling to denoising score matching and nonequilibrium thermodynamics. High-quality image samples established a durable alternative to adversarial training, while iterative sampling remained slow. NeurIPS (opens in a new tab).
8. 2022 | Compute-optimal training. Jordan Hoffmann and collaborators varied model size and token count under fixed compute and reported that many large language models were undertrained. Chinchilla showed the practical value of jointly scaling parameters and data, but the fitted relationship was empirical, regime-dependent, and silent about data quality and rights. Primary paper (opens in a new tab).

Objective: compare k-nearest neighbors, logistic regression, a random forest, and a small neural network on a public tabular dataset.

Data: use a UCI dataset with a clear target and at least one feature that can define a prespecified shift. Record the exact dataset version and license.

Procedure: create one random in-distribution split and one shift split based on a range or category. Standardize features using training statistics only. Tune every model with the same validation budget. Report accuracy or AUROC, calibration error, training time, prediction time, and memory. Repeat with five fixed seeds.

Expected result: no method should dominate every metric. k-NN may be competitive in-distribution but expensive at inference; linear models may remain stable when the relationship is simple; trees may handle nonlinear tabular interactions; the neural network may need more tuning and data.

Success criterion: the experiment reveals at least one ranking change between in-distribution and shifted tests and reports uncertainty across seeds.

Falsification: if rankings and confidence intervals remain stable, the chosen shift did not expose a meaningful inductive-bias difference.

Pitfalls: preprocessing leakage, unequal tuning, hidden duplicated records, and using the test set repeatedly.

Claim: a larger or more complex model is not automatically more reliable under deployment shift.

• Baseline: the best in-distribution model.
• Intervention: select models using shift-aware validation or robust criteria.
• Primary metric: worst-group error on a held-out shifted test.
• Uncertainty: five seeds plus bootstrap intervals within each test group.
• Stopping rule: one locked final evaluation after preprocessing and hyperparameters are frozen.
• Disconfirming result: model scale or complexity predicts improved worst-group error across datasets without an offsetting cost or calibration penalty.

MIT CSAIL (opens in a new tab), Berkeley AI Research (opens in a new tab), Mila (opens in a new tab), Vector Institute (opens in a new tab), Google Research (opens in a new tab), Microsoft Research (opens in a new tab), Meta FAIR (opens in a new tab), and Apple Machine Learning Research (opens in a new tab) for learning methods, efficiency, robustness, and on-device ML. Foundational perspectives remain especially relevant from Richard Sutton (opens in a new tab), Yoshua Bengio (opens in a new tab), Geoffrey Hinton (opens in a new tab), and Yann LeCun (opens in a new tab).

• High confidence, 6–24 months: evaluation reports will more often include cost, latency, memory, calibration, and shifted-distribution results alongside average accuracy. Disconfirming signal: benchmark rankings remain stable and deployment failures show little relationship to those measures.
• Medium confidence, 12–36 months: smaller specialized or distilled models will take a larger share of repeated, constrained workloads even as frontier models improve. Disconfirming signal: total cost and reliability consistently favor frontier general models after routing and operations are included.

---