Language models and representation

Language-model development now spans base-model training, post-training, retrieval, tool use, memory, long-context management, and system-level evaluation. The current evidence in the daily tracker suggests that context compaction and retained reasoning state can materially affect a model-plus-harness result. That does not show that longer context always helps or that a specific base model improved; it shows why the evaluation unit must be declared.

Open educational implementations such as Andrej Karpathy's microgpt make the learning mechanism inspectable. Open-model ecosystems expand experimentation, but weights alone do not establish data provenance, contamination status, or reproducibility.

A language model assigns a probability to a token sequence using the chain rule:

P(x_1,…,x_T) = ∏_{t=1}^T P(x_t | x_1,…,x_{t-1}).

Training commonly minimizes negative log-likelihood, equivalent to cross-entropy for one-hot targets:

L = -Σ_t log P_θ(x_t | x_<t).

Perplexity is exp(L/T). It is the geometric average number of equally likely choices implied by the model's loss; lower is better only on the same tokenization and evaluation distribution.

Worked example: if a model assigns probabilities 0.5, 0.25, and 0.5 to the three observed next tokens, average negative log-likelihood is -(ln .5 + ln .25 + ln .5)/3 ≈ 0.924. Perplexity is e^0.924 ≈ 2.52.

Representations are vectors learned so that useful regularities become geometrically accessible. An embedding lookup maps token i to row E_i. A recurrent model updates hidden state h_t = f(W_h h_{t-1} + W_x x_t). Attention later generalized direct interactions among positions, but its pre-2010 intellectual roots include distributional semantics, statistical language modeling, recurrent networks, and differentiable representation learning.

1. 1966 — ELIZA. Joseph Weizenbaum's pattern-matching dialogue program showed that surface cues can create an impression of understanding without a grounded world model. CACM (opens in a new tab).
2. 1972 — SHRDLU. Terry Winograd linked language to a constrained blocks-world representation and action system. It achieved depth through a tiny engineered world. Stanford archive (opens in a new tab).
3. 1990 — Simple recurrent networks. Jeffrey Elman showed that recurrent hidden state could learn temporal structure from sequence prediction. Cognitive Science (opens in a new tab).
4. 1990 — Latent Semantic Analysis. Deerwester, Dumais, Furnas, Landauer, and Harshman used matrix factorization to recover latent associations in term–document data. JASIS (opens in a new tab)1097-4571(199009)41:6%3C391::AID-ASI1%3E3.0.CO;2-9).
5. 1992 — Class-based n-grams. Brown and colleagues clustered words to improve statistical language modeling and address sparse counts. Computational Linguistics (opens in a new tab).
6. 1997 — LSTM. Sepp Hochreiter and Jürgen Schmidhuber introduced gated recurrent memory to reduce vanishing-gradient problems. Neural Computation (opens in a new tab).
7. 2003 — Neural probabilistic language model. Yoshua Bengio and collaborators jointly learned word representations and next-word probabilities. JMLR (opens in a new tab).
8. 2010 — RNN language model. Tomáš Mikolov and colleagues demonstrated practical recurrent language modeling improvements in speech recognition. Interspeech (opens in a new tab).

1. 2011: Generating text with recurrent neural networks. Ilya Sutskever, James Martens, and Geoffrey Hinton demonstrated recurrent sequence models that learned character-level regularities. Generated fluency did not establish grounded understanding. Primary paper (opens in a new tab).
2. 2013 | word2vec. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean introduced efficient continuous bag-of-words and skip-gram training for large corpora. The work scaled distributional semantics into useful vector arithmetic, but a single static vector still conflated a word's senses and contexts. Google Research (opens in a new tab).
3. 2014: Sequence-to-sequence learning. Ilya Sutskever, Oriol Vinyals, and Quoc Le used one LSTM to encode an input sequence and another to decode an output sequence. Results remained bounded by training data, vocabulary, decoding, and task distribution. NeurIPS (opens in a new tab).
4. 2016: Deep Speech 2. Dario Amodei, Adam Coates, Linxi Fan, Andrew Ng, and Baidu collaborators developed an end-to-end speech-recognition system for English and Mandarin. Speech recognition is not equivalent to general language understanding. Primary source (opens in a new tab).
5. 2017 | Transformer. Ashish Vaswani and collaborators replaced recurrence with self-attention in an encoder-decoder translation system. Parallel training and direct token-to-token interactions built on learned representations and attention, while standard attention introduced quadratic sequence cost and did not solve grounding or factuality. NeurIPS (opens in a new tab).
6. 2019: GPT-2. Alec Radford, Jeffrey Wu, Dario Amodei, Ilya Sutskever, and OpenAI collaborators reported transferable zero-shot behavior from scaled autoregressive language modeling. The first-party report did not establish dependable general reasoning. Technical report (opens in a new tab).
7. 2019 | BERT. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova pretrained bidirectional transformer representations with masked-token and sentence objectives, then fine-tuned them across language tasks. This consolidated transfer learning for language, but the masking objective differed from downstream generation and context remained bounded. NAACL (opens in a new tab).
8. 2020 | GPT-3 and in-context learning. Tom Brown, Ilya Sutskever, and collaborators scaled autoregressive language modeling to 175 billion parameters and evaluated zero-, one-, and few-shot prompting without gradient updates. The experiments made the prompt a task interface, but contamination, cost, bias, and unstable reliability limited broad claims. NeurIPS (opens in a new tab).
9. 2022 | InstructGPT. Long Ouyang and collaborators combined supervised demonstrations, a learned preference model, and reinforcement learning to improve instruction following. The work built directly on human feedback and preference learning, but inherited dependence on labeler populations, reward-model coverage, and robustness against reward gaming. Primary paper (opens in a new tab).

Objective: reproduce the transition from count-based to learned distributed representations on a small, licensed corpus.

Data: a public-domain text corpus small enough to train on a laptop. Freeze a train/validation/test split by document, not by overlapping random windows.

Procedure:

1. Build a character or word tokenizer and record vocabulary and unknown-token policy.
2. Train a smoothed trigram model.
3. Train a small recurrent model or tiny decoder model with a similar parameter budget.
4. Compare held-out negative log-likelihood, perplexity, training compute, inference latency, and samples.
5. Create a rare-context subset and a repeated-phrase subset. Compare errors separately.
6. Check whether any generated passage exactly matches long training spans.

Expected result: the neural model should share statistical strength across related contexts and often improve held-out perplexity; the n-gram should be easier to audit and may remain strong for repeated local phrases.

Success criterion: report a reproducible test-set difference with fixed seeds and a qualitative error analysis that explains at least two failure types.

Falsification: if the neural model does not improve held-out performance after a fair tuning budget, complexity was not justified at this data scale.

Pitfalls: comparing different tokenizations by raw perplexity, splitting overlapping windows across train and test, and interpreting fluent samples as semantic understanding.

Claim: context-management policy can change long-horizon task success independently of base-model choice.

• Baseline: fixed context window with truncation.
• Interventions: rolling summary, retrieval of prior state, and explicit checkpoint records.
• Task: a deterministic multi-step text environment with hidden dependencies across at least 30 actions.
• Primary metric: verified end-state success; secondary metrics are token cost, lost-fact rate, and recovery after injected distraction.
• Uncertainty: at least 100 task instances with paired seeds and bootstrap confidence intervals.
• Stopping rule: freeze policies before the final test.
• Disconfirming result: no meaningful paired success difference, or gains disappear once token budget and latency are matched.

OpenAI Research (opens in a new tab), Anthropic Research (opens in a new tab), Google DeepMind (opens in a new tab), Meta FAIR (opens in a new tab), Mistral AI (opens in a new tab), Cohere Research (opens in a new tab), EleutherAI (opens in a new tab), Hugging Face (opens in a new tab), and Allen Institute for AI (opens in a new tab). For implementation-level interpretation: Andrej Karpathy (opens in a new tab), Sebastian Raschka (opens in a new tab), Nathan Lambert (opens in a new tab), and Lilian Weng (opens in a new tab).

• High confidence, 6–18 months: public evaluations will report more model-plus-harness configurations rather than one model score. Disconfirming signal: rankings remain invariant across context, memory, and tool-interface choices.
• Medium confidence, 12–36 months: smaller open and specialized models will increasingly serve as components in routed systems rather than replacements for every frontier task. Disconfirming signal: routing overhead and maintenance erase their cost or privacy advantages.

---