Agent planning and cognitive architectures

Current agent systems rediscover old architectural questions in a new substrate: how to represent goals, select operators, maintain working state, learn from execution, detect failure, replan, and decide when to consult a person. Recent reports on context compaction, explicit skills, tool interfaces, checkpointing, and long-horizon failure diagnosis all concern the control architecture around a model.

The strongest engineering implication is to treat the agent as an observable state machine with a policy, tools, memory, and evaluators—not as a loop that merely resubmits prose to a model.

A Markov decision process is (S, A, P, R, γ): states, actions, transition probabilities, reward, and discount. A policy π(a|s) selects actions. The value of state s is

V^π(s) = E_π[Σ_{t=0}^∞ γ^t r_t | s_0=s].

When the true state is hidden, a partially observable model maintains a belief b(s). After action a and observation o, Bayesian filtering updates

b'(s') ∝ O(o|s',a) Σ_s P(s'|s,a)b(s).

Worked example: an agent believes a server is healthy with probability 0.8 and failed with 0.2. A warning appears with probability 0.1 when healthy and 0.9 when failed. The posterior failure probability is (0.9×0.2)/(0.9×0.2 + 0.1×0.8) = 0.692. The observation changes which action—retry, inspect, or escalate—has highest expected value.

Symbolic planning represents actions with preconditions and effects. Dyna adds learned experience: take real action, update value/model, then use simulated transitions for additional planning.

1. 1960 — General Problem Solver. Newell, Shaw, and Simon used means–ends analysis over symbolic operators and goals. RAND paper (opens in a new tab).
2. 1971 — STRIPS. Richard Fikes and Nils Nilsson represented actions with logical preconditions and effects for Shakey. Artificial Intelligence (opens in a new tab)90010-5).
3. 1972 — Human Problem Solving. Newell and Simon connected problem-space search with empirical study of human protocols. Princeton edition (opens in a new tab).
4. 1980 — Hearsay-II. Erman, Hayes-Roth, Lesser, and Reddy coordinated specialist knowledge sources through a shared blackboard. ACM Computing Surveys (opens in a new tab).
5. 1987 — Soar. Laird, Newell, and Rosenbloom proposed a general cognitive architecture based on problem spaces and production rules. Artificial Intelligence (opens in a new tab)90050-6).
6. 1988 — IRMA. Bratman, Israel, and Pollack separated beliefs, desires, intentions, deliberation, and means–ends reasoning. Computational Intelligence (opens in a new tab).
7. 1991 — Dyna. Richard Sutton integrated acting, model learning, and planning from simulated experience. Machine Learning Proceedings (opens in a new tab).
8. 1995 — BDI agents. Rao and Georgeff formalized belief–desire–intention systems for rational agents. Springer (opens in a new tab).

1. 2015 | Deep reinforcement learning from pixels (cross-link). See the complete DQN milestone in Machine-learning foundations.
2. 2016 | Differentiable Neural Computer (cross-link). See the complete DNC milestone in AI paradigms and knowledge representation.
3. 2016 | AlphaGo. David Silver and collaborators combined supervised policy learning, reinforcement learning, value estimation, and Monte Carlo tree search. The system defeated the European Go champion 5–0, linking classical search to learned evaluation. Go is a closed, deterministic game, so the result did not establish reliable planning in open worlds. Nature record (opens in a new tab).
4. 2017: AlphaGo Zero. David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Demis Hassabis, and collaborators combined self-play, learned policy and value estimates, and tree search without human game records. Go's closed rules and exact outcomes sharply bound the result. Primary source (opens in a new tab).
5. 2020 | MuZero. Julian Schrittwieser and collaborators learned a latent model that predicts reward, policy, and value for tree search without reconstructing every observation. It matched strong game-playing systems across Go, chess, shogi, and Atari, but remained compute-intensive and task-bounded. Nature (opens in a new tab).
6. 2023 | ReAct. Shunyu Yao and collaborators prompted language models to interleave reasoning traces with tool actions. Experiments on question answering and interactive environments showed how observations can correct a plan, while prompted reasoning traces remained neither guaranteed to be faithful nor reliable under new tools and failure modes. ICLR (opens in a new tab).
7. 2023 | Voyager. Guanzhi Wang and collaborators built a Minecraft agent with an automatic curriculum, an executable skill library, and iterative self-verification. The system accumulated reusable code skills without human intervention, but the evidence came from one game and depended on a closed language model and forgiving simulator. Primary paper (opens in a new tab).
8. 2024: AMAGO. Jake Grigsby, Linxi “Jim” Fan, and Yuke Zhu developed an in-context reinforcement-learning system for long-horizon, partially observable tasks. Benchmark evidence does not establish safe physical deployment. ICLR (opens in a new tab).

Objective: build and evaluate a small agent whose plan, state, tools, and recovery behavior are observable.

Environment: a deterministic file-free “office” simulator with tasks such as locate a record, transform a field, request approval, and submit. Provide five tools with typed inputs, including one irreversible tool that requires approval.

Procedure: implement a baseline reactive policy; then add an explicit task-state record, precondition checks, checkpointing, and a failure-recovery policy. Generate 100 tasks with known end states and inject tool errors into 20 percent. Log every state, action, observation, error, retry, and approval.

Metrics: verified end-state success, prohibited-action rate, average actions, recovery rate, human interventions, and cost. Score plans only as a diagnostic—not as the outcome.

Expected result: explicit state and recovery should improve error handling, though it may add actions and latency.

Success criterion: a statistically and practically meaningful success gain without increasing prohibited actions.

Falsification: no paired gain after equal action/token budgets, or a gain caused only by extra human intervention.

Pitfalls: using evaluator text instead of environment state, allowing the agent to see hidden answers, and changing the simulator between conditions.

Claim: explicit checkpointing improves recovery on long tasks.

• Baseline: full transcript with truncation.
• Intervention: compact checkpoint containing goal, completed steps, open dependencies, tool outputs, and next allowed actions.
• Primary metric: end-state success after a controlled interruption at step 15.
• Secondary metrics: repeated actions, stale-state errors, and recovery latency.
• Uncertainty: paired tasks across at least 100 seeds.
• Stopping rule: freeze checkpoint schema before final evaluation.
• Disconfirming result: no success difference after equal context budget, or checkpoints create more stale-state errors.

OpenAI (opens in a new tab), Anthropic (opens in a new tab), Scale Labs (opens in a new tab), NVIDIA (opens in a new tab), MIT CSAIL (opens in a new tab), Stanford CRFM (opens in a new tab), Microsoft Research (opens in a new tab), METR (opens in a new tab), and ARC Prize (opens in a new tab) for agent architecture, tools, reasoning, and evaluation. Andrew Ng's The Batch (opens in a new tab) and Lilian Weng (opens in a new tab) provide implementation-oriented synthesis.

• High confidence, 6–18 months: mature agent stacks will version state schemas, tool contracts, checkpoint policies, and evaluators alongside model versions. Disconfirming signal: those controls show little relationship to field reliability.
• Medium confidence, 12–30 months: “when to ask” will become a first-class policy optimized against uncertainty, consequence, interruption cost, and reversibility. Disconfirming signal: fixed escalation rules dominate calibrated policies in controlled field tests.

---