Reliability, uncertainty, and evaluation

Agent evaluation is broadening from static correctness toward executable outcomes, trajectories, recovery, help-seeking, and side effects. Terminal-Bench 3.0, HORIZON, Gaia2, HiL-Bench, and METR-style long-horizon evaluation are different artifacts, but together they support the synthesis that evaluation must match the interaction structure of deployment.

Three distinctions matter: capability is whether success is possible; reliability is how consistently it occurs under stated conditions; calibration is whether stated confidence corresponds to observed frequency. None of these alone establishes safety.

For a binary outcome y∈{0,1} and predicted probability p, the Brier score is (p-y)^2. Lower is better. A perfectly calibrated set of predictions made at p=0.7 should succeed about 70 percent of the time.

Worked example: predictions (0.9, 0.7, 0.2) face outcomes (1,0,0). The Brier score is [(.9-1)^2 + (.7-0)^2 + (.2-0)^2]/3 = (0.01+0.49+0.04)/3 = 0.18. The confident wrong 0.7 prediction dominates the loss.

Selective prediction lets a system abstain. At threshold τ, coverage is the fraction acted on; selective risk is error among acted-on cases. A useful ask–act policy minimizes expected cost:

act if E[loss_act | information] < cost_ask + E[loss_after_help].

Trajectory evaluation adds structure: final success Y can be decomposed with events such as wrong goal, invalid tool call, stale observation, failed recovery, or unsafe side effect. The decomposition is diagnostic and should not replace end-state verification.

1. 1975 — MYCIN certainty factors. Shortliffe and Buchanan developed a practical uncertainty scheme for expert reasoning. Its semantics were useful but not identical to probability. Mathematical Biosciences (opens in a new tab)90047-4).
2. 1980 — Chinese Room. John Searle challenged behavioral inferences from symbol manipulation to understanding. Whatever one's philosophical position, the argument cautions against equating fluent output with grounded competence. Behavioral and Brain Sciences (opens in a new tab).
3. 1986 — System Usability Scale. John Brooke's compact questionnaire showed how standardized human-centered measurement could complement technical metrics. Original chapter copy (opens in a new tab).
4. 1988 — Bayesian networks. Judea Pearl provided a framework for reasoning under uncertainty with graphical structure. Book record (opens in a new tab).
5. 1997 — No Free Lunch. Wolpert and Macready formalized that performance gains require assumptions about the problem distribution. IEEE (opens in a new tab).
6. 1997 — Use, misuse, disuse, and abuse. Parasuraman and Riley analyzed human reliance failures around automation. Human Factors (opens in a new tab).
7. 2000 — Levels of automation. Parasuraman, Sheridan, and Wickens treated automation as choices across information acquisition, analysis, decision, and action—not one scalar. IEEE (opens in a new tab).

1. 2017 | Calibration of modern neural networks. Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Weinberger showed that accuracy gains did not imply calibrated probabilities and found temperature scaling effective on held-out in-distribution data. The method does not guarantee calibration under distribution shift. ICML (opens in a new tab).
2. 2017 | Deep ensembles. Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell trained independent probabilistic neural networks and aggregated their predictions. The simple baseline improved uncertainty and out-of-distribution behavior in tested settings, but multiplied training and inference cost and offered no guarantee under arbitrary shifts. NeurIPS (opens in a new tab).
3. 2019 | Model Cards. Margaret Mitchell and collaborators proposed structured documentation of intended use, evaluation conditions, subgroup results, and limitations. The contribution made reporting part of evaluation practice, while disclosure quality and independent verification remained outside the artifact itself. FAT* (opens in a new tab).
4. 2022 | HELM. Percy Liang and collaborators evaluated language models across scenarios and multiple metrics, treating model choice as a multidimensional decision rather than a single leaderboard score. Its coverage was intentionally broad but necessarily incomplete and vulnerable to model-version and benchmark changes. Primary paper (opens in a new tab).
5. 2025 | Measuring AI task-completion time horizons. Megan Kinniment and collaborators estimated the duration of software tasks frontier agents could complete with 50 percent success and fitted a historical growth trend. The metric made long-horizon capability more measurable, but its task mix was software-heavy and trend extrapolation does not establish deployment reliability. Primary paper (opens in a new tab).

Objective: measure whether confidence can support an ask–act policy.

Data: a public binary classification dataset split into train, calibration, and locked test sets.

Procedure: train a classifier; collect uncalibrated probabilities; fit Platt scaling or isotonic regression on the calibration set; compare Brier score, log loss, reliability diagram, and expected calibration error. Then define an abstention threshold and a cost matrix for false action, missed action, and asking a human. Plot risk versus coverage and expected cost versus threshold.

Expected result: calibration may improve probability quality without changing ranking accuracy. A selective policy should lower acted-on risk as coverage falls.

Success criterion: the calibrated policy lowers prespecified expected cost on the locked test set.

Falsification: calibration does not improve proper scoring rules or the ask–act policy fails to improve cost once human latency/error is included.

Pitfalls: calibrating on the test set, using equal-width bins with too few observations, reporting ECE without a proper score, and choosing costs after seeing results.

Claim: trajectory labels improve repair decisions beyond final pass/fail labels.

• Baseline: train or configure a repair policy using only final outcome.
• Intervention: add prespecified trajectory failure categories annotated without seeing the repair result.
• Primary metric: successful recovery on held-out failures.
• Secondary metrics: unnecessary edits, cost, and new side effects.
• Uncertainty: stratify by failure type and bootstrap task-level differences.
• Stopping rule: one locked repair evaluation after taxonomy and prompts are frozen.
• Disconfirming result: categories do not improve recovery or annotator agreement is too low for stable use.

Scale Labs (opens in a new tab), METR (opens in a new tab), Stanford CRFM (opens in a new tab), ARC Prize (opens in a new tab), UK AI Security Institute (opens in a new tab), NIST (opens in a new tab), Anthropic (opens in a new tab), OpenAI (opens in a new tab), and Hugging Face (opens in a new tab). Relevant current artifacts include Terminal-Bench 3.0 (opens in a new tab), HORIZON (opens in a new tab), and HiL-Bench (opens in a new tab).

• High confidence, 6–18 months: serious agent benchmarks will report end-state success plus trajectory or failure-mode data. Disconfirming signal: trajectory categories fail to improve prediction or repair of field failures.
• Medium confidence, 12–24 months: help-seeking, reversibility, and side-effect measures will join capability scores in procurement evaluations. Disconfirming signal: buyers and incident data show no relationship between these metrics and operational outcomes.

---