Agent Evaluation, Replay & Backtesting

Why evaluation is where 2026 buying converges

In the May 2026 r/CIO thread that captured the shift, a practitioner wrote: 'we aren't able to rigorously generate and test all production scenarios before launching a new AI agent into customer support.' The idea of backtesting the agent on production data before launch suddenly felt obvious. That reaction is the signal.

The buying pattern that follows is consistent. Buyers no longer trust vendor demos or aggregate accuracy numbers. They want a release gate that runs their own tickets through their own tools with their own approval logic, and blocks the promotion if regressions appear.

The four evaluation layers

A certification harness has four cooperating layers:

• Golden scenarios — curated, redacted historical cases with expected state changes, expected non-actions, and maximum allowed disclosure.
• Adversarial variants — perturbations covering injection, exfiltration, ambiguous instructions, and edge cases the golden set doesn't naturally contain.
• State-diff scoring — evaluates what changed in the system of record, not just whether the reply 'sounds right'. Correctness, containment, escalation quality, latency, and privacy each get a dimension.
• Release gate — wired into CI so every candidate blocks on regression, with time-boxed overrides and named-owner logs.

The failure taxonomy is the deliverable

Aggregate pass rates hide the useful signal. The failure taxonomy — hallucinated lookup, wrong tool, wrong parameter, wrong delegation, privacy over-disclosure, missed escalation — is what the improvement backlog is written against.

In production, every incident ticket auto-seeds a new golden scenario. Every override of a failing gate becomes a review item. The harness becomes an institutional memory of failure modes.

Drift is a first-class metric

Weekly drift monitors compare production behavior against the golden set. Divergence over threshold auto-creates a triage ticket. This is how you find out an agent broke before customers do — because the harness catches drift the day it happens, not the week the churn shows up.

8–12 week implementation sequence

• Weeks 1–3: historical case pull + redaction; golden set v1 curated with process owner.
• Weeks 4–6: state-diff scorer + adversarial variant generator.
• Weeks 7–9: CI release gate + failure-taxonomy dashboard.
• Weeks 10–12: drift monitor + handoff + first regression-catch retrospective.

Failure modes to avoid

• Overfitting to past logs — inject out-of-distribution scenarios monthly.
• Weak business acceptance criteria — define success per workflow with the process owner, not by generic accuracy.
• Skipping the privacy sub-suite — regulated workflows need it as a release-blocking dimension.
• Manual override of failing gates without a logged reason — the audit trail is a governance requirement, not a courtesy.