The short answer
Production agent quality is a panel of metrics, not a single score. Track task completion rate, factual accuracy on retrieved facts, policy compliance, escalation precision (did the agent escalate the right tickets), latency, cost per task, human-override rate, and CSAT or downstream business KPI. Any single-number 'accuracy' metric is a red flag — it usually hides regressions on the metric that actually matters to the business.
Why single-number accuracy misleads
An agent can be 95% accurate on a benchmark and still be a disaster in production — because the 5% it misses are the high-stakes edge cases, or because accuracy on text generation does not predict downstream business impact. Production quality requires a panel of metrics that together describe the agent's behavior.
The production quality panel
Eight metrics, tracked weekly, reviewed monthly with the business owner. Every change in prompts, tools, or model must show this panel before deploy.
- ›Task completion rate: did the agent successfully complete the task it attempted?
- ›Factual accuracy: when the agent cited a fact, was the fact correct against ground truth?
- ›Policy compliance: did the agent adhere to documented policies (tone, scope, escalation rules)?
- ›Escalation precision: did the agent escalate the right tickets at the right time?
- ›Latency: p50 and p95 task duration, including tool calls.
- ›Cost per task: model + infrastructure cost, watched for unexplained drift.
- ›Human-override rate: how often did the reviewer change the agent's output, and on which fields?
- ›Downstream KPI: the business outcome the agent exists to move (DSO, CSAT, win rate, etc.).
How LLM-as-judge fits in
LLM-as-judge is a cost-effective way to score the first three metrics at scale, but it must be calibrated against human-graded samples — at least monthly. Uncalibrated judge models drift, especially after the judge's own model is upgraded.
What 'monitoring' looks like in practice
Daily automated runs of the regression suite. Weekly random sampling of production outputs scored by a judge model and reviewed by a human. Monthly business review with the metric panel. Quarterly recalibration of the judge model and the regression suite against new edge cases.
Key takeaways
- Track a panel, not a number — single-number accuracy hides the regressions that matter.
- Downstream business KPI is the metric that matters; the others explain why it moved.
- Calibrate LLM-as-judge against human-graded samples on a monthly cadence.
- Versioned regression suite + weekly sampling + monthly business review = production discipline.
Frequently asked questions
How big should the production sample size be?+
At least 50 outputs per week per workflow for human review, plus 100% automated scoring via judge model. Smaller samples produce too-noisy signals to act on.
What's an acceptable human-override rate?+
Depends on the workflow. For drafting agents (proposals, emails), 20–40% is healthy — humans should be editing. For decision agents (routing, classification), under 5% on confident outputs is the target.
Do we need an evaluation platform like Braintrust or Langfuse?+
If you have more than one agent in production, yes. Spreadsheet evals scale to one workflow; beyond that the operational tax is brutal. The 'build vs buy' framework applies to eval platforms too — most teams should buy.