Observability and evaluation: an Agent Certification Harness for a pre-production AI support agent

The closed loop, applied

1. Capture

   Ingested historical tickets, transcripts, internal notes, tags, priorities, SLA state, resolution codes, escalation history, approvals, related Salesforce metadata, KB articles, and ticket state transitions from Zendesk, Jira Service Management, Salesforce, Confluence, SharePoint, Slack, and Snowflake — enough to replay each ticket as if the candidate agent were handling it from the start.

2. Normalize

   PII redaction and tokenization replaced names, emails, account IDs, IPs, and logs with stable synthetic tokens so the same entity stayed consistent across a scenario. Scenarios were grouped into task families with expected state changes, prohibited state changes, escalation expectations, allowed tools, prohibited disclosures, and KB/policy context.

3. Memory

   Versioned KB and policy snapshots from day one so scenarios were reproducible against a known knowledge state. Curated golden scenario sets, weighted by business risk (billing, SLA, privacy, security) rather than ticket frequency, plus an adversarial and edge-case generator.

4. Reason

   Agent Adapter Layer ran multiple candidate agent versions — prompt, retrieval, model, tool-schema, and orchestration variants — against the same scenarios through a Tool Sandbox and Mock APIs. No production writes: proposed state changes were captured as structured intents, not executed.

5. Act

   Evaluation Engine scored each replay across state-diff, privacy and policy, escalation, latency and cost, and regression. A Release Gate produced Pass, Fail, or Manual Review, with hard-fail conditions (unauthorized write, PII leak, prompt-injection compliance, fabricated policy citation, disallowed ticket closure) treated as automatic blockers.

6. Evaluate

   Certification report joined pass rate by task family and risk tier, hard-fail count, privacy test pass rate, escalation recall, and regression pass rate against thresholds. Production drift monitoring fed failures back into the scenario backlog on a monthly certification cadence, and structured failure taxonomies routed retrieval, tool-schema, escalation, and privacy fixes to the right owner.