Reinforcement learning trains alignment auditors toward systematic investigation

What changed. A pairwise-reference reward with 50% benign calibration targets trained Haiku 4.5 to match Opus 4.6 on the authors' composite audit evaluation while keeping false positives below 1%. A held-out AuditBench checkpoint reached 28.1% detection versus an 11.5% base rate.

Technique / discovery. Multi-turn tool-using investigations, pairwise reference rewards, benign calibration targets, and strategy clustering over 81,000 transcripts.

Why it matters. Automated auditing skill is trainable, but reward design determines whether the agent learns controlled experiments or reward-hacking behavior.

Application. Pre-deployment alignment audits, behavioral anomaly discovery, red teaming, and model monitoring

Evidence maturity. Preprint (source tier A)

Identified bottleneck. Same-family LLM judging, no human validation, one policy scale, low realism, and system-prompt-planted behavior limit the conclusion.

Caveat / evidence note. Anthropic-affiliated arXiv preprint with released code and data. All four evaluation dimensions used an Opus 4.6 judge from the policy family.

Review status. Reviewed. User requested: Yes.

• Anthropic (opens in a new tab)

A reproduction tutorial is not yet available for this entry. The closest reviewed material is Safety, security, and alignment and Reliability, uncertainty, and evaluation.