Reproduction tutorial · September 18, 2026
Evaluator-independence protocol
These tutorials test the mechanism behind each paper at a practical scale. They do not reproduce frontier-scale headline results unless explicitly stated.
Source of this tutorial
This tutorial tests the mechanism behind reviewed development 03 in the immutable briefing of September 18, 2026.
Scope and limits
- Reproduction level
- simulated organizational evaluation protocol
- Claim under test
- whether versioned internal access improves material incident detection without constraining independent reporting compared with outside-only review.
- Excluded claim
- this tutorial does not test the Anthropic–Accenture partnership or establish that embedded evaluation has improved real-world safety outcomes.
- Reader, effort, and cost
- use a simulated organization, a fixed incident corpus, two review conditions, and blinded adjudication.
Canonical resources
- Partnership announcement: https://www.anthropic.com/news/accenture-embedded-evaluation (opens in a new tab)
Project plan
| Step | Stage | Effort |
|---|---|---|
| 01 | Freeze rights and evidence | Before review |
| 02 | Run outside-only review | First condition |
| 03 | Run embedded review | Second condition |
| 04 | Blind adjudication and comparison | After both reviews |
01Freeze rights and evidence(Before review)
Use a simulated organization and a fixed incident corpus. Predefine access, funding, conflict, escalation, and publication rights.
02Run outside-only review(First condition)
Give the outside-only condition the fixed evidence available under that review design and preserve its complete findings.
03Run embedded review(Second condition)
Give the embedded condition versioned training logs, decision records, and staff interviews while keeping the incident corpus fixed.
04Blind adjudication and comparison(After both reviews)
Blind adjudicators to condition. Score incident recall, severity calibration, time to finding, unsupported allegation rate, and publication completeness. Falsify the claimed value if inside access adds no material detection or constrains reporting.
Controls, failure modes, and safety
Controlled variables
incident corpus, scoring rubric, adjudicator blinding, review conditions, access rights, and publication rights.
Failure modes
condition leakage to adjudicators, changing the incident corpus, unequal scoring rules, hidden access constraints, or suppressing material findings.
Safety
use a simulated organization and fixed incident corpus; do not expose private organizational records or real confidential incidents.