Reproduction tutorial · September 10, 2026
Research-agent throughput under a blinded crossover
These tutorials test the mechanism behind each paper at a practical scale. They do not reproduce frontier-scale headline results unless explicitly stated.
Source of this tutorial
This tutorial tests the mechanism behind reviewed development 02 in the immutable briefing of September 10, 2026.
Scope and limits
- Reproduction level
- partial mechanism reproduction at team scale
- Claim under test
- whether agent assistance raises accepted-result throughput per human workday, rather than raising started or running work.
- Excluded claim
- this tutorial does not reproduce the reported 3.1 agent-workdays per human workday figure and does not generalise to frontier research operations.
- Reader, effort, and cost
- a team of at least four practitioners; 3 to 4 weeks of ordinary work with measurement overhead; model spend depends on your own workload.
- Prerequisites
- an existing backlog of comparable tasks, a reviewer who does not know which arm produced a result, and a pre-registered acceptance rubric.
Canonical resources
- Vendor operations report: https://openai.com/index/research-acceleration-view-inside-openai/ (opens in a new tab)
- Artifact availability notice: No task ledger, denominator definition, acceptance criteria, or intervention logs were published at review time. The workday ratio is vendor reported and internally defined.
Project plan
| Step | Stage | Effort |
|---|---|---|
| 01 | Define the denominator before starting | 1 day |
| 02 | Blinded crossover assignment | 1 day |
| 03 | Instrument interventions | 2 days |
| 04 | Run the crossover | 2 to 3 weeks |
| 05 | Analysis | 1 day |
01Define the denominator before starting(1 day)
Write down what one human workday means, what counts as an accepted result, and who accepts it. Freeze the rubric. Checkpoint: two reviewers score five past tasks with the rubric and agree.
02Blinded crossover assignment(1 day)
Split practitioners into two groups and alternate assisted and unassisted weeks so every person serves in both arms. Strip arm labels from submitted work before review.
03Instrument interventions(2 days)
Log every human intervention with a timestamp, a reason code, and the elapsed task time when it occurred. Record task length bands, including the 4 to 8 hour band.
04Run the crossover(2 to 3 weeks)
Work normally. Record accepted results, rejected results, defects found after acceptance, interventions, wall time, and compute or model cost per task.
05Analysis(1 day)
Report accepted results per human workday, post-acceptance defect rate, interventions per task hour, and cost per accepted result, each with paired within-person differences and confidence intervals. Falsification: assisted arms raise started work and runtime without raising accepted results, or defects rise enough to erase the throughput gain.
Controls, failure modes, and safety
Controlled variables
task pool, acceptance rubric, reviewer blinding, task length bands, tooling, and per-person crossover order.
Failure modes
counting runtime instead of accepted output, unblinded review, cherry-picked task selection into the assisted arm, and ignoring post-acceptance defects.
Safety
use only work your organisation already owns. Do not place confidential material into third-party tools without your own approval process.