Reproduction tutorial · September 2, 2026
Safe patch validation under equal budget
These tutorials test the mechanism behind each paper at a practical scale. They do not reproduce frontier-scale headline results unless explicitly stated.
Source of this tutorial
This tutorial tests the mechanism behind reviewed development 01 in the immutable briefing of September 2, 2026.
Scope and limits
- Reproduction level
- mechanism demo only
- Claim under test
- iterative patching with test feedback can raise the correct-patch rate over one-shot patching at an equal attempt budget, without adding regressions.
- Excluded claim
- this tutorial does not reproduce any vendor cyber benchmark, does not discover or exploit vulnerabilities, and does not test any live target. Restricted models are not used.
- Reader, effort, and cost
- intermediate Python; 4 to 6 hours; local CPU; no paid model required if a scripted patcher is used.
- Prerequisites
- Python 3.11, Git, and a local test runner. Record resolved versions and the commit you run.
Canonical resources
- OpenAI, the path to Astra: https://openai.com/index/path-to-astra/ (opens in a new tab)
- Google, Gemini 3.8 Flash and 3.8 Flash Cyber: https://blog.google/innovation-and-ai/models-and-research/gemini-models/3-8-flash-and-3-8-flash-cyber/ (opens in a new tab)
- Artifact availability notice: The models, internal vulnerability sets, tools, and budgets behind both reports are restricted. This tutorial uses a disposable local repository with seeded defects as a disclosed substitute.
Project plan
| Step | Stage | Effort |
|---|---|---|
| 01 | Containment first | 30 minutes |
| 02 | Seed the defect set | 90 minutes |
| 03 | One-shot condition | 60 minutes |
| 04 | Iterative condition | 90 minutes |
| 05 | Evaluate | 45 minutes |
| 06 | Publish and clean up | 20 minutes |
01Containment first(30 minutes)
Confirm no network access, no credentials, no live targets, and no exploitation code. All work happens inside a disposable local repository in a temporary directory.
02Seed the defect set(90 minutes)
Create a disposable local repository with 30 intentionally seeded defects across independent modules. Each defect has a failing test and a passing regression suite. Record content hashes for every fixture.
03One-shot condition(60 minutes)
Produce one patch per defect with no test feedback. Run seeds 1 through 5. Log correct patch, regression introduced, attempts, and wall time.
04Iterative condition(90 minutes)
Allow the same patcher to read failing-test output and revise, capped at the same total attempt and time budget as the one-shot arm. Use the same five seeds and the same fixtures.
05Evaluate(45 minutes)
Compare correct-patch rate, regression rate, attempts, and time. Report Wilson intervals for rates and paired bootstrap intervals for the seed-level difference. Success requires a higher patch rate without a higher regression rate at equal budget.
06Publish and clean up(20 minutes)
Publish fixtures, content hashes, and run logs. Delete the disposable repository and retain only synthetic fixtures and aggregate results.
Controls, failure modes, and safety
Controlled variables
defect set, seeds, attempt budget, time budget, test harness, and patcher capability.
Failure modes
unequal budgets, test overfitting, silent regressions outside the failing test, defect-difficulty imbalance, and cherry-picked runs.
Safety
no network, no credentials, no live targets, and no exploitation. Seeded defects are synthetic and stay in a disposable local repository.