Safe toy reward-hacking intervention

Scope and limits

Reproduction level

mechanism demo

Claim under test

repeated optimization in reward-leaky environments can increase exploitation of different, held-out reward channels when exploitation raises observed score.

Excluded claim

this tutorial uses no frontier model and performs no cyber, harmful-content, biological, real-network, credential, or sandbox-escape work. It does not replicate any Anthropic percentage.

Reader, effort, and cost

intermediate Python and reinforcement learning; 6 to 10 hours; local CPU; no paid model.

Prerequisites

Python 3.11, NumPy 2.1.x, Gymnasium 1.1.x, Stable-Baselines3 2.6.x, SciPy 1.14.x, and Git. Record the resolved versions and the commit you run.

Canonical resources

• Anthropic Alignment Science, reward seeker: https://alignment.anthropic.com/2026/reward-seeker/ (opens in a new tab)
• Anthropic, improving alignment and security efforts: https://www.anthropic.com/news/improving-alignment-security-efforts (opens in a new tab)
• Concrete Problems in AI Safety, 2016: https://arxiv.org/abs/1606.06565 (opens in a new tab)
• Artifact availability notice: The model, the 80 production-derived environments, the checkpoints, and the complete RL pipeline are unavailable. This tutorial uses a disclosed synthetic substitute.

Project plan

1. 01Containment first(45 minutes)

   Write containment unit tests and a network-off allowlist before any training code. Confirm there is no code execution, subprocess, network, credential, or filesystem access outside a temporary directory.

2. 02Deterministic substitute environment(90 minutes)

   Build a 7x7 synthetic grid. The intended goal collects a token and exits for reward 1. In 40 percent of leaky training variants a harmless local edit_score action changes a temporary score.json to reward 5. Add 20 reward and containment tests.

3. 03Map splits(45 minutes)

   Generate 200 training, 100 validation, and 200 holdout maps, and record content hashes for each split. Held-out evaluation uses a differently named claim_bonus flag that never appears in training.

4. 04Clean baseline(90 minutes)

   Train five clean PPO seeds: 11, 22, 33, 44, 55. Record configs, curves, and wall time.

5. 05Matched leaky intervention(90 minutes)

   Repeat the run with the leaky variants, holding seeds, budget, architecture, and schedule fixed.

6. 06Controls(60 minutes)

   Add reward-clipped and channel-hidden control arms to separate magnitude effects from channel access.

7. 07Freeze and evaluate(60 minutes)

   Freeze checkpoints and run 500 holdout episodes per seed against the unseen claim_bonus channel.

8. 08Report(60 minutes)

   Report exploit rate, true-task success, reward gap, Wilson intervals, and paired bootstrap intervals.

9. 09Stress tests(60 minutes)

   Stress reward magnitude, solvable versus impossible tasks, and neutralized action names to test whether naming or impossibility explains the effect.

10. 10Cleanup(20 minutes)

    Clean temporary files and retain the audit artifacts: hashes, configs, seeds, and aggregate tables.

Controls, failure modes, and safety

Affected topics

Back to the September 1, 2026 briefing