Reproduction tutorial · August 28, 2026
WikiSkill persistent knowledge and skill evolution
These tutorials test the mechanism behind each paper at a practical scale. They do not reproduce frontier-scale headline results unless explicitly stated.
Source of this tutorial
This tutorial tests the mechanism behind reviewed development 04 in the immutable briefing of August 28, 2026.
Scope and limits
- Reproduction level
- mechanism demo
- Claim under test
- separating persistent knowledge from executable skills can improve later skill revisions and transfer.
- Excluded claim
- this tutorial does not reproduce the paper's full benchmark suite or cross-family headline results.
- Reader, effort, and cost
- intermediate Python; 4 to 8 hours; local CPU; optional model API budget below $10.
Canonical resources
- Paper: https://arxiv.org/abs/2608.27454 (opens in a new tab)
- Canonical resource notice: At review time the arXiv landing page did not expose a canonical public repository, dataset, or benchmark package, so the tutorial uses a synthetic file-transformation benchmark.
Project plan
| Step | Stage | Effort |
|---|---|---|
| 01 | Create tasks | 45 minutes |
| 02 | Define stores | 45 minutes |
| 03 | Run three conditions | 120 minutes |
| 04 | Introduce change and poison tests | 45 minutes |
| 05 | Evaluate | 60 minutes |
| 06 | Cleanup | 10 minutes |
01Create tasks(45 minutes)
Generate 60 synthetic CSV-cleaning tasks from six transformation families. Hold out two parameter combinations and all phrasing variants for transfer evaluation.
02Define stores(45 minutes)
Raw experience stores traces and errors. Wiki entries store sourced observations, conditions, counterexamples, and timestamps. Skills store executable functions with input schema, tests, provenance, compatible environment, and version.
03Run three conditions(120 minutes)
Compare no persistent store, skills-only revision, and wiki-plus-skill co-evolution. Give each condition the same task order, model, token budget, retry limit, and test harness.
04Introduce change and poison tests(45 minutes)
Change one CSV schema, insert one false wiki entry, and revoke one skill assumption. Measure detection, correction, rollback, and negative transfer.
05Evaluate(60 minutes)
Across five seeds report held-out task success, first-attempt success, tokens, retries, stale-skill failures, time to repair, and transfer to a second model or scripted planner. Success requires better held-out performance than skills-only without more uncorrected stale-memory failures. Falsification occurs if gains disappear on held-out transformations or accumulated knowledge increases harm.
06Cleanup(10 minutes)
Delete raw traces that contain prompt content, retain synthetic tasks, redacted wiki entries, skill versions, and aggregate results.
Controls, failure modes, and safety
Uncertainty
bootstrap seed-level differences and report the complete task-family breakdown.
Failure modes
benchmark memorization, hidden task leakage, self-evaluation bias, stale knowledge, poisoned memory, and incompatible skill transfer.
Safety
execute generated skills in a temporary directory with no network, credentials, or access outside synthetic fixtures.