Reproduction tutorial · September 10, 2026
Variant-score calibration against a held-out open assay
These tutorials test the mechanism behind each paper at a practical scale. They do not reproduce frontier-scale headline results unless explicitly stated.
Source of this tutorial
This tutorial tests the mechanism behind reviewed development 04 in the immutable briefing of September 10, 2026.
Scope and limits
- Reproduction level
- partial evaluation against one open held-out assay
- Claim under test
- whether the published variant scores add calibrated discrimination over conservation and established variant-effect baselines on an assay held out from the reader.
- Excluded claim
- this tutorial does not validate genome-wide coverage, does not assess clinical utility, and does not reproduce internal benchmark figures.
- Reader, effort, and cost
- intermediate Python and basic genomics; 8 to 12 hours; API usage within the published free tier where available.
- Prerequisites
- Python 3.11, pandas, scikit-learn, and access to one openly licensed deep mutational scanning or MPRA dataset.
Canonical resources
- Vendor announcement: https://deepmind.google/blog/alphagenome-atlas-a-predictive-map-of-every-possible-dna-letter-change-in-the-human-genome/ (opens in a new tab)
- Use restriction notice: Research use only. The AVI score and the portal are vendor products; nothing in this tutorial is clinical evidence or a basis for any clinical decision.
Project plan
| Step | Stage | Effort |
|---|---|---|
| 01 | Choose and freeze a held-out assay | 90 minutes |
| 02 | Retrieve predictions | 2 hours |
| 03 | Baselines | 2 hours |
| 04 | Discrimination and calibration | 2 hours |
| 05 | Subgroup calibration | 2 hours |
| 06 | Report and disclose | 1 hour |
01Choose and freeze a held-out assay(90 minutes)
Select one openly licensed assay, record its license and accession, and freeze the variant list and labels before retrieving any prediction.
02Retrieve predictions(2 hours)
Pull the published scores, including the AVI score, for the frozen variant list through the portal or API. Record retrieval date and any version string.
03Baselines(2 hours)
Compute at least one conservation baseline and one established variant-effect predictor over the same variants. Baselines must be applied to the identical frozen list.
04Discrimination and calibration(2 hours)
Report AUROC, average precision, calibration curves, and Brier scores with bootstrap confidence intervals for every method.
05Subgroup calibration(2 hours)
Repeat the analysis by region class, gene, and allele frequency band. A method that is well calibrated overall and badly calibrated in a subgroup is a reported finding.
06Report and disclose(1 hour)
Publish the frozen list, retrieval date, versions, and all subgroup results, including the ones that favour the baselines. State explicitly that no clinical conclusion follows.
Controls, failure modes, and safety
Controlled variables
variant list, labels, assay, baselines, retrieval date, score version, and subgroup definitions.
Failure modes
training-data leakage into the chosen assay, selecting the assay after seeing scores, subgroup cherry-picking, and comparing discrimination without calibration.
Safety
research use only. Do not use any output for diagnosis, prognosis, treatment, screening, or any other clinical or reproductive decision, and do not analyse identifiable personal genomic data without approvals.