Reproduction tutorial · September 10, 2026
Prospective station backtest of an hourly global forecast
These tutorials test the mechanism behind each paper at a practical scale. They do not reproduce frontier-scale headline results unless explicitly stated.
Source of this tutorial
This tutorial tests the mechanism behind reviewed development 05 in the immutable briefing of September 10, 2026.
Scope and limits
- Reproduction level
- prospective evaluation at 20 stations
- Claim under test
- whether the published forecasts beat an operational baseline at matched issue and lead times on temperature, humidity, and wind at fixed stations.
- Excluded claim
- this tutorial does not reproduce global skill scores, does not evaluate the model's own gridded verification, and does not test tropical-cyclone or severe-weather guidance.
- Reader, effort, and cost
- intermediate Python; roughly 6 hours of setup plus 30 days of unattended archiving and 4 hours of analysis; low cost.
- Prerequisites
- Python 3.11, a scheduler such as cron, and access to one operational baseline forecast and one open station-observation feed.
Canonical resources
- Vendor product page: https://deepmind.google/science/weathernext/ (opens in a new tab)
- Operational notice: Reported outputs include 5 km station temperature and humidity and 10 km wind. Never use this or any model output in place of official warnings from your national meteorological service.
Project plan
| Step | Stage | Effort |
|---|---|---|
| 01 | Fix 20 stations and freeze the protocol | 2 hours |
| 02 | Build the prospective archive | 2 hours setup, 30 days running |
| 03 | Match issue and lead times | 2 hours |
| 04 | Score | 2 hours |
| 05 | Report | 1 hour |
01Fix 20 stations and freeze the protocol(2 hours)
Choose 20 stations spanning coastal, inland, mountain, and urban settings before collecting anything. Freeze variables, lead times, issue times, and metrics in writing.
02Build the prospective archive(2 hours setup, 30 days running)
Archive, on a schedule, the model forecast, the operational baseline forecast, and the later verifying observation. Prospective collection is what prevents hindcast leakage.
03Match issue and lead times(2 hours)
Discard any case where the two forecast systems do not share the same issue time and lead time. Record how many cases were discarded and why.
04Score(2 hours)
Report MAE for temperature, humidity, and wind speed, Brier scores for any probabilistic output, interval coverage, and recall of the top one percent extreme observations, all with bootstrap confidence intervals by station.
05Report(1 hour)
Publish the station list, protocol, discard counts, and per-station results. Falsification: no matched-time advantage over the operational baseline, or better mean error with worse extreme recall.
Controls, failure modes, and safety
Controlled variables
station list, variables, issue times, lead times, baseline system, observation source, and metric definitions.
Failure modes
hindcast leakage, unmatched lead times, station selection after seeing scores, aggregating away station-level failure, and reading mean error as extreme-event skill.
Safety
evaluation only. Never replace or reinterpret official warnings from a national meteorological service, and do not use these results for operational or emergency decisions.