BixBench3 measures research-study-scale computational biology agents

What changed. Across 20 studies, 138 artifacts, and 13 frontier models, the best mean score was 0.48. Average performance fell to 0.10 above 100 GB and to 0.24 for artifacts at dependency depth three or greater.

Technique / discovery. Artifact dependency graphs, raw-data execution, expert-calibrated programmatic grading, and 24-hour containerized runs.

Why it matters. Complete scientific workflows expose error accumulation, data-scale limits, and recovery failures hidden by isolated question-answer benchmarks.

Application. Computational biology, bioinformatics, scientific agents, and long-running data analysis

Evidence maturity. Preprint (source tier A)

Identified bottleneck. Large datasets, long dependency chains, environment setup, premature termination, retry loops, and scientific alternative-method grading remain hard.

Caveat / evidence note. New arXiv preprint. Tasks prescribe the question and high-level method, programmatic grading can penalize valid alternatives, and source-paper errors may propagate.

Review status. Reviewed. User requested: Yes.

• arXiv / FutureHouse (opens in a new tab)

A reproduction tutorial is not yet available for this entry. The closest reviewed material is Agent planning and cognitive architectures and AI for science.