HORIZON: diagnosing long-horizon agent failures

What changed. HORIZON collected more than 3,100 trajectories across four domains and proposed a trajectory-grounded LLM-judge pipeline for attributing failure modes, with reported human-judge agreement of kappa 0.84.

Technique / discovery. Cross-domain task construction, horizon-controlled evaluation, trace analysis, and validated LLM judging.

Why it matters. The field is moving from pass/fail scores toward causal diagnosis of where an agent's trajectory breaks.

Application. General long-horizon agents

Evidence maturity. Preprint (source tier A)

Identified bottleneck. Longer tasks compound state, planning, and recovery errors; automated judges may still share model biases.

Caveat / evidence note. Preprint and evolving benchmark; model names and harnesses age quickly.

Review status. Reviewed. User requested: Yes.

• arXiv (opens in a new tab)

A reproduction tutorial is not yet available for this entry. The closest reviewed material is Reliability, uncertainty, and evaluation and Multi-agent coordination.