Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI Research and Development
Abstract
Frontier autonomous agents excel at engineering optimization but show unstable performance, limited novelty, and variable experience reuse across long-horizon tasks.
Autonomous agents are increasingly capable of improving models, systems, and other technical artifacts through long-horizon experimentation. To understand the current state of this capability, however, evaluation must go beyond final scores, which neither reveal where progress is gained or lost nor indicate whether accumulated experience improves later decisions. We therefore present a systematic evaluation of seven frontier models on 36 long-horizon tasks based on a new framework that uses rule-based metrics to characterize within-run behavior through Solution Framing, Execution, and Feedback Control and controlled comparisons to assess experience reuse within and across tasks. The results show that current agents operate more like engineering optimizers than fully autonomous researchers: they can formulate and implement practical solutions, but their performance varies substantially across runs, their strongest solutions mainly adapt or combine established techniques, and genuine methodological novelty remains rare. Detailed analysis reveals that observed performance is shaped by multiple factors, including distinct process bottlenecks behind similar final outcomes, experience reuse that can help or mislead subsequent decisions, and harness designs that affect performance stability. These findings suggest concrete directions for improving model training, inference-time strategies, experience management, and harness design.
Community
🚀 We are excited to share Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI Research and Development.
🔍 As AI agents increasingly tackle long-horizon research and engineering tasks, evaluating them by final scores alone is no longer enough. How do they actually approach, execute, and improve throughout the research process?
We systematically evaluate 7 frontier models across 36 long-horizon tasks, going beyond final performance to examine how agents frame solutions, execute experiments, respond to feedback, and reuse experience.
🧑🔬 Our results suggest that today's agents are better characterized as engineering optimizers than fully autonomous researchers: they can formulate and implement practical solutions, but performance remains highly variable across runs, strong solutions largely adapt or combine established techniques, and genuine methodological novelty is still rare.
📊 We also find that:
- similar final scores can hide very different process bottlenecks;
- experience reuse can either help or mislead later decisions;
- harness design substantially affects performance stability.
💡 We hope this study provides a more fine-grained view of where current research agents succeed, where they fail, and what needs to improve next.
Would love to hear your thoughts and discussions!
Get this paper in your agent:
hf papers read 2608.13417 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper