OA gents: An Empirical Study of Building Effective Agents (2025)

Abstract: Recently, Agentic AI has become an increasingly popular field of research. However, we argue that current practices on agent research are far from standard, rigorous scientific research, which makes it hard to conduct apples-to-apples comparisons among and against existing methods. As a result, it is still obscure how different design choices in an agent framework impact its effectiveness, and measuring progress on agent research remains very hard. In this work, we conduct a systematic empirical study on the GAIA benchmark to investigate the impact of different popular design choices within key agent components in a fair and rigorous way. To begin with, we find that the lack of a standard evaluation protocol makes previous works, even the open-sourced ones, not reproducible, and the variance between different random runs is often non-negligible. Therefore, we first introduce a more robust evaluation protocol to make comparisons more stable. Our empirical study then unveils which components and designs, as well as correlations between these designs, are the keys for building effective agents, while others are not and redundant, despite seemingly making sense. With the insights gained from our empirical study, we build and open-source OAgents, a new foundation agent framework that achieves state-of-the-art performance among open-source projects, providing a good starting point and guidelines for building effective agents. More importantly, supports various design choices for agent components in a modularized way, facilitating future scientific research on Agentic AI.

## introduction in recent years, language agents (significant-gravitas, 2023; @xcite @xcite @xcite have received significant attention due to their potential in resolving general, complex tasks that traditionally required human intervention. however, despite the surge in the number of research works and open-sourced agent frameworks, current practices in agentic ai research are far from being rigorous and scientific. specifically, the current landscape of agent research suffers from a lack of standardized designs and implementation details. critical components such as planning @xcite @xcite , memory @xcite @xcite , and tool use @xcite vary widely across different papers and frameworks, making it difficult to attribute performance improvements to specific innovations. compounding this issue, reported results are often hard to reproduce due to inconsistent evaluation settings or undisclosed framework configurations @xcite . this fragmentation undermines the scientific rigor of the field, as findings cannot be reliably compared or built upon. take the widely researched gaia benchmark @xcite as an example. despite the organizers provide a public leaderboard with evaluation code and a number of papers and projects being open-sourced, it is still very hard, if not impossible, for other researchers to reproduce their results because a number of inconspicuous factors are not standardized, including the implementation details of tools and prompts, as well as details in the evaluation protocol such as how many runs are performed, how errors and failures are handled, and how different results are ensembled or aggregated. these factors often lead to a large impact on the overall performance, sometimes the impact is even larger than some new architecture innovations in new research papers. however, they are generally not mentioned in the technical reports of different agent frameworks and are not even included in their open-sourced codebases. moreover, the engineering design and details in different agent research papers and codebases are so large that it makes it impossible to conduct apples-to-apples comparisons on specific technical designs. this makes it very hard for the research community on agentic ai to properly conduct scientific research instead of digging into tricks on engineering details and evaluation protocols. as a result, despite a lot of agent research papers being released and the numbers on public benchmarks keeping increasing, the best practices on building effective agents are still very obscure. in this work, to promote truly scientific research on agentic ai and provide researchers with a clear understanding of key factors in building effective agents, we conduct a systematic empirical study on the gaia benchmark to sort out the core design choices in current agent research, analyze their impact on performance, and report practical tips for improving experimental stability and reproducibility. specifically, we: (1) carefully implement and compare different designs on key agent components including planning, tool use, memory, testtime scaling strategies, etc., (2) systematically investigate the impacts of different llm backbones and their combinations; (3) thoroughly analyze different practices for evaluation and provide a more robust evaluation protocol. based on the empirical study, we implement and release oagents, a language agent framework that achieves state-of-the-art performance among open-sourced agent frameworks on the gaia benchmark. more importantly, oagents supports modularized integration of almost all critical designs and features in critical components of language agents, including: (1) different agentic planning mechanisms including static and dynamic workflow designs; (2) a complete tool box including web search with different search sources, browsing tools and web crawlers, parsing tools compatible with more document types. (3) different design of the agentic memory module; (4) test-time scaling strategies for agents including different search algorithms and reflection/self-refine mechanisms. hopefully, oagents will facilitate scientific research on language agents by promoting apple-to-apple comparisons and standardizing evaluation protocols. in summary, our main contributions are as follows. ## related work existing work primarily develops agent frameworks along two dimensions: role specialization paradigms construct collaborative networks through differentiated tool allocation (e.g., auto-gpt (significant-gravitas, 2023), autogen @xcite , and camel @xcite ) or functional partitioning(e.g., barcelona2, omne, agen-tim ). smolagents @xcite combines the react @xcite and code act @xcite architectures to build a multi-functional agents hierarchy to perform multiple rounds of interactions and actions in code to accomplish complex tasks. magentic-one @xcite achieves efficient processing of visionlanguage tasks by decoupling perception @xcite , planning @xcite , and execution modules @xcite . dynamic orchestration mechanisms include trase-agent @xcite which proposes task reallocation strategies based on real-time feedback, while tapeagents @xcite employs an asynchronous communication framework to enhance system resilience. experimental evidence suggests that stable sub-agent environment interactions provide greater task success rates than complex orchestration algorithms. autoagent @xcite enables intelligent task execution and personalized agent creation without coding through the core components such as natural language-driven multi-agent coordination, customizable workflows, and self-managing file systems. hybrid architecture exploration is exemplified by h2ogpte-agent (h2o.ai, 2024), which transfers single agent multimodal toolkit text voice image video planning dynamic plan revise step trigger previous steps 𝑆𝑇1 𝑆𝑇2 test-time scaling diversity enhancement verifier progress reword model search algorithms temperature mix of agents reflection multi-nodes list score voting memory memory module ## gaia benchmark the gaia benchmark has emerged as a prominent evaluation framework for assessing the performance of autonomous agents in real-world scenarios. as the leaderboard for this benchmark continues to grow, it becomes increasingly evident that reported results often vary in terms of evaluation metrics-particularly in the use of different pass@k criteria. while some methods report pass@1, others adopt more lenient metrics such as pass@3 or even pass@5. this inconsistency complicates fair comparisons across different agent frameworks and limits the transparency of their actual capabilities. to address this issue and ensure alignment with the leaderboard standards, we reimplement the owl-roleplaying framework to obtain its pass@1 ## conclusion in this work, we conduct a systematic study on the gaia benchmark. we identify key components for effective agents, such as planning, memory, and tool use, and propose a robust evaluation protocol. we release oagents, an open-source modular agent framework achieves state-of-the-art performance on gaia (73.93), providing a foundation for future research on agent systems.