A gent Q uest: A Modular Benchmark Framework to Measure Progress and Improve LLM Agents (2024)

Abstract: The advances made by Large Language Models (LLMs) have led to the pursuit of LLM agents that can solve intricate, multi-step reasoning tasks. As with any research pursuit, benchmarking and evaluation are key corner stones to efficient and reliable progress. However, existing benchmarks are often narrow and simply compute overall task success. To face these issues, we propose AgentQuest – a framework where (i) both benchmarks and metrics are modular and easily extensible through well documented and easy-to-use APIs; (ii) we offer two new evaluation metrics that can reliably track LLM agent progress while solving a task. We exemplify the utility of the metrics on two use cases wherein we identify common failure points and refine the agent architecture to obtain a significant performance increase. Together with the research community, we hope to extend AgentQuest further and therefore we make it available under https://github.com/nec-research/agentquest.

## introduction generative agents @xcite are software systems that leverage foundation models like large language models (llms) to perform complex tasks, take decisions, devise multi-steps plans and use tools (api calls, coding, etc.) to build solutions in heterogeneous contexts @xcite . the potential ability to solve heterogeneous tasks with high degrees of autonomy has catalysed the interest of both research and industrial communities. nonetheless, it is still unclear to which extent current systems are successfully able to fulfil their promises. in fact, methodologies to benchmark, evaluate and advance these systems are still in their early days. we identify a couple of gaps. firstly, benchmarking agents requires combining different benchmark types (liu et al., 2023; chalamalasetti et al., 1 demo provided at https://youtu.be/0jnkifwnoak . agentquest defines a common interface to interact with the benchmarks and to compute progress metrics, easing the addition of new benchmarks and allowing researchers to evaluate and debug their agent architectures. 2023). for example, some benchmarks focus on specific capabilities and provide gaming environments, which we refer to as "closed-box" -i.e. with a finite set of actions @xcite @xcite -whereas other benchmarks provide open-ended tasks and access to general tools, like web browsing @xcite @xcite . as benchmarks are developed independently, significant effort goes into custom integration of new agent architectures with each benchmark. secondly, and more critically, existing benchmarks mostly focus on providing a success rate measure, i.e. a binary success/fail evaluation for each of the proposed tasks. while success rate is helpful to measure overall advances of an agent technology, it has limited use in guiding improvements for new generative agent architectures. here, it is important to consider that generative agents often combine foundation models with multiple other components, such as memory and tools. develop-ers can reason about these individual components in terms of architecture and their inter-dependence, and could actively change and evolve them using deeper insights about how an agent performs in a benchmark. that is, developers need benchmarks to both evaluate and debug agents. for example, current benchmarks make it hard to answer questions like does the agent fail completely the tasks or does it partially solve them? does the agent fail consistently at a certain step? would extra run time lead to finding a solution? answering these questions would require tracing and inspecting the execution of the agent. we argue that providing a more efficient approach that is consistent over multiple benchmarks is a stepping stone towards evolving generative agents. we address these gaps introducing agentquest, a modular framework to support multiple diverse benchmarks and agent architectures (see figure 1 ), alongside with two new metrics -i.e. progress rate and repetition rate -to debug an agent architecture behaviour. agentquest defines a standard interface to connect an arbitrary agent architecture with diverse benchmarks, and to compute progress and repetition rates from them. we showcase the framework, implementing 4 benchmarks in agentquest: alfworld @xcite , lateral thinking puzzles @xcite , mastermind and sudoku. the latter two are newly introduced with agentquest. additional benchmarks can be easily added, while requiring no changes to the tested agents. our final contribution is to present our experience leveraging the proposed metrics to debug and improve existing agent architectures as implemented in langchain @xcite . in particular, we show that in the mastermind benchmark the combination of progress rate and repetition rate identifies a limitation in the ability of the agent to explore the full space of potential solutions. guided by this insight we could improve the success rate in this benchmark by up to ≈20%. in lateral thinking puzzles we show that partially repeating actions is part of the agent strategy, whereas in alfworld, we show that monitoring the progress rate makes it possible to identify that the final success rate is limited by the allowed runtime of the agent, and that more steps lead to a better performance. finally, in the sudoku benchmark, we show that the low success rate is actually paired with low progress rate, making clear that the tested agent is unable to solve this type of tasks. ## generative ai agents in a nutshell generative ai agents are automated systems relying on software components integrated with llms pre-trained on large amount of data for language understanding and processing. when assigned a task, an agent engages in a systematic process: it iteratively formulates self-generated instructions, executes them, and observes the outcomes until the ultimate objective is achieved. next, we showcase the basic interaction between agents and the environment in which they operate and describe the standard benchmarking techniques. ## agentquest overview we designed agentquest as a separation layer between agent and environment (see figure 1b ). essentially, it offers (i) a unified interface (i.e. the driver) ensuring compatibility between different agent architectures and benchmarks with minimal programming efforts (section 3.1); (ii) the implementation of two metrics beyond task success (i.e. progress rate and repetition rate) aimed at monitoring the agent advancement toward the final goal and allowing us to understand the reasons behind failures (section 3.2); (iii) a unique vantage point and interface for implementing new metrics to monitoring and measuring the execution (section 3.3). ## insights via agentquest we investigate agent behaviours in different reasoning scenarios by proposing a starting set of four benchmarks. we implemented from scratch sudoku @xcite and mastermind @xcite environments, while alfworld @xcite and lateral thinking puzzles (ltp) @xcite are existing implementations @xcite . we emphasise that this evaluation is not aimed at providing a thorough evaluation and comparison of agent architectures, but rather to show how to use agentquest and how monitoring progress and action repetition can provide relevant insights to developers, even after a few executions. experimental setup. we use as reference architecture the off-the-shelf chat agent provided by langchain @xcite powered by gpt-4 (ope-nai, 2023b) as llm because it is intuitive, easy to extend and open source. we run 15 instances of the four benchmarks within agentquest, setting the maximum number of execution steps as 60 5 . in appendix b we provide examples on how to use agentquest with two additional agent architectures and gaia @xcite as open-ended environment. experimental results. for mastermind, figure 2a shows the progress rate pr t and repetition rate rr t . in the first 22 steps, the agent explores different solutions (rr [0,22] < 5%). this leads to growing progress towards the final goal, reaching half of the milestones (pr 22 ≈ 55%). then, the agent starts performing the same actions, exhibiting a repetitive pattern (see also figure 3a rightmost part) and failing to reach the final goal within the 0 10 20 30 40 50 60 execution steps 1 11 22 actions id (a) mastermind 0 10 20 30 40 50 60 execution steps 1 5 10 actions id (b) ltp figure 3: examples of repeated actions in mastermind and ltp. mastermind: there is a set of unique actions at first, but then gets stuck repeating the same actions over and over. ltp: repeated actions are small variations of the same question that lead to progress. to overcome this agent limitation we incorporate a memory component @xcite into the agent architecture. the agent stores the past guesses in a local buffer. then, at each step, if the agent outputs an action already in the buffer, it is prompted to provide a new one. for ltp, the agentquest metrics reveal a dif-ferent agent behaviour, where repetitions are part of the agent reasoning strategy, enhancing the progress rate (figure 2b ). from the initial steps, the agent changes aspects of the same questions until a local solution emerges. this leads to horizontal indicators in figure 3b and rr 20 ≈ 30%. despite solving only a few riddles (@xmath0.2), these repetitions contribute to progress, achieving 46% of the milestones by the end of the execution, with a final repetition rate of rr 60 = 81%. this shows us how the interplay of progress and repetition rates provides an insight on how agents behave across the different time steps. consider the benchmark alfworld in finally, we look at sudoku, known for its high level of difficulty @xcite . the low progress and repetition rates achieved after 60 steps (pr 60 = 8% and rr 60 = 22%) indicate that the current agent architecture struggles in finding correct solutions solving this task. we report the metrics trend in appendix a. ## conclusions agentquest allows the research community to keep track of agent progress in a holistic manner. starting out with a first set of four benchmarks and two new metrics, agentquest is easily extendable. furthermore, the two proposed metrics, progress and repetition rates, have the great advantage of allowing to track how agents advance toward the final goal over time. especially studying their interplay can lead to important insights that will allow the research community to improve agent performance. finally, we believe that promptly sharing agentquest with the research community will fa-cilitate benchmarking and debugging agents, and will foster the creation and use of new benchmarks and metrics.