Paper: AgentBench: Evaluating LLMs as Agents (ICLR-2024)

Reviewer comment (Experiments):
The first issue is with the selection of task domains. The main motivating factor for the authors behind their current selection seems to be potential example scenarios where such LLM-based agents are currently being considered for deployment. While this can be a helpful metric, this approach overlooks an opportunity to curate datasets or even create random problem generators where you can accurately tune the complexity of the tasks being considered (quantified in objective terms such as computational complexity). This could have been done over the current task set (for example, one could quantify the complexity of certain OS operations by looking at factors like the size of filesystems, etc.), but it is currently missing. The current method of scoring each task depends on the abilities of the current set of LLMs being considered and isn’t an objective measure. One potential source for finding task sets where you can objectively quantify task hardness may be considering classical planning and logical reasoning literature, where many benchmark problems with varying degrees of complexity can be found.

Author response:
Thank you for your insightful feedback regarding the selection of task domains and the quantification of task complexity.

1. **Choice of Scenarios**:
    - We agree that it is not feasible to cover every potential scenario in our benchmark. However, we carefully selected well-known, real-world scenarios after research. Our decision-making process is grounded in two fundamental principles:
    - Recognition Criterion: Priority is given to task types universally recognized as pivotal within the language agent community. For instance, benchmarks like NL2Bash[4] and InterCode[5] emphasize the significance of Operating System (OS) tasks. In a similar vein, Wob[6], MiniWob++[7], Webshop[8], Mind2Web[9], and WebArena[10] draw attention to the vital nature of web-based tasks.
    - LLM Applicability: A crucial factor in our selection process is the suitability of tasks for the existing capabilities of Large Language Models (LLMs), specifically in terms of their ability to handle few-shot text prompts in multi-round interactions. For example, while datasets like Minedojo[5] offer more complex challenges in open-world game environments, they often necessitate visual information processing, which is beyond the scope of current LLM capabilities.
2. **Difficulty Quantification:**
    
    Thanks for your great suggestion. We thoroughly re-examine our task settings to make the following improvements:
    
    - **Tasks that are objectively controllable:** For many tasks, we can give an objective definition of difficulty in a specific domain. For example, for an MDP with known environment state transition and limited action space, after limiting interaction rounds, we can calculate the expected score in the same way as the score of Curated List in Text World[1] is defined, and normalize the real score on this basis. .
    - **Tasks that reflect complex real-world needs**: As our aim is to create diverse tasks across multiple domains, we find it infeasible to uniform difficulty quantification for all tasks we consider. For example, tasks in OS operations can span various domains like file manipulation, user management, and network settings, each with inherently different complexity metrics. These tasks cannot be properly assigned difficulty control, or their comprehensiveness to reflect real-world cases would be significantly harmed.

Reviewer comment (Experiments):
Many datasets used in the benchmark were prepared using large language models, including GPT-4 and GPT-3.5. This could potentially influence or bias the results. For example, in the database task, data augmentation was carried out using GPT-3.5-turbo. This was also one of the tasks where GPT-3.5-turbo outperformed GPT-4. There is a question of whether this is correlated.

Author response:
We have also taken into consideration the potential biases that may arise from using Large Language Models (LLMs) for data augmentation, as you mentioned. During the data generation process, we have paid special attention to this issue. Taking the data generation process of DB as an example, the augmentation can be divided into two steps: table augmentation and row labeling. 

- During the table augmentation phase, we ask LLMs to generate similar data entries based on existing rows. After generation, we verify the legitimacy of these data and incorporate them into the table. The purpose of this step is to increase the complexity of the task. Since the LLM simply imitates and repeats based on existing entries, this process should not introduce bias.
- In the INSERT and UPDATE tasks of DB, we used LLMs to generate questions and answers. Specifically, we provide LLMs with header information and row data of the table. We first ask the LLM to generate an SQL statement. If the statement is valid and changes the database, we retain this data. On this basis, we further require the LLM to generate an Instruction describing the changes made in the SQL.

In summary, during the data augmentation phase, we provide as much information as possible to the LLM, enabling it to perform a relatively simple process of generating from a larger to a smaller amount of information. In contrast, during the evaluation phase, we provide limited information to the LLM, requiring it to infer from a smaller to a larger amount of information, a more complex and challenging process. This approach helps to minimize potential biases introduced during the data augmentation stage.

To add credibility, we re-executed a small batch of data labeling for INSERT and UPDATE in DB using claude-2 and re-tested on gpt-4 and gpt-3.5-turbo. The results showed that gpt-3.5-turbo still outperforms gpt-4 in the UPDATE category, as it did originally. This indicates that the superiority of gpt-3.5-turbo over gpt-4 is not due to biases introduced in the data augmentation process, but rather due to other reasons. (Cf. Appendix C.4)

Reviewer comment (Experiments):
The claim that open-source LLMs fall behind closed-source ones may not be entirely fair, since only OSS models with less than or equal to 70B parameters were considered. The authors should consider rephrasing the claim.

Author response:
Thank you for pointing this out. However, in our defence, in fact, the size smaller than 70 billion has already encompassed the majority of open-source LLMs that have undergone fine-tuning for chat or instruction tasks, which constitutes the focus of our testing. Nevertheless, for the sake of rigor, we have revised all the statements in the paper regarding this conclusion to employ more cautious and circumspect expressions.

Regarding the issue of task-specific fine-tuning that you mentioned, it is worth noting that the trajectory formed by our benchmark is indeed applicable for fine-tuning. Some recent efforts have also explored attempts to fine-tune on the trajectory of the agent, such as FireAct[2] and AgentTuning[3]. Due to space constraints and the limited extent of our exploration in this regard, we refrain from making unwarranted claims and, therefore, have not extensively discussed the possibilities in this aspect in the main body of the paper.

Reviewer comment (Experiments):
The evaluation and the benchmark don’t currently appear to allow any form of task-specific fine-tuning, and there is no discussion regarding that topic.

Author response:
Thank you for pointing this out. However, in our defence, in fact, the size smaller than 70 billion has already encompassed the majority of open-source LLMs that have undergone fine-tuning for chat or instruction tasks, which constitutes the focus of our testing. Nevertheless, for the sake of rigor, we have revised all the statements in the paper regarding this conclusion to employ more cautious and circumspect expressions.

Regarding the issue of task-specific fine-tuning that you mentioned, it is worth noting that the trajectory formed by our benchmark is indeed applicable for fine-tuning. Some recent efforts have also explored attempts to fine-tune on the trajectory of the agent, such as FireAct[2] and AgentTuning[3]. Due to space constraints and the limited extent of our exploration in this regard, we refrain from making unwarranted claims and, therefore, have not extensively discussed the possibilities in this aspect in the main body of the paper.

Reviewer comment (Experiments):
The choice to do prompt omission is surprising. While in the most general POMDP case the full history of actions and observations is important, for fully observable cases one could create a new prompt where the current state of the task (along with any task specification) is fully summarized.

Author response:
Thank you for your insightful comment on prompt omission. To clarify, the majority of tasks designed in our study are intended to be completed within 2000 tokens. In fact, our analysis of the interaction trajectories supports this design choice. For completed tasks, we observed an average of 7.95 rounds per task and an average token usage of 2220.1. Additionally, it was rare for the model to use more than 3000 tokens. These findings are detailed in Appendix J.2.3 of our revised version.

Therefore, if the total token count in an interaction exceeds 3500, it indicates a low likelihood of the task being completed. This approach to omission minimally impacts our experimental results. Moreover, this strategy was consistently applied across all models, ensuring a fair evaluation framework.

Furthermore, another interesting observation is made, as detailed in Appendix J.2.4 of our revised version. We find that the majority of instances where models failed to complete tasks were due to a tendency to repeat content already generated. This insight further supports our approach, as a summary that condenses only the current state of the task would not necessarily mitigate this issue of content repetition.

We believe these explanations address your concerns, and we thank you again for your constructive feedback, which has helped improve our analysis and manuscript.

Reviewer comment (Writing):
The writing of the paper requires significant polish, as there are numerous typos and malformed sentences throughout the paper, and it could benefit from thorough proofreading.

Author response:
Thank you for your careful and detailed reading! We have revisited and polished the expression in our paper, rectifying any grammatical errors. Regarding the term "turn-taking" that you mentioned, in our paper, "turn" primarily refers to a round of interaction between the user (in the context of this benchmark, it refers to the task environment) and the chat model, rather than referring to the turn of each agent in a multi-agent interaction. To ensure clarity in terminology throughout the text, we have replaced the term "turn" with "round" in all instances where it refers to a single interaction with the chat model. We hope this enhances the reading experience for our audience.

Reviewer comment (Writing):
The authors' use of certain terms may be inaccurate or misleading. For example, the term "incompleteness" is a technical term used in many areas of AI and computer science, including proof systems and sequential decision-making and planning. It is unclear whether the authors are using it in the technical sense, as in the paper it is related to the task, which doesn’t align with its traditional use. Similarly, the term "turn-taking" is usually associated with multi-agent games, but the authors use it liberally in the context of chat-based models, possibly intending to mean multiple-step or sequential decision-making problems.

Author response:
Thank you for your careful and detailed reading! We have revisited and polished the expression in our paper, rectifying any grammatical errors. Regarding the term "turn-taking" that you mentioned, in our paper, "turn" primarily refers to a round of interaction between the user (in the context of this benchmark, it refers to the task environment) and the chat model, rather than referring to the turn of each agent in a multi-agent interaction. To ensure clarity in terminology throughout the text, we have replaced the term "turn" with "round" in all instances where it refers to a single interaction with the chat model. We hope this enhances the reading experience for our audience.

Reviewer comment (Evaluation):
The benchmark does not seem to offer any insights for improvement. (i.e. If my model is not doing well on web-browsing, what should I do?)

Author response:
Thank you for you input. We address your concerns as follows:

- **Observations from Interaction Trajectories**: Our research offers valuable insights, particularly in scenarios where tasks are not successfully completed. We've identified two primary issues:
    - **Instruction Following**: Models often struggle with adhering to given instructions. This leads to outputs that are either format-incompatible or lack a valid action. (Cf. Appendix J.2.1)
    - **Output Repetition**: There's a tendency for models to repeat previous outputs after a certain number of interactions. This repetition prevents task completion and causes interactions to reach the maximum turn limit. (Cf. Appendix J.2.3)
    - **Code Training with Ambivalent Impacts**: We show that code training helps certain tasks (e.g., Database, WebShop). However, its introduction might harm other tasks (e.g., Digital Card Games, Operating System) that require more ability in interacting. Later work should be aware of this double-edged sword. (Cf. Section 4.3 & Appendix J.2.5)
    - **High-Quality Alignment Data**: By comparing models like vicuna-13b and llama-2-13b (which share the same base model), we show that alignment data distilled from GPT-4 (e.g., ShareGPT), significantly enhances LLM performance. Vicuna-13b, aligned with ShareGPT data, notably outperforms its counterpart and rivals larger models. (Cf. Section 4.3)
- **Implications for Fine-Tuning**: These observations suggest key areas for fine-tuning LLMs-as-Agents, one is enhancing instruction compliance, and another is addressing repetitive outputs.
- **Future Work and Community Contribution**: While our current focus is on establishing a robust agent benchmark for the community, we acknowledge the importance of fine-tuning exploration. We plan to delve into this in our future work. It's also worth mentioning that recent studies like **FireAct[3]** and **AgentTuning[4]** have already been inspired and begun exploring fine-tuning approaches for LLMs-as-Agents.

In summary, our work lays the groundwork for future advancements in fine-tuning LLMs-as-Agents, offering a clear direction for addressing key challenges identified in our research.

Reviewer comment (Experiments):
The embodied tasks seem quite contrived. AlfWorld drops all 2d/3d aspects of the environment and could be mastered by a fine-tuned GPT-2 [1].

Author response:
Thank you for your further insights regarding the selection of ALFWorld as an evaluation task in our study. We appreciate the opportunity to clarify the reason of choosing it.

While ALFWorld, as a text-based environment, does simplify the complexity, it is still a problem relevant and challenging for language models, particularly those not fine-tuned on the task. It requires strategic planning and decision-making within a given scenario. As illustrated in Appendix J.2.2, these tasks demand a level of cognitive engagement and planning ability that goes beyond the scope of basic language understanding, thereby offering a valuable framework for evaluating the depth and adaptability of language models.

As a result, ALFWorld has been widely adopted in recent typical LLM-as-Agent study (e.g., ReAct[1], Reflextion[2]), and we thus adopt it in AgentBench in an effort to unify the evaluation and results for later LLMs. However, we do agree that it might not be so challenging for LLMs after task-specific fine-tuning, and we are closely working on finding more complement datasets for the task in our next planned update.

Reviewer comment (Evaluation):
The benchmark seems to be mostly coding based. Non-coding LLMs could potentially still behave as good agents, but would underperform on this benchmark.

Author response:
Thanks for your valuable comment. Mindful of this important aspect, we wish to clarify that we have conscientiously designed AgentBench to be not mostly coding based, ensuring a balanced representation of tasks.

In our benchmark, only 3 out of 8 tasks, specifically Operating system, Database, and Knowledge Graph, involve coding aspects. These tasks are designed to evaluate the agent ability of language models when they are grounded on coding. However, it is crucial to highlight that a significant portion (5 out of 8) of our benchmark, including Digital Card Game, Lateral Thinking , House Holding, Web Shopping, and Web Browsing, does not involve coding at all. These tasks focus on different aspects like decision-making, language comprehension, and reasoning, offering a comprehensive evaluation of language models in non-coding contexts.

This balance between coding and non-coding tasks ensures a holistic assessment of language models, capturing their versatility and range of capabilities. We believe this approach accurately reflects the diverse applications and potential of contemporary language models.

Reviewer comment (Presentation):
The use of abbreviations. I find the abbreviations "OS DB KG DCG LTP HH WS WB" make little sense to me. It took me a long time to dig through different pages to understand the benchmark. I hope the authors would re-format the tables for better readability.

Author response:
We sincerely apologize for any misunderstanding and confusion caused by the abbreviations. We have replaced all abbreviations with their full forms in the paper.

Reviewer comment (Presentation):
Figure (a) is really hard to read. I struggle to tell the colors apart. I suspect that this figure is not color-blind friendly.

Author response:
We sincerely apologize. We have chosen a clearer and more distinct method to present this figure to ensure better readability and accessibility, including reduce 9 displayed models to 6, use black-and-white to indicate API-based LLMs, and use seaborn’s `colorblind` palette to color OSS LLMs.

Reviewer comment (Experiments):
The benchmark seems to use the same prompt for all models, which might give an unfair advantage to the model where these prompts were developed for.

Author response:
Thanks for your pointing out. In fact, our task prompts are chosen based on a combination of representative LLMs (rather than an individual series of models) to ensure their robustness. We experimented with various prompt paradigms, such as ReAct[1] and Chain of Thought (CoT)[2], and tested them across multiple models including claude-2, gpt-4, gpt-3.5-turbo, llama-2 and so on to ensure their universal validness. We did not select task prompts specifically tailored to any particular model. All models were tested under relatively equal conditions.

Reviewer comment (Experiments):
There could be data leakage to the tasks selected from the pretraining data over the internet.

Author response:
Thanks for your pointing out. We acknowledge that there is a risk of data leakage when selecting tasks from internet-based pretraining datasets. However, a significant difference of AgentBench to other benchmarks is that to solve AgentBench requires leakage of groundtruth trajectories based on our dynamic environments (e.g., filenames and paths in linux dockers of our OS dataset), which does not exist on the web as text corpora.

As a result, we believe for AgentBench, there is very minimal risks of data leakage in the evaluation.

Reviewer comment (Evaluation):
How is the success rate calculated with the runs failed to complete?

Author response:
We calculate success rate as follows:

$SR = \frac{C}{N}$,

where $C$ represents the count of correct results, and $N$ denotes the total count of samples.

More specifically, for tasks whose evaluation metrics are success rate, the following situations may occur after each sample has finished:

- Completed:  the final status is a termination status (e.g., submit an answer in OS or DB)
    - Correct:  the final status is accepted as a correct solution.
    - Wrong:  the final status is NOT accepted as a correct solution.
- Context Limit Exceeded
- Invalid Format
- Invalid Action
- Task Limit Exceeded

Only **Correct** results are the part of successful ones, meaning:

$SR = \frac{\\#Correct}{\\#Completed + \\#CLE + \\#IF + \\#IA + \\#TLE}$

Reviewer comment (Presentation):
Which model does table 4 correspond to?

Author response:
Thank you for pointing this out. Table 4 represents an overall ratio across all models. We have fixed this ambiguity in the paper.

Reviewer comment (Novelty):
The contributions of the paper seem very limited -- the paper does not propose any new technical insights and simply applies a variety of LLMs on many existing environments.

Author response:
We are afraid that there might be some misunderstandings about our goal and contributions:

1. The comment ``simply applies a variety of LLMs on many existing environments'' isn't  accurate. 
    1. **Most environments are newly created in this work**: in AgentBench 5 out of 8 tested environments (OS, DB, KG, LTP, DCG) are created for the first time (see citation marks in Section 1), rather than ``many existing environments''. 
    2. **It is challenging to set up the evaluation**: to create testable new environments and integrate them for scalable testing on LLMs require quite arduous algorithmic and engineering efforts, rather than a ``simply'' achieved work.
2. Our contributions on technical insights are significant. 
    1. It is the very first work to quantitatively benchmark many LLMs (up to 27) on multiple real-world agent tasks in a unified setting. 
    2. The techniques we develop in creating 5 new environments out of 8 (Cf. Appendix B-F, 14 pages) are useful for later agent environment creation. The handy integrated testing toolkit we develop would serve as a cornerstone for all later new agent tasks to use.
    3. The LLM behaviors and fall-shorts we discover and analyze during the evaluation are beneficial for improving LLMs’ agent abilities (Cf. Appendix J, 11 pages).

Reviewer comment (Evaluation):
The analysis is not particularly insightful and I'm not sure if the conclusions are fully accurate from the analysis. For instance, task length exceeded may just be due to the fact that many LLMs are trained on short fixed context lengths. In settings such as this, it would be better to first summarize the long context to keep inputs in distribution to the input LLM.

Author response:
Thank you for your feedback. We are afraid there might be some obfuscation about the `task length exceeded` you mentioned, which should refer to the term "Task Limit Exceeded (TLE)" in our paper as we understand. It has no relationship with LLMs' `short fixed context lengths` (i.e., the CLE as we reported, which only accounts less than 1% of all failure cases in most tasks as shown in Table 4), but holds the different meaning of "unexpected exceeding of maximum rounds of interaction as we design".

In fact, we have carefully designed our tasks to be mostly solvable within around 1000 tokens, and a relatively harder limit is 3500 tokens when earlier histories begin to be folded. Our findings in Appendix J.2.3 proves that 95% of completed results are around this limit. Exceeding this limit typically indicates a divergence from the correct solution path (i.e., a failure solution).

Notably, as shown in Figure 9 in Appendix J.2.4, in 90% of instances where Task Limit Exceeded occurred, the LLM began to deadly loop over previously generated content, resulting in repetition of identical contents in the final 10 rounds of interaction. It is the repetition that triggers the exceeding of maximum interaction rounds (i.e., TLE), rather than context length exceeding.

We hope this clarification would address your concerns.

Reviewer comment (Novelty):
In the related work section, it would be good to also add some references to multiagent approaches to reasoning such as multiagent debate. Also age 2023 seems to be incorrectly formulated.

Author response:
Thanks for your great suggestion. We have updated related work and added multi-agent references in our revised version.

Reviewer comment (Writing):
Some of the references in the paper are incorrectly cited. For instance, in the intro, Reed et al. 2022 does not use a LLM to learn an embodied policy, a better reference would be [1]. Similarly, Ahn et al. 2022 does not use a complex multi-modal simulator based off games but rather a real robot.

Author response:
We appreciate your feedback on our paper. It seems there might be a misunderstanding about our context to cite these two papers. The section in our paper that references ``embodied agents'' is meant to encompass agents that interact with their environment through a physical presence within that environment, regardless of using LLMs. The articles by Reed et al. 2022 and Ahn et al. 2022 are discussed in this context of embodied agents. Furthermore, Ahn et al. 2022 specifically integrates language and vision modalities, exemplifying a multi-modal approach to embodied agents.

Additionally, we are grateful for your recommendation of the paper “Pre-Trained Language Models for Interactive Decision-Making.“ We have included this reference in the revised version of our paper.

Reviewer comment (Experiments):
Have you tried seeing the results of each method assuming a small number of demonstrations to fine-tune each LLM to each domain?

Author response:
Our work does provide insights into the fine-tuning of models. For instance, in cases where tasks are not successfully completed, we observed interaction trajectories and identified two prevalent issues. Firstly, the models often fail to follow instructions, leading to outputs that either don't conform to the specified format or lack a valid action. Secondly, models tend to repeat previously given outputs after a certain number of turns, preventing the task from reaching a proper conclusion and causing interactions to hit the maximum turn limit. This suggests that the ability to follow instructions might be a critical aspect to focus on during the fine-tuning of LLM-as-Agents. Moreover, implementing penalties for repetitive content could be beneficial.

However, due to our focus is to first set up a usable agent benchmark for the community, we would like to leave the fine-tuning exploration and the tackling of our benchmark in the future work. But it is noteworthy that some recent works, such as **FireAct[1]** and **AgentTuning[2]**, have been inspired and thus engaged in fine-tuning LLMs-as-Agents.

Reviewer comment (Experiments):
What is the sensitivity of the performance of each LLM to chosen prompts? It seems like prompts used for evaluation are very complex and it would be good to understand how they were chosen and how prompts would affect the performance of each LLM.

Author response:
Our task prompts are chosen based on a combination of representative LLMs (rather than an individual series of models) to ensure their robustness. We experimented with various prompt paradigms, such as ReAct[3] and Chain of Thought (CoT)[4], and tested them across multiple models including claude-2, gpt-4, gpt-3.5-turbo, llama-2 and so on to ensure their universal validness. We did not select task prompts specifically tailored to any particular model. All models were tested under relatively equal conditions.

Reviewer comment (Experiments):
Can the authors come up with a method to try tackle all of these benchmarks?

Author response:
Our work does provide insights into the fine-tuning of models. For instance, in cases where tasks are not successfully completed, we observed interaction trajectories and identified two prevalent issues. Firstly, the models often fail to follow instructions, leading to outputs that either don't conform to the specified format or lack a valid action. Secondly, models tend to repeat previously given outputs after a certain number of turns, preventing the task from reaching a proper conclusion and causing interactions to hit the maximum turn limit. This suggests that the ability to follow instructions might be a critical aspect to focus on during the fine-tuning of LLM-as-Agents. Moreover, implementing penalties for repetitive content could be beneficial.

However, due to our focus is to first set up a usable agent benchmark for the community, we would like to leave the fine-tuning exploration and the tackling of our benchmark in the future work. But it is noteworthy that some recent works, such as **FireAct[1]** and **AgentTuning[2]**, have been inspired and thus engaged in fine-tuning LLMs-as-Agents.

Reviewer comment (Experiments):
Can the authors explain more why the chosen set of text benchmarks comprehensively evaluate the ability of LLMs to act as agents? For instance, why isn't there a benchmark to test multiagent interaction between LLMs?

Author response:
Thank you for your question. 

- Task selection: We conducted extensive research and evaluated numerous tasks that could potentially form part of our benchmark, as we have heavily discussed in Section 1 & 5. Generally we follow two principles to make decisions:
    - Recognition: We first consider task types that are universally acknowledged as crucial in the language agent community. For instance, NL2Bash[6] and InterCode[7] underscore the importance of OS tasks. Similarly, Wob[8], MiniWob++[9], Webshop[10], Mind2Web[11], and WebArena[12] highlight the critical role of web-based tasks.
    - LLM Applicability: Another key criterion is that the task should be compatible and of proper difficulty levels for current LLMs’ few-shot text prompting with multi-round interactions. For example, though datasets like Minedojo[5] can provide more challenging open-world game environments, it generally requires vision information beyond existing LLMs capability.
- Multi-agent evaluation: Thanks for your suggestion. Multi-agent interaction evaluation do count as an aspect in LLM-as-Agent evaluation, but we think currently it might be too hard for existing LLMs (especially those smaller ones with less than maybe 30B parameters, which cannot even handle single-agent tasks). Additionally, widely-acknowledged metrics and tasks for evaluating LLMs’ multi-agent are still not that clear at this time. Therefore, we decide to first place our focus on single-agent evaluation at present, but envisage our benchmark as something that will grow and expand over time to incorporate multi-agent interaction scenarios in the near future.