Paper: SmartPlay : A Benchmark for LLMs as Intelligent Agents (ICLR-2024)

Reviewer comment (Reproducibility):
The paper omits important details necessary for functioning as a benchmark, such as how the evaluation of new LLMs/agents should be done (e.g., number of episodes at minimum, whether users are allowed to change the environment, and whether users can change the prompt manually before feeding it to the agent).

Author response:
W1.1 (evaluation protocols) Please see Section 3.1 Environment Interface and Evaluation Protocol, and Table 1 for all the details required, including num episodes for each game, and what the user may or may not change.

Reviewer comment (Evaluation):
Details and motivation for the different capabilities are missing. These need to be clarified and carefully motivated to make a solid case for why these capabilities matter and how they are properly measured.

Author response:
Q2 (Motivation for different capabilities) Identifying the 9 capabilities is part of the contribution of the paper. Our choice and definitions are largely motivated from challenges that game-design try to capture [1]. "Randomness" refers to understanding of randomness, which is important because many real-world events are probabilistic (covered in introduction).  "Rollback" is discussed in tower-of-Hanoi Section 2.4 as undo incorrect moves. 

We discussed in detail how each challenge instantiates in individual games in Section 2, but we did not offer a formal introduction.
Thank you for pointing out this omission, we will update the final version of our paper with a formal introduction to how each level of the 9 capabilities are defined.

Reviewer comment (Experiments):
Other important details, such as how the human baseline was collected, are missing.

Author response:
W1.3 (Human baseline) Please see Q5.1

Q5.1 (human baseline) 3 players (including the authors) who are very familiar with the environments and API played the games through the API. We will add clarification to this in the paper.

Reviewer comment (Experiments):
There is a lack of certain baselines/experiments. While having a fixed, single way of defining prompts makes the benchmark fixed and results comparable across models, there should be an option to try different techniques of prompting the models. For example, chain-of-thought prompting can, with simple modifications to the prompt, improve results in the LLM space.

Author response:
W1.4 (Different prompts) To be fair to all LLMs, the experiments we conduct (page 7, Section 4) uses the popular minimalist prompt of “let’s think step by step”. However, as explained in the general rebuttal (The SmartPlay Interface section), SmartPlay is compatible with any prompting frameworks. We hope that SmartPlay will help future research on creating

Reviewer comment (Writing):
It is unclear how the capabilities were chosen and how their different degrees and importance to LLM research are exactly defined. The definitions appear to stem from the environments chosen but are still vague (e.g., what exactly is "Randomness"? Randomness in what? What is "Reasoning with object dependencies"? What is "rollback" error handling?). The paper would benefit from exact definitions and a detailed section explaining them, which would allow other researchers to bring new environments/dimensions into the mix.

Author response:
Q2 (Motivation for different capabilities) Identifying the 9 capabilities is part of the contribution of the paper. Our choice and definitions are largely motivated from challenges that game-design try to capture [1]. "Randomness" refers to understanding of randomness, which is important because many real-world events are probabilistic (covered in introduction).  "Rollback" is discussed in tower-of-Hanoi Section 2.4 as undo incorrect moves. 

We discussed in detail how each challenge instantiates in individual games in Section 2, but we did not offer a formal introduction.
Thank you for pointing out this omission, we will update the final version of our paper with a formal introduction to how each level of the 9 capabilities are defined.

Reviewer comment (Reproducibility):
Exact details on the environments used in the benchmark package should be provided. While the code will allow exact reproduction, it might get updated on GitHub, so having a solid description of the originally proposed setup would be beneficial.

Author response:
Q3 (Environment Details) We cover environment rules, mechanics, and observation in Section 2 “Games in SmartPlay”. In addition, we provide example observations from all environments in the appendix.

Reviewer comment (Experiments):
It is unclear whether ways to improve model performance were experimented with, such as chain-of-thought prompting. Such techniques seem to improve model performance across different tasks, and they should be supported by the benchmark. For example, a flag, mutator, or wrapper could be added to the environment to modify the instructions/observations. Initial results with these would strengthen the paper, although they are not strictly necessary. The main request is to consider and support these possibilities.

Author response:
Q4 (Chain-of-thought) To be fair to all LLMs, the experiments we conduct (page 7, Section 4) uses the popular minimalist prompt of “let’s think step by step”, which is a form of chain-of-thought prompting. It is unclear how the specific ``chain-of-thought’’ prompting paper cited in the review should be applied to our setting, since the paper depends on few-shot examples of similar reasoning, but the 6 games in SmartPlay differ from each other greatly in terms of game mechanics. In addition, we explicitly avoid heavily engineered prompting frameworks in our experiments like ReAct or Voyager since results with those frameworks could soon be out-of-date given the current speed of the academia.

Reviewer comment (Experiments):
Details on how the "human baseline" was obtained are missing. For example: Did humans play the games through the same API as the LLMs (i.e., via text)? Were they familiar with the environments beforehand? Were there multiple human players? Similarly, how was the zero value defined in Table 2—was it a random agent or a literal 0 score?

Author response:
Q5.1 (human baseline) 3 players (including the authors) who are very familiar with the environments and API played the games through the API. We will add clarification to this in the paper.

Q5.2 (values in table 2). Values in table 2 are normalized according to human baseline with (score-min_score)/(human-min_score). A score of 0 corresponds to achieving the minimum possible score in the game (failing all tasks and performing worse than random). We will add clarification to this in the paper.

Reviewer comment (Evaluation):
**concerns on fairness for different LLM**: The author solely relied on a pre-trained LLM for evaluation in these environments. However, since various language models are trained on different corpora and some remain undisclosed (such as close-source OpenAI language models), it becomes challenging to ascertain whether the LLM has been exposed to these specific environments. For instance, GPT possesses extensive knowledge of Minecraft, whereas LLAMA has relatively limited knowledge in this area. Consequently, ensuring fairness in direct comparisons of evaluations becomes difficult. To address this issue, I recommend that the author gather sufficient data for each environment and evaluate the capabilities of different language models separately using zero-shot learning, few-shot in-context learning, and instruction tuning approaches.

Author response:
W1 (fairness due to LLM knowledge difference): Thank you for pointing out this important concern. Test set contamination is indeed one of the key challenges of LLM evaluation as noted by [1]. SmartPlay attempts to mitigate contamination by 1) featuring 6 different games 2) featuring games (Messenger and Crafter) that are created for research with little online documentation 3) Using games with procedural generation (Messenger, Crafter, Minecraft). 

In addition, games seem to be quite robust to memoization due to their nature of exponential branching states, (in section 4.2) as we experimentally observe with the Tower of Hanoi. Although all LLMs could provide a memoized solution at the starting configuration where all disks are on the first rod (some may even write out the recurrence for the solution), most LLMs could not solve the problem and get confused quickly after a few moves, especially when the disks are distributed over all three rods.

Reviewer comment (Experiments):
**concerns on 3d spatial reasoning performance of LLM**: I noticed that current Large Language Models (LLMs) struggle with 3D spatial reasoning in environments like Minecraft. This could be due to the lack of visual information in their training data, making it difficult to directly apply LLMs to tasks in this dataset. The author attempted to describe visual images using text, but previous experiments have shown this approach to be impractical. Additionally, the source of these visual descriptions was not explained by the author. Can models like LLaVA, GPT-4V, and Flamingo overcome these challenges by being fine-tuned with visual images and other relevant information?

Author response:
W2 (VLMs): Please note that playing games requires accurate object location/interaction perception. The spatial reasoning aspect can go beyond object recognition. Controlling in-game agents requires precise and accurate identification of interactions and changes in the environment [2], current open-source VLMs lack such capabilities. We are hopeful for future VLMs will do better at spatial reasoning aspects than LLMs potentially, but the goal of our work is to show that LLMs are generally weaker at spatial reasoning.

Reviewer comment (Experiments):
**concerns on difficulty computation**: The setting of difficulty for each game seems quite arbitrary. How are these difficulties determined? Are they based on human evaluation? I suggest the author provide clear explanations regarding this.

Author response:
W3 (Difficulty definition): Please see Figure 2 (left table) or Table 3 (Appendix page 14) for how each game challenges different capabilities, and how we define the difficulty levels. For continuous difficulty spectrums like text understanding, reasoning, instruction following, interactions, spatial reasoning, planning, we define the levels by defining difficulty bins. For categorical difficulty spectrums like generalization, randomness, error-handling, we define the levels by enumerating all options. Thank you pointing out this omission, we will include more information on how we defined the difficulty levels in the final paper.

Reviewer comment (Experiments):
**concerns on prompt design for different llms**: Despite instruction tuning, the performance of LLM still relies on prompt design. It is best to conduct explicit experiments and explanations on prompt design.

Author response:
W4 (Prompt design for LLMs): As explained in the general rebuttal (The SmartPlay Interface section), SmartPlay is compatible with any prompting frameworks. 

To be fair to all LLMs, the experiments we conduct (page 7, Section 4) uses the popular minimalist prompt of “let’s think step by step”. 
Although some LLMs might do better at specific types of prompts, such behavior is generally considered undesirable for downstream applications and should not be explicitly accommodated.

New prompting frameworks are invented constantly, and future LLMs may be tuned accordingly. If we include complex prompting frameworks, we may need to constantly update the core evaluation frameworks, making past results incompatible.

Reviewer comment (Experiments):
The author can use **one** simulator to assess the various abilities of the LLM agent, rather than relying on multiple simulators for each ability. For instance, Minecraft offers an open world with numerous engaging tasks that require long-term planning and learning from interactions. However, the author only utilized Minecraft to evaluate the LLM agent's 3D spatial reasoning skills. Another convenient and popular option is a readily installable simulator package that is user-friendly.

Author response:
Q1 (single simulator like Minecraft): In regards to user-friendliness, SmartPlay widely supported OpenAI gym protocol, and could be installed easily with pip. 

In regards to only using one simulator like Minecraft, we note that this will significantly reduce the diversity of the benchmark and increase the risk of test set contamination. In addition, ALL existing LLM models fail on simple navigation tasks in Minecraft, not to mention harder tasks that involve multiple dependencies (i.e., mine diamond), and may depend on navigation. Therefore, it is unclear if a single simulator could capture the challenges at different difficulty levels.

Reviewer comment (Experiments):
**Comparison with Existing Benchmarks**: The paper could be strengthened by comparing SmartPlay’s performance and effectiveness in evaluating LLMs against existing benchmarks or evaluation methodologies, if any are available.

Author response:
Q2 (Existing benchmarks): Apologies for the potential misunderstanding, we are unclear about how the "performance and effectiveness in evaluating LLMs" could be defined and compared. SmartPlay is created to benchmark specific capabilities of *LLMs as Agents* and is currently one of its kind as noted by reviewers itPd, Jri2. Therefore, there is no similar benchmark for direct meaningful comparison.

Reviewer comment (Writing):
There are some typos and table type errors.

Author response:
Q3 (Typos) Thank you for pointing out this important issue. We will proof-read the paper for spelling and correctness.

Reviewer comment (Experiments):
It is unclear which games correspond to which of the nine abilities and to what levels.

Author response:
W1.2 (Game - Ability correspondence) Please see Table 3 (Appendix page 14) for how each game challenges different capabilities.

Reviewer comment (Evaluation):
There is confusion regarding why humans have a score of 1 for all the games and why the davinci-model has a score of 1.04 on bandit.

Author response:
W1.3 (human normalized scoring for Table 2) We normalize the LLM scores by the ratio with human expert score on all games. Therefore, human baseline score is always 1, and text-davinci achieves slightly better score than human expert on bandit, so has a score of 1.04. For un-normalized scores, see Table 4 (Appendix page 18).

Reviewer comment (Evaluation):
There may be concerns regarding the scalability of the benchmark and whether it can keep pace with the rapid advancements in LLMs.

Author response:
W2 (scalability) we note that it is easy to extend SmartPlay with more complicated games as described in the general rebuttal (The SmartPlay Interface section).

Reviewer comment (Evaluation):
The paper might benefit from a more detailed discussion on the implications of these benchmarks in real-world scenarios and how they reflect the complexities of actual agent tasks.

Author response:
W3 (real-world agent tasks) Please note that the field of LLMs as agents has just started to grow. With OpenAI releasing new GPTs models earlier this week, it is unclear how one would predict what type of tasks would eventually be "real-world". Although there has been growing attention on web-browsing, gaming, and VR/AR applications, we note that none of these agent applications are at the scale of real-world agent tasks.

Therefore, we take the perspective of identifying fundamental challenges with agents: text understanding, reasoning with dependencies, instruction following, planning, generalization, understanding and estimating randomness, error/mistake handling, and spatial reasoning. We believe those challenges not only exist in the field of games, but also ALL real-world agent tasks.

Reviewer comment (Reproducibility):
The GitHub repository for the project appears to be empty (https://github.com/LLMsmartplay/SmartPlay).

Author response:
Q1 (Empty GitHub Repo): Our repo is ready but cannot be anonymized due to legal issues. We guarantee public release of (unanonymized) code immediately upon acceptance.

Reviewer comment (Evaluation):
Evaluation metrics proposed in the paper are commonly used in the domain of reinforcement learning (RL). While these metrics are established and provide a common ground for comparison, they may not be entirely suited to capture the unique nuances of planning and reasoning that are specific to LLMs functioning as agents. The paper could be strengthened by proposing or developing novel metrics that are tailored to the particular dynamics of LLMs' operational framework, offering a more precise measurement of their planning, reasoning, and adaptability skills in agent-based scenarios.

Author response:
W1 - Thank you for pointing out this important aspect of LLM evaluation. Indeed, the original reward functions designed for RL algorithms are not the best representations of agent achievements in many games. Therefore, we introduced "game score" (bottom of page 6, section 3.3), which is defined to best represent the accuracy of choices or the progress toward task completion. The normalized game score in table 2 further denotes the gap between human LLM capability and human capability at specific games.
In regards to capability-specific metrics, we believe deeply in the benefit of having capability-specific metrics, but we think that defining capability-specific games and then reporting the "game score" should serve equivalent effect.

Reviewer comment (Experiments):
Is it possible to craft the prompt to show the flow of reasoning and planning when LLMs as agents play the games?

Author response:
W2 - Apologies about the potential misunderstanding. Do you mean: “Using a chain-of-thought prompt?” Yes, as explained in the general rebuttal (The SmartPlay Interface section), SmartPlay is compatible with any prompting frameworks. In addition, the experiments we conduct (page 7, Section 4) uses the “let’s think step by step” prompt to promote explicit reasoning for all LLMs. Because of the explicit reasoning prompt, we were able to make qualitative observation that the LLMs are confused with the Tower of Hanoi game in section 4.2