Paper: Can Language Agents Approach the Performance of RL? An Empirical Study On OpenAI Gym (ICLR-2024)

Reviewer comment (Writing):
I would encourage the authors to avoid assertive and absolute claims, such as the first sentence "large language models lack grounding in external environments", and at the bottom of page 1, "we adopt `OpenAI Gym`, the most classic and prevalent...". These arguments, though eye-catching, are debatable, and such debates are irrelevant to the central points that the authors are trying to make, so why not avoid them?

Author response:
Thank you very much for your valuable comments! We have polished our text and avoided the over-claims.

Reviewer comment (Writing):
In the final paragraph on page 2, "after *sufficient* numerical experiments..." I suppose that it should be left for the readers to decide whether the experiments are sufficient.

Author response:
Thank you very much for your valuable comments! We have polished our text and avoided the over-claims.

Reviewer comment (Presentation):
The Problem Formulation section is somewhat redundant, as none of the math symbols are reused anywhere in the main text.

Author response:
Thanks for your advice! We have moved it to Appendix.

Reviewer comment (Reproducibility):
It remains unclear to me (after reading the Appendix) how different levels of agents are implemented. It would be useful to provide at least a brief description.

Author response:
We are apology for the missing description. We have added pseudocode for them in the revised version.

Reviewer comment (Presentation):
The methodology of the paper is not very clear, as the algorithm proposed by the authors is not introduced formally but in charts, making it challenging to understand the algorithm or why it works.

Author response:
Thank you very much for your valuable advices! We have improved our presentation of the algorithm and $5$ levels.

Reviewer comment (Experiments):
The five training scenarios are not described in enough detail, and it is not clear why L3 performs the best out of the setups.

Author response:
Thank you very much for your valuable advices! We have improved our presentation of the algorithm and $5$ levels.

Reviewer comment (Reproducibility):
The experiments are not presented with enough detail, including the omission of information on the number of seeds used to evaluate the agents.

Author response:
Thank you very much for your questions! 

1. we have extended our evaluations to $5$ seeds. 
2. PPO and SAC performances are added. 
3. The threshold description is moved to the main text.

Reviewer comment (Presentation):
The learning curves or the converged performance of the PPO/SAC agents are not reported in the main text but only in the Appendix.

Author response:
Thank you very much for your questions! 

1. we have extended our evaluations to $5$ seeds. 
2. PPO and SAC performances are added. 
3. The threshold description is moved to the main text.

Reviewer comment (Evaluation):
The threshold for determining that a task is solved is not defined in the main text.

Author response:
Thank you very much for your questions! 

1. we have extended our evaluations to $5$ seeds. 
2. PPO and SAC performances are added. 
3. The threshold description is moved to the main text.

Reviewer comment (Evaluation):
If SAC can solve tasks that PPO cannot solve in the described setup, it is unclear why the PPO scores are reported for the majority of the tasks instead of the SAC scores.

Author response:
Thank you very much for your questions! SAC performances are added. There are also tasks that SAC can not solve.

Reviewer comment (Writing):
The partial observability of some environments, such as Acrobot and Cartpole, is unclear and depends on how the states are translated into text; if they are indeed partially observable, this should be clarified in the text.

Author response:
This is a very interesting and important question! It is true that not all of the environments are partially observable. We have summarized the challenges for each environment in Appendix C.1 We have altered the informal description "non-intuitive" to "complex dynamic". For instance, it is easy to reason that the cart will move right if we push right when the cart speed is $0$. We call this dynamic simple. For `Acrobot`, it is hard to reason how the two links will behave when applying $-1$ torque to the actuated joint.

Reviewer comment (Evaluation):
The authors state that Language Agents perform better in more intuitive and deterministic environments, yet environments like Acrobot and LunarLander are deterministic (after the random first state) and Blackjack, which is stochastic, can be solved well by EXE according to the Radar plot; the meaning of "intuitive" in this context should be clarified.

Author response:
This is a very interesting and important question! It is true that not all of the environments are partially observable. We have summarized the challenges for each environment in Appendix C.1 We have altered the informal description "non-intuitive" to "complex dynamic". For instance, it is easy to reason that the cart will move right if we push right when the cart speed is $0$. We call this dynamic simple. For `Acrobot`, it is hard to reason how the two links will behave when applying $-1$ torque to the actuated joint.

Reviewer comment (Evaluation):
The motivation for the proposed algorithm is unclear, and the evaluation of the different agents is not very thorough, leading to the recommendation for rejection in its current state.

Author response:
Thank you very much, it is a really good suggestion. In the revised version:
1. We take `CliffWalking` as our motivating example. 
2. We make further analysis on EXE, especially comparing it to Reflexion.

Reviewer comment (Writing):
The paper could be improved significantly by better motivating the algorithm and a specific recommended training scenario.

Author response:
Thank you very much, it is a really good suggestion. In the revised version:
1. We take `CliffWalking` as our motivating example. 
2. We make further analysis on EXE, especially comparing it to Reflexion.

Reviewer comment (Experiments):
The evaluation of standard RL algorithms like PPO and SAC could be performed more thoroughly.

Author response:
Thank you very much, it is a really good suggestion. In the revised version:
1. We take `CliffWalking` as our motivating example. 
2. We make further analysis on EXE, especially comparing it to Reflexion.

Reviewer comment (Evaluation):
It would be valuable to understand why EXE works better than the Language Agents.

Author response:
Thank you very much, it is a really good suggestion. In the revised version:
1. We take `CliffWalking` as our motivating example. 
2. We make further analysis on EXE, especially comparing it to Reflexion.

Reviewer comment (Presentation):
Some details in the presentation are confusing or unnecessary. As a significant example, why should one highlight that GPT-4 generated the text version of the environment, if that one is not a feature that is being scientifically evaluated? As of now, it reads an attempt to yield to GPT-4 the responsibility concerning the correctness of the new environments that are being proposed. But this is totally irrelevant for the matter of the presentation of the scientific evaluation of a system: regardless of whether the code for evaluating the AI agents was generated by GPT-4 writing code, humans writing code, or copied from a random repository on the web, the only thing that matters is its correctness and alignment with the claims that the rest of the paper is trying to make.

Author response:
Thank you very much for your advices. We have moved it to the appendix and taken a more compact way to present it. 
However, we also want to clarify that the results that GPT-4 is able to generate a translator based on the documentation and template is very helpful for researchers who want to use language agents as alternatives to PPO.

Reviewer comment (Writing):
Can you remove any mention to GPT-4 automatically generating environments?

Author response:
Thank you very much for your advices. We have moved it to the appendix and taken a more compact way to present it. 
However, we also want to clarify that the results that GPT-4 is able to generate a translator based on the documentation and template is very helpful for researchers who want to use language agents as alternatives to PPO.

Reviewer comment (Experiments):
Why did you use PPO for some environments and SAC for some others?

Author response:
Thank you very much for your question! For `MountainCarContinuous`, PPO fails to generate expert-level performance trajectories due to the exploration challenge in `MountainCarContinuous`. PPO gets easily to stuck in sub-optimal strategy (not move) even with grid searching hyper-parameters. Although our main focus is the comparison of PPO, the infeasible expert trajectory will make the `Lv4` scenario infeasible. Thus we take SAC to generate the trajectories and we take it to construct the L4 scenario. See also in the **Response for reviewer DgbS.**

Reviewer comment (Experiments):
Lack of many seeds for a more thorough evaluation is a main weakness. The paper mentions in Appendix B.3 that only one seed is used for evaluation (except 100 for BlackJack), which feels too little given that the kinds of environments in there do not seem expensive.

Author response:
We have used 5 seeds to make evauation of Language Agents in our revised manuscript. While hyperparameter tuning is not a concern for our experiments, the decision-making process with LLM-based agents is inherently resource-intensive. This restricts the scale of our experiments. For additional context, see **Response to all reviewers: Q3**.

Reviewer comment (Evaluation):
The analyses in sections 5.2 and 5.3 seemed to lack in detail.

Author response:
We have streamlined the presentation for code generation and relocated the GPT-related content to the appendix to allow for a more concise main text. In Experiment section, we have added more analysis and many case studies are provided for each environment in the Appendix D.

Reviewer comment (Reproducibility):
Could the authors elaborate on the compute costs?

Author response:
For detailed information on the compute and time costs, please refer to our **Response to all reviewers: Q4**.

Reviewer comment (Evaluation):
The insights and analyses in sections 5.2 and 5.3 seem too high-level. For example, in Fig. 3b, L3 agents did better than L5 and L4 agents, L4 was very bad even though it is supposed to be the second easiest scenario, and L2 and L1 are sometimes better than L3 and L5 agents even though these are harder. The reasons for such unintuitive results could be discussed in much more detail. Possible reasons for lack of depth may include a substantial portion of the paper being dedicated to presenting the benchmark and important information (e.g., Fig. 5 and the code template in listing-1) being placed in the Appendix, suggesting that perhaps a journal format would allow for more detail. Another possible reason is the lack of running many seeds, leading to noisier insights. Additionally, the first key finding at the end of section 5.3 seems to be a claim without proper evidence that EXE's exploration and exploitation capabilities helped it perform better. Could the authors elaborate?

Author response:
We have streamlined the presentation for code generation and relocated the GPT-related content to the appendix to allow for a more concise main text. In Experiment section, we have added more analysis and many case studies are provided for each environment in the Appendix D.

Reviewer comment (Writing):
In the problem formulation section 2.1, it is unclear what the index i is for.

Author response:
The issue with the index \( i \) in the problem formulation section will be clarified and we now move the formulation section to the appendix.

Reviewer comment (Novelty):
I think the general perspective of the paper is confusing. This is reflected in the paper title already: The authors ask if language agents is competitive to RL, but language-based agent and RL-based agent are not competing methods at all. In fact, the Lv3 language agent discussed in this paper -- which seem to be the only working language agents, according Fig. 4a -- is exactly a reinforcement learning procedure that improves decision policy from evaluative feedbacks collected from environmental interaction. You can't beat "RL" with a RL agent.

Author response:
Thank you for your insightful suggestions. We have repositioned our paper to explore the potential of language agents as an alternative to PPO, which is reflected in the updated title: "Can Language Agents Be Alternatives to PPO? A Preliminary Empirical Study On OpenAI Gym". This revision also addresses the concern of "beat RL with RL".  
See details in Common Response Q1.

Reviewer comment (Experiments):
Even if we re-position this paper as just a comparison between two specific methods (EXE vs PPO) on some specific simple tasks as a preliminary study, the current experiment setup may be still a bit biased. For example, I suspect the model size in PPO is orders-of-magnitude smaller than the GPT-3.5 in the language agents.

Author response:
We direct the reviewer to **Response to all reviewers: Q2** for a more detailed explanation.  

For details on PPO, please see Appendix C.4 in our revised manuscript. In our experiments, we utilized Tianshou in almost every environment and `Taxi-v3` with stable-baselines3. These were trained and evaluated in OpenAI Gym, not TextGym. Regarding model size, our policy model has 8,964 trainable parameters when there are 3 actions and the input dimension is 1. This figure is the sum of elements across all parameter tensors that require gradient computation.

Reviewer comment (Experiments):
Task-specific heuristics is provided (only) to the language agents.

Author response:
We've included additional scenarios in the Appendix D.2, where EXE's performance in environments such as Cartpole, MountainCar, and MountainCarContinuous was notably lower without the basic game description. This demonstrates that EXE's effectiveness may be constrained when essential game details are not provided.

It's important to note that the game description is not crafted to enhance agent performance artificially; it typically exists as part of the task's framework. If a simulator for a task has been developed, it is reasonable to assume that a description of the game would be available. Human knowledge at Level 5 (L5) differs significantly from a mere game description; it is often constructed to intentionally boost the performance of Language Agents. For instance, in Cliffwalking, humans might outline the cliff locations, the goal, and even provide demonstrations for Language Agents—information that may not be readily available for a new task.

Reviewer comment (Experiments):
The chosen PPO implementation seems to have weird result on some tasks such as MountainCarContinuous.

Author response:
For MountainCarContinuous, we conducted a comprehensive grid search covering various hyperparameters:

- Learning rate: `{1e-3, 1e-4, 1e-5}`
- Discount factor: `{0.99, 0.95, 0.9}`
- Weight for entropy loss: `{0.01, 0.05, 0.1}`
- Number of repetitions for policy learning: `{10, 20}`

Despite these efforts, the performance issues persisted. A significant challenge in this environment is the substantial penalty received by the agent when failing to reach the target, which often leads to a strategy of inactivity as a quick convergence solution. We also tested the implementation of PPO using stable baselines3.

It is worth mentioning that our goal was not to fine-tune PPO to achieve superior performance, as extensive parameter searching can equate to expert guidance akin to the L5 scenario. Achieving excellent performance through extensive effort (akin to prompt designing) is possible but usually not the preferred approach.

Reviewer comment (Novelty):
In terms of novelty, the paper may need to better contrast with prior works, especially with Reflexion (Shinn et al. 2023) which is similar to the proposed EXE method. Current appendix A.1 is not enough in this regard.

Author response:
Firstly, we have provided pseudocode for both EXE and Reflexion in the Appendix to elucidate their differences and the novelty of EXE. There are three main distinctions: 
1. EXE can generate suggestions without prior experience; 
2. The critic in EXE is a language model that evaluates the trajectory based on suggestions guiding the actor's sampling process;
3. EXE specifically considers insights, exploration, exploitation, and their balance, whereas Reflexion focuses solely on score improvement.

We also added case studies between them in Experiment section 4.4, with a more detailed analysis presented in the Appendix D.

Reviewer comment (Reproducibility):
The current presentation leaves many things unclear. Experimentation code is not released, which is not ideal as an "empirical study" that proposes a new "benchmark".

Author response:
Our code has been open-sourced.