Paper: LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Models (ICLR-2024)

Reviewer comment (Experiments):
LLM with RL/PPO in this complicated task is very sensitive to the hyper-parameters. I did not find the analysis or report on these.

Author response:
> “LLM with RL/PPO in this complicated task is very sensitive to the hyper-parameters. I did not find the analysis or report on these….Any details about the hyperparameters for baselines?”  

We have reported our hyperparameters for training the baselines in Appendix D.

Reviewer comment (Reproducibility):
The Data Generation Approach is a little disappointing. I did not find the details that can replicate this process or any validation of the data. How can I make sure these data can be used to evaluate RL?

Author response:
> “Any validation of the data generation process? How to make sure the game/task you generate is POMDP?”  

To validate the data generation process, we report statistics as can be seen in Table 1. We also provide further details on the dataset generated in Appendix C. For the text-games we ensure it is a POMDP, by creating environments such that the next state is only dependent on the previous state and action. For dialogue tasks, the state can be thought of as a concatenation of all previous utterances. With this definition, this situation will also be a POMDP.

Reviewer comment (Experiments):
The baseline results are not convincing enough. PPO Instabilities in GPT-2-small are very obvious. I do not think the results in the benchmark should only use GPT-2-small. Any technique you use here to overcome it?

Author response:
> “The baseline results are not convincing enough. PPO Instabilities in GPT-2-small are very obvious. I do not think the results in the benchmark should only use GPT-2-small. Any technique you use here to overcome it?”  

We overcome instabilities in PPO by 1) increasing the number of rollouts 2) tuning the KL coefficient and 3). Regarding using GPT2-small, we were limited by compute and simply cannot train 7B+ models on all these settings. However, for the models we use as the environment, we have used GPT2-XL for the Car Dealer task.

Reviewer comment (Reproducibility):
Any details about the hyperparameters for baselines? Thanks.

Author response:
> “LLM with RL/PPO in this complicated task is very sensitive to the hyper-parameters. I did not find the analysis or report on these….Any details about the hyperparameters for baselines?”  

We have reported our hyperparameters for training the baselines in Appendix D.

Reviewer comment (Reproducibility):
How to replicate the data generation process?

Author response:
> “How to replicate the data generation process?”  

The data generation process can be replicated by using the GPT2 checkpoints provided to regenerate the data. We also provide our prompts and data generation scripts in our code base, for which we have provided an updated version.

Reviewer comment (Experiments):
How did you evaluate the quality of your data?

Author response:
> “How did you evaluate the quality of your data?”  

We evaluate the quality of our data by 1) inspecting the generations for signs of problems 2) performing a human evaluation where humans interact with the simulator model that generated the data. If humans are able to successfully interact with the models, it is a clear signal that our data is also natural and contains the desired properties. Additionally, we have found better performance from our algorithms compared with the BC models, signaling that the data the simulator was trained on is providing a useful signal for the improvement of RL algorithms.

Reviewer comment (Experiments):
I think I might miss it. You use the data from the GPT-3.5 and create several games for the 8 tasks?

Author response:
> “You use the data from the GPT-3.5 and create several games for the 8 tasks?  

Yes, for our three dialogue tasks, we do the following: 1) Generate data with GPT-3.5 based on specific prompts (for an example for Twenty Questions, see Appendix, Section C.6) 2) Distill this to two GPT2 models one for the agent, one for the environment 3) Use these distilled models to generate a new dataset that we will use for training our algorithms. See Section 4.4 of our paper for further information on data generation and reference Figure 2.

Reviewer comment (Evaluation):
While the state and action spaces are language-based, it is not clear the extent to which some of the environments assess performance on language-based tasks, such as the maze, chess, and chess endgames. While the environments assess reasoning abilities, they do not assess the full complexity of reasoning abilities a LLM needs nor abilities that are key to LLM success. The reasoning abilities are more general abilities we would want from a RL-train agent. For example, it is important for LLMs to be factual and harmless, which means correcting behavior learned during pretraining.

Author response:
It's definitely a valid point that not all the tasks test performance on realistic language-based tasks, though some do. However, each of the tasks in the benchmarks serves a different purpose. Some tasks (Guess My City, Car Dealer) aim to evaluate tasks with realistic natural language. Some tasks aim to test specific RL properties without the complexities of realistic language, while others focus on complex language. Algorithms developers would be expected to evaluate their methods on the totality of all the tasks. We've expanded Section 4.3 to clarify this. We do not believe that this is a weakness, as our benchmark does include a number of tasks with realistic language, and it seems valuable to test a variety of RL algorithm capabilities in isolation (and comparatively less valuable to have redundant tasks that all test similar things). However, we hope that the revised discussion in Section 4.3 more clearly explains this rationale.

Reviewer comment (Reproducibility):
It is not clear what safeguards are in place to prevent hacking of the LLM that is part of the environment.

Author response:
We thank the reviewer for this question. We clarify the methodology with which we generated our datasets to train our simulators. As shown in Figure 2, we train simulators that serve as an “oracle” for the task, which defines the ground truth environment. In most cases, our simulator is performing an easier task than the agent, which does not require strategic reasoning. Additionally, our tasks are designed such that any noise in our environment will not necessarily break the task, but merely add an additional inference challenge for the agent. For example, the role of the oracle in the Twenty Questions task is to provide objective yes/no answers to questions about the object, and in Guess My City, to provide more open-ended information about a query on the city. OpenAI’s GPT-3.5 has been shown to be able to generate reasonable questions and answers when used out of the box, which is why we leveraged it to collect our initial dataset. We have provided prompts that we use to generate the data to train our oracle models in our Appendix, Section C.6, and snippets below to show our thought process to maintain high accuracy. 

With respect to the Car Dealer task, we spent a considerable effort to ensure diversity in the responses of sellers, by providing different desired brands, features, classifications (i.e. car or truck), and budgets in our prompting to generate the datasets. We provide a sample of conversation between the oracle model and MC returns vs. oracle and the BC model to illustrate in our Appendix, Section H.

Reviewer comment (Evaluation):
To what extent does the policy's actions on the environments like 20Qs, car dealer, and guess my city look like reasonable sentences?

Author response:
We find that all of the policy’s actions on the environment appear to be reasonable sentences. We have verified this by looking at the generated datasets and interacting with the environments ourselves, as well as performing a user study of having humans evaluate if the text used to train the simulator from GPT and from one of our models (MC returns) was natural on a scale of 1-5 (where 5 is most natural). On average, we found humans to say the text from GPT-4 was 55.56% natural, and from our model was 58.51% natural. We provide detailed statistics and explanations of the user study evaluating naturalistic language in our Appendix, Section J. We are also currently conducting a larger human evaluation study of humans interacting with our models as a baseline, and provide partial results in to Table 2 in our paper with some discussion. We also show sample interactions of our BC and MC policy model with the oracle in Appendix, Section H for the Car Dealer Task.

Reviewer comment (Presentation):
For some of the environments the specific skills addressed are called out. It would be great to highlight the assessed skills for each environment.

Author response:
We agree that it is important to highlight the capabilities enabled by every task. As per your suggestion, we have updated our draft with Section 4.2.1 and an additional Figure 3 to more clearly highlight which tasks enable which RL capability and why.  For example, the Maze and Text-Nav tasks contain both partially observed and fully observed versions to highlight the impact of partial observability. In addition, the Text-Nav task is very similar to the Maze task, but has the added complexity of learning to generate realistic text.>

Reviewer comment (Experiments):
Are there other LLM-specific policy learning algorithms that can be assessed?

Author response:
Yes, since our submission, we have performed a new evaluation experiment by few-shot prompting GPT-4. This is a LLM-specific policy learning algorithm because only LLMs can be few-shot prompted in this way. In addition, we have run an ablation with Online Filtered BC, which is an algorithm that collects data using the current policy and selects the most successful trajectories for fine-tuning. This is an algorithm that is often used in RLHF pipelines. In general, we have found that these evaluations did not outperform our previous baselines. We include these results in our updated results section.

Reviewer comment (Evaluation):
To what extent are the performance differences between RL algorithms proportional to their differences on non-language based tasks?

Author response:
This is an excellent question and one that we investigated by creating a non-text based version of the maze task. We found that simple online and offline Q-learning was able to get an optimal score on the maze. Therefore, performance symbolic maze is comparable to the fully observed Maze task. However, on the partially observed Maze task, the language based methods perform significantly worse. This highlights room for improvement on dealing with partial observability in RL with language. We have added these details in our updated revision in the Appendix, Section G.

Reviewer comment (Experiments):
The benchmark is artificial and quite less natural. Only two of the eight tasks (Car Dealer and Guess My City) is multi-turn conversation or conversational QA with more unbounded state and action space. All other tasks have obvious restrictions, especially, the Chess and Endgames tasks are not natural language generation to me.

Author response:
> “The benchmark is artificial and quite less natural. Only two of the eight tasks (Car Dealer and Guess My City) is multi-turn conversation or conversational QA with more unbounded state and action space. All other tasks have obvious restrictions, especially, the Chess and Endgames tasks are not natural language generation to me.”

It's definitely a valid point that not all of our tasks have realistic natural language, though some do. However, each of the tasks in the benchmarks serves a different purpose. Some tasks (Guess My City, Car Dealer) aim to evaluate tasks with realistic natural language. Some tasks aim to test specific RL properties without the complexities of realistic language, while others focus on complex language. Algorithm developers would be expected to evaluate their methods on the totality of all the tasks. We've expanded Section 4.3 to clarify this. We do not believe that this is a weakness, as our benchmark does include a number of tasks with realistic language, and it seems valuable to test a variety of RL algorithm capabilities in isolation (and comparatively less valuable to have redundant tasks that all test similar things). However, we hope that the revised discussion in Section 4.2.1 more clearly explains this rationale.

Reviewer comment (Evaluation):
Generated examples by the trained agents can be added. Although the authors have put emphasis on “our goal is not to utilize this approach to benchmark whether LLMs are good at talking to humans, but rather as a way to test RL algorithms with datasets that are sufficiently difficult and complex so as to gauge how effective they might be if they were then trained on data from real humans.“ in Introduction, it is good to see how relevant the obtained rewards and the conversation's naturalness are.

Author response:
> “Generated examples by the trained agents can be added. Although the authors have put emphasis on “our goal is not to utilize this approach to benchmark whether LLMs are good at talking to humans, but rather as a way to test RL algorithms with datasets that are sufficiently difficult and complex so as to gauge how effective they might be if they were then trained on data from real humans.“ in Introduction, it is good to see how relevant the obtained rewards and the conversation's naturalness are.”

Thank you for this question. Rewards for the tasks are simple and intuitive, because they are based on success of selling the car or guessing the word. To address your concern regarding the naturalness of the conversation, we have provided a few examples in the Appendix, Section H. In addition, we have evaluated all of our tasks with human evaluators playing the games to assess how natural they are, and show an example in Appendix, Section I.

Reviewer comment (Writing):
The writing of section 4.1 can be further improved. For now, I can guess the meaning from the terminologies themselves, but I will get confused from the content in each paragraph trying to explain the corresponding terminology.

Author response:
> “The writing of section 4.1 can be further improved. For now, I can guess the meaning from the terminologies themselves, but I will get confused from the content in each paragraph trying to explain the corresponding terminology. … Some details can be moved from Appendix B to section 4.3. For example, the desired capability for every task. For now some are left in the appendix.”

In order to make Section 4.1 clearer, we have improved the definitions of the RL capabilities. We have added Section 4.2.1 and an additional Figure 3 to more clearly highlight which tasks enable which RL capability and why.

Reviewer comment (Writing):
Does the timestep in Table 1 mean for one response instead of for one token, and the length mean the number of turns in a conversation? If so, the wording of timestep and length may be a bit confusing with the terminologies in single-turn text generation, where the timestep often refers to the step of generating a token and the length often refers to how many tokens per response.

Author response:
> “Does the timestep in Table 1 mean for one response instead of for one token, and the length mean the number of turns in a conversation? If so, the wording of timestep and length may be a bit confusing with the terminologies in single-turn text generation, where the timestep often refers to the step of generating a token and the length often refers to how many tokens per response.”

Yes, we acknowledge that this is confusing. By “timestep” we mean “one response by the agent” and length is “number of responses by the agent”. We have updated the caption in Table 1 and definitions in Section 3 to clarify this. As we are attempting to do something that has not been explored much before, working out the proper terminology is tricky. Thank you for your feedback!

Reviewer comment (Presentation):
Some details can be moved from Appendix B to section 4.3. For example, the desired capability for every task. For now some are left in the appendix.

Author response:
> “The writing of section 4.1 can be further improved. For now, I can guess the meaning from the terminologies themselves, but I will get confused from the content in each paragraph trying to explain the corresponding terminology. … Some details can be moved from Appendix B to section 4.3. For example, the desired capability for every task. For now some are left in the appendix.”

In order to make Section 4.1 clearer, we have improved the definitions of the RL capabilities. We have added Section 4.2.1 and an additional Figure 3 to more clearly highlight which tasks enable which RL capability and why.

Reviewer comment (Writing):
Typo: Section 3.1 POMPDP -> POMDP

Author response:
We have also corrected the typos the reviewer has found and updated our paper. We thank the reviewer for their helpful feedback in improving the paper, and hope that you find these reasons convincing in raising your score!

Reviewer comment (Experiments):
The reasoning and generalization required on each task needs to be formalized more clearly. What inferences must the model make? How different are test vs training distributions? This formalization will further help understand how the complexity of tasks varies across different tasks.

Author response:
> “The reasoning and generalization required on each task needs to be formalized more clearly. What inferences must the model make?”

In Section 4, we delineate the capabilities enabled by RL, including strategic decision making, complex language, credit assignment, partial observability, and trajectory stitching. We have added Appendix A to formalize the reasoning required for each task. To address your question, we have added Section 4.2.1 and an additional Figure 3 to more clearly highlight which tasks enable which RL capability and why.

Reviewer comment (Experiments):
An alternative to improve interaction and performance at tasks is to use prompting and factoring of LLM calls. I found a discussion on prompting vs RL training as alternatives to each other missing.

Author response:
> “An alternative to improve interaction and performance at tasks is to use prompting and factoring of LLM calls. I found a discussion on prompting vs RL training as alternatives to each other missing.”

As per your suggestion, we have included an additional experiment for few-shot prompted GPT-4 against all of the tasks. We have found that prompted GPT-4 was not able to outperform our baselines on the tasks. We have updated Section 6 of our paper with further details and discussion.

Reviewer comment (Evaluation):
For complex tests, 20 questions, guess the city, car dealership where the model relies on GPT-3.5 for the initial test and GPT-2 for the env, there need to be evaluations of the text produced by the models.

Author response:
> “For complex tests, 20 questions, guess the city, car dealership where the model relies on GPT-3.5 for the initial test and GPT-2 for the env, there need to be evaluations of the text produced by the models.”

To address the concerns regarding the evaluation of text produced by the models, we have performed an additional evaluation of having humans evaluate if the text from both GPT and from one of our models (MC returns) was natural on a scale of 1-5 (where 5 is most natural). On average, we found humans to say the text from GPT-4 was 55.56% natural, and from our model was 58.51% natural. We provide detailed statistics and explanations of the user study evaluating naturalistic language in our Appendix, Section J. We are also currently conducting a larger human evaluation study of humans interacting with our models as a baseline, and provide partial results in our updated paper. 

Additionally, we would like to clarify the methodology with which we generated our datasets to train our simulators. As shown in Figure 2, we train simulators that serve as an “oracle” for the task, which defines the ground truth environment. In most cases, our simulator is performing an easier task than the agent, which does not require strategic reasoning. Additionally, our tasks are designed such that any noise in our environment will not necessarily break the task, but merely add an additional inference challenge for the agent. For example, the role of the oracle in the Twenty Questions task is to provide objective yes/no answers to questions about the object, and in Guess My City, to provide more open ended information about a query on the city. OpenAI’s GPT-3.5 has been shown to be able to generate reasonable questions and answers when used out of the box, which is why we leveraged it to collect our initial dataset. We have provided prompts that we use to generate the data to train our oracle models in our Appendix, Section C.6, and snippets below to show our thought process to maintain high accuracy. 

With respect to the Car Dealer task, we spent a considerable effort to ensure diversity in the responses of sellers, by providing different desired brands, features, classifications (i.e. car or truck), and budgets in our prompting to generate the datasets. We provide a sample conversation between the oracle model and MC returns vs. oracle and the BC model to illustrate in our Appendix, Section H.

Reviewer comment (Reproducibility):
In wordle, I’d love to see a discussion of how the authors deal with the tokenization. Since the models are bad at tasks where individual characters need to be tokenized.

Author response:
> “In Wordle, I’d love to see a discussion of how the authors deal with the tokenization. Since the models are bad at tasks where individual characters need to be tokenized.”

We have added additional details regarding experiments for tokenization in Wordle in our Appendix, Section C.3.

Reviewer comment (Experiments):
Why did the authors not choose to start with human data in domains where that was available? Eg: Craigslist dataset, Deal or No Deal.

Author response:
> “Why did the authors not choose to start with human data in domains where that was available? Eg: Craigslist dataset, Deal or No Deal”

Regarding why we did not choose to start with human data in domains that were available for the following reasons: 1) These datasets do not come with simulators that are faithful and produce data that is similar to the dataset 2) These datasets are not designed for RL and do not have clear and intuitive reward functions. Past works required various special choices to use them, for example, Verma et. al replaced the prices in the Craiglist dataset dialogue and used GPT to generate templates to substitute such prices, as they are not very realistic. Overall, we have found that these datasets do not encourage strong bargaining capabilities. 

Works Cited: 

[1] Verma, Siddharth, et al. "Chai: A chatbot ai for task-oriented dialogue with offline reinforcement learning." arXiv preprint arXiv:2204.08426 (2022).

Reviewer comment (Evaluation):
There are no error bars for the RL methods. The authors should report at least 5 runs.

Author response:
> “There are no error bars for the RL methods. The authors should report at least report 5 runs.”

We did not include training error bars for our methods due to compute constraints, as it may take up to 10 days to train these algorithms. However, we did do multiple runs for many tasks with different hyperparameters, which we show in the Appendix. Additionally, we do multiple rollouts for our evaluation, which is also displayed in the Appendix.