Paper: Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization (ICLR-2024)

Reviewer comment (Evaluation):
The paper's examples of bad/good prompts do not obviously show that the finetuned reflection model produces better prompts. For instance, in Figure 5, the Reflexion plan is better formatted but still doesn't point out that it was incorrect to return two series rather than one. It would be useful to see an analysis of how often the plan correctly identifies an improved plan (e.g. have a human hand-label 100 prompts for Reflexion and 100 from the frozen LM and record their success rates.)

Author response:
A1: Thanks for the comment! Given the limited time, we’ve manually labeled 5 reflection responses in the HotPotQA dataset, including the “Teen Titans” examples mentioned in the review. We found that with human labeling of the root cause of failure and next plans, the agent LM can always succeed in the next trial. For the reflection responses from the frozen LM, i.e, the Reflexion method, 1 out of 5 tasks succeeds in the next trial and with our Retrofomer fine tuned by policy gradient, under the same content for the remaining prompt, 3 out of 5 tasks succeed. 

For the 100 tasks, one can refer to Figure 4. For example, under GPT-4, at episode 1, Reflexion has 48 successful tasks while Retroformer (r=4) has 51 successful tasks. This means that 8 (48-40) plans of frozen language model identifies an improved plan and 11 (51-40) plans of Retroformer identifies an improved plan, which shows the effectiveness of our approach. 

We appreciate your suggestion for hand labeling, and we will label (the rest of) the reflection responses and add them for fine tuning the Retrospective model, as an ablation model for human-in-the-loop expert demonstrations, similar to the Dagger RL method .

[1] Ross, Stéphane, Geoffrey Gordon, and Drew Bagnell. "A reduction of imitation learning and structured prediction to no-regret online learning." Proceedings of the fourteenth international conference on artificial intelligence and statistics. JMLR Workshop and Conference Proceedings, 2011.

Reviewer comment (Evaluation):
I'm confused how the "f1 score" reward works.

Author response:
A2: Thanks for spotting this! We’ve updated the Appendix C.3 to include the description of reward functions used in 3 environments. To answer your question, I quote the updated descriptions below.

> F1 reward is used in the HotPotQA environment for comparing the matching of a generated answer to a question against the ground truth answer.  After removing the stopwords in both answers, we calculate the number of common tokens in two answers. Then Precision is # of common tokens divided by # of generated answer tokens and the Recall is # common tokens divided by # ground truth answer tokens. We can then compute f1 from precision and recall. 
This reward was defined in the ReAct paper, which is used as the baseline model in this paper.

[2] Yao, Shunyu, et al. "React: Synergizing reasoning and acting in language models." arXiv preprint arXiv:2210.03629 (2022).

Reviewer comment (Experiments):
I'd like to see the following additional curves added to Fig 4 and 6 (possibly in an appendix), which might make it clearer how the finetuning contributes:  
  - Retroformer, rolling out the base policy (before RL finetuning). This would make it easier to see how much of the improvement above Reflexion is due to finetuning vs other algorithmic details.  
  - Retroformer, rolling out the base policy, but with GPT-3 as the reflection model. This would answer the question of whether it's typically a better strategy to finetune a small model or just prompt a more capable cloud model.

Author response:
A3-1: Thanks for the comment! In light of the suggestions, we’ve added the curve Retroformer (GPT-3, lora r=0) to Fig 4 and 6 to denote the base policy Retroformer agent with the frozen longchat-3b as the reflection model (before RL finetuning). The improvement due to finetuning is now entirely represented by the area, for example, of the curve Retroformer (GPT-3, lora r=4) above Retroformer (GPT-3, lora r=0).  

A3-2: Thanks for the suggestions! The Reflexion (GPT-3) curve in Figure 4 and 6 is actually the requested “Retroformer, rolling out the base policy, but with GPT-3 as the reflection model”.(By the way, we believe you meant GPT-4 for the reflection model as a more capable cloud model; please let us know if we are wrong.) 
Therefore, We’ve also tried using GPT-4 as the reflection model, see Reflexion (GPT-4) and found that, for example, Retroformer (GPT-3/4, r=4) still outperformed it by a lot in all environments. This demonstrates that for generating reflective responses in a given environment, fine tuning a small model is more effective than prompting a more capable general-purpose model.

Reviewer comment (Reproducibility):
I'd recommend writing out the exact algorithm you used for PPO finetuning, especially since when PPO is used with language models slight details from the original algo are changed. Also, it's typically an online algorithm, so it would be helpful to detail any changes you made to make it work well in the offline setting.

Author response:
A4: Thanks for pointing it out! We’ve included an algorithm table for the PPO training in Appendix C.1 - Algorithm 1. We’ve also added a description of the three steps.

> The offline PPO algorithm we used for finetuning the Retrospective component in this paper is presented below in \cref{Algo: PPO}. It contains three steps: offline data collection, reward model learning, and policy gradient finetuning. We use the offline ratings data to train a reward model first, and plug in the reward model for PPO finetuning

Reviewer comment (Theory):
Equation 6 seems to be optimizing for each (x, y) pair individually (i.e. not treating a sequence of (x1, y1), (x2, y2) (x3, y3)... as a trajectory). Is this correct? If so, I'd recommend making this clearer in the text.

Author response:
A5: Thanks for carefully reading our paper! The presentation of this equation is actually correct.  $x$ in the equation refers to the instruction prompt, which includes textual descriptions of the agent trajectory, episodic returns, environment feedback at the end of one episode and $y$ denotes the reflective response generated by the Retrospective component taking this prompt as input. Therefore, for fine tuning the Retrospective model (Retroformer’s task), there is only one step in this model’s trajectory. In light of your comment, we’ve updated the presentation to make it clearer!

> (note there is only 1 step in the Retrospective model’s trajectory in its episode)

Reviewer comment (Evaluation):
The improvements brought by Retroformer are limited. There are no significant improvements in HotPotQA and WebShop, only meaningful improvement is observed in AlfWorld.

Author response:
A1: Thanks for the comments. We’d like to discuss them in the following three aspects: (1) Improvements in HotPotQA, (2) Improvements in WebShop, and (3) Improvements against RL methods.

**(1) Improvements in HotPotQA** 

We believe that there are significant improvements in HotpotQA by using Retroformer. For example, as in Table 2, Retrofomer (GPT-3, lora r=4) was improved by 14% in success rate within just one episode. The improvement is also 6% higher than the Reflexion (GPT-3) method that uses a frozen model without our proposed RLHF finetuning. In light of your comments, we’ve also tested GPT-4 as an agent in the updated experiments and found even GPT-4 gets stuck at 54% success rate. We believe that 54% is the upper bound of the success rate under existing LLMs.
It deserves notice that HotPotQA, which is a text-based game, has an extremely large state and action space (you can treat it as a RL problem with a continuous action of 768 dimensions). It should take more than 500 episodes to observe certain improvements in some much simpler text-based environments in [1]. Our improvements (14%) by Retroformer in just one episode are significant. 

[1] Yuan, Xingdi, et al. "Counting to explore and generalize in text-based games." arXiv preprint arXiv:1806.11525 (2018).

**(2) Improvements in WebShop** 

Thanks for carefully reading our paper! However, the difficulty of solving this web shop environment is well known in all recent language agent publications [2,3,4] that use this environment. As we stated in our previous submission, we agree that the performance improvement by Retroformer over the frozen baselines is observed but the improvements may be limited. This limitation of improvement was due to the fact that “web browsing requires a significant amount of exploration with more precise search queries”. Nevertheless, our Retroformer method is still performing better than Reflexion and React across episodes, which use a frozen model without our proposed RLHF fine tuning. In light of your comments, we’ve added GPT-4 agents in the updated experiments and found similar results in the updated Figure 6 (b). 

We’ve explained these results in the updated draft in Section 4 - Web Browsing.
The results probably indicate that the verbal feedback approach (Reflexion, Retroformer) is not an optimal method for this environment, but our fine-tuning method still proves effective. 

[2] Yao, Shunyu, et al. "React: Synergizing reasoning and acting in language models." arXiv preprint arXiv:2210.03629 (2022).
[3] Shinn, Noah, Beck Labash, and Ashwin Gopinath. "Reflexion: an autonomous agent with dynamic memory and self-reflection." arXiv preprint arXiv:2303.11366 (2023).
[4] Liu, Xiao, et al. "Agentbench: Evaluating llms as agents." arXiv preprint arXiv:2308.03688 (2023).

Reviewer comment (Experiments):
The experiments are not solid enough. It lacks comparisons with RL methods that have been recognized to perform well within the same framework of interaction-feedback-learning. Additionally, there is no comparison with the currently acknowledged GPT-4 model, which has impressive decision-making capabilities, making it insufficient to demonstrate the contribution of this work.

Author response:
A2: Thanks for raising the concern! Please refer to the detailed response above for the section Improvements against RL methods. In summary, we’ve added Soft Actor-Critic as the baseline. We run SAC for 5 episodes but do not observe significant improvement across all 3 environments with the updated results in Table 2. 
|     Method |     #Params |     #Retries |     HotPotQA |   |     AlfWorld |   |     WebShop |
|------------|-------------|--------------|--------------|---|--------------|---|-------------|
| SAC        |     2.25M   | N=1          | 27           |   | 58.95        |   | 30          |
|            |             | N=4          | 27           |   | 59.7         |   | 30          |

Reviewer comment (Experiments):
Only the prompt used in the HotPotQA environment is displayed, and it is difficult to determine whether special prompt engineering is needed in different environments. Therefore, it is insufficient to verify the claim of 'automatically tunes the language agent prompts'.

Author response:
A4: Thanks for carefully reading our paper! In light of your comments, we have updated Appendix Section E to include all the prompts used in HotPotQA, Alfworld and Webshop environments. If there is anything unclear, please kindly let us know!

Reviewer comment (Experiments):
The feedback obtained from the interaction between the agent and the environment is usually sparse, and the computed rating $r=G_{k,i+1}-G_{k,i}$ represents the difference in return between two consecutive episodes. This means that the data used for finetuning retrospect models are not significantly different within a certain period. How does Retroformer address the overfitting issue caused by fine-tuning on a large amount of repetitive data?

Author response:
A5: Thanks for the comment. We agree with the reviewer that the computed rating is usually sparse in online settings. This is exactly why we collect the data offline for fine-tuning the Retrospective component. It can also be found in our previous submission in Appendix, that the ratings in the offline dataset are not sparse. In fact, the instruction-response pairs with positive rating account for around 40% and the pairs with negative (sparse) ratings account for 60% in the training set. One could refer to our updated algorithm table in Appendix C.1 for the details on how we collected the data.

> For HotPotQA environment, we collected 3,383 reflection samples by running the base rollout policy for 3 trials (N = 3) for 3,000 tasks in the training set, in which 1,084 instruction-response pairs have positive ratings. For AlfWorld, we collected 523 reflection samples and for WebShop, we collected 267 reflection samples.”

Finally, The queries or task descriptions in the offline training set do not overlap with the ones we use to report the success rate (validation tasks) so there is no repetitive data or overfitting.

Reviewer comment (Experiments):
Are there any differences in the prompts used by Retroformer and baselines methods, and are there experimental results to analyze this part through ablation analysis?

Author response:
A6: Thanks for your question. No, there is no difference in the prompts used by Retroformer, ReAct and Reflexion. This is exactly why the three methods (ReAct, Reflextion, Retroformer) have the same success rate at episode 0. Reflextion is built upon ReAct by adding verbal feedback at the end of the episode. Retroformer further finetunes the Retrospective component by policy gradient. Therefore, the three language agent methods share the same prompt template.

Reviewer comment (Evaluation):
What are the possible reasons for limited performance improvements in HotPotQA and WebShop?

Author response:
A7: Thanks for the questions. As we pointed out earlier, for complex text-based games, due to their extremely huge state-action space, one should not expect the method to solve the game within 5 episodes. For HotPotQA, we believe that 14% in success rate with one episode and 20% improvements within 5 episodes are significant performance improvements. For WebShop, it requires a significant amount of exploration with more precise search queries but our Retroformer method still performs better than Reflexion and React across episodes. The results under Webshop indicate that the verbal feedback approach (Reflexion, Retroformer) is not an optimal method for this environment, but our fine-tuning method still proves effective.  If there is anything unclear, please kindly let us know!

Reviewer comment (Experiments):
How much efficiency is gained by using a retrospective model to tune prompts, rather than directly tuning the LLMs?

Author response:
A8: Thanks for the question. In light of your comments, we added one RL baseline (Soft Actor-Critic) in the updated experiments which directly tunes the LLMs instead of tuning the prompts. For a fair comparison with the language agent methods in this paper, which all showed improvements in 5 episodes, we run SAC for 5 episodes but do not observe improvement across all 3 environments. These results show that tuning the agent prompts are much more efficient than tuning the agent LLMs directly, under our problem setting. 
|     Method |     #Params |     #Retries |     HotPotQA |   |     AlfWorld |   |     WebShop |
|------------|-------------|--------------|--------------|---|--------------|---|-------------|
| SAC        |     2.25M   | N=1          | 27           |   | 58.95        |   | 30          |
|            |             | N=4          | 27           |   | 59.7         |   | 30          |

Reviewer comment (Reproducibility):
Are there any details about the hyperparameters of PPO?

Author response:
A9: Thanks for pointing it out. We have updated in Appendix C.1 the details about the hyperparameters of PPO. We’ve also written out the exact algorithm we used for PPO finetuning in the updated Appendix C.1. Please kindly let us know if anything is not clear.

> The offline PPO algorithm we used for finetuning the Retrospective component in this paper is presented below in Algorithm 1. It contains three steps: offline data collection, reward model learning, and policy gradient finetuning. We use the offline ratings data to train a reward model first, and plug in the reward model for PPO finetuning.