Reflect- RL : Two-Player Online RL Fine-Tuning for LM s (2024)

Abstract: As language models (LMs) demonstrate their capabilities in various fields, their application to tasks requiring multi-round interactions has become increasingly popular. These tasks usually have complex dynamics, so supervised fine-tuning (SFT) on a limited offline dataset does not yield good performance. However, only a few works attempted to directly train the LMs within interactive decision-making environments. We aim to create an effective approach to fine-tune LMs with online reinforcement learning (RL) in these environments. We propose Reflect-RL, a two-player system to fine-tune an LM using SFT and online RL, where a frozen reflection model (player) assists the policy model (player). To generate data for the warm-up SFT stage, we use negative example generation to enhance the error-correction ability of the reflection model. Furthermore, we designed single-prompt action enumeration and applied curriculum learning to allow the policy model to learn more efficiently. Empirically, we verify that Reflect-RL outperforms SFT and online RL without reflection. Testing results indicate GPT-2 XL 1.56B fine-tuned with Reflect-RL outperforms larger open-source LMs, such as Mistral 7B. The benchmarks, dataset, and code involved in this work are publicly available: https://github.com/zhourunlong/Reflect-RL.

## introduction large language models (llms) have shown promising results in problem-solving, coding, and document retrieval @xcite . while performing these ˚university of washington. part of this work done when runlong was an intern at microsoft research, redmond. email: vectorzh@cs.washington.edu ## preliminaries notations. for any set x , we use ∆px q to denote the probability simplex over x . let the tokenizer be fixed throughout the paper. for a string s, we use |s| to denote the number of tokens in s after using this fixed tokenizer. ## benchmarks motivated by the llf-bench @xcite , we have created a natural language environment base class (natlangenv) that is compatible with the ope-nai gym framework, characterized by its unique approach of utilizing textual representations for both observations and actions. this adjustment allows us to effectively train and test language models. autoexplore. to verify our methodology of reflect-rl on the exploration example mentioned in section 1.1, we built a complete benchmark for autonomous exploration. this benchmark contains three components: a autoexploresandbox for file protection, a multi-agent system autoexplorecopilot for interactive decision-making, and a labeled dataset for performance assessment. the @xcite @xcite , gpt-2 xl fine-tuned with baselines, and with reflect-rl. react and memory mechanism, as shown in figure 1 , have been incorporated to improve performance. for conciseness, we have not performed prompt optimization for the open-source models, and their performance could potentially be improved with different prompting techniques in the future. explanation for baselines: "sft`rl (w/o reflection)" means the policy model is the only model involved, and the reflection field is removed from sft data. "sft`rl (w/o negative)" means there are no negative examples in sft data, so both the reflection model and the policy model are trained on expert demonstrations. we only ran this ablation on autoexplore. explanation for tasks: for autoexplore, we tested on 44 user queries, each with 10 runs. "depth i" includes the tasks with target file depth exactly i. for dangeroustaxi, we ran on 100 random maps. "pickup" computes the success rate of picking up the passenger, and "`dropoff" computes the overall success rate. for alfworld, we tested on 4 tasks, each with 25 runs. during interaction with autoexplorecopilot, each step the agent receives ´1 reward as the cost of time. after 15 steps or the agent explicitly terminates, if the correct file is identified, a `15 reward is given; otherwise a ´15 reward is given. dangeroustaxi. we extended the openai gym's taxi environment to introduce a higher level of challenge, thereby creating the "dangeroustaxi " environment. this game concludes prematurely if the player commits any invalid action, such as colliding with a wall, or incorrectly picking up or dropping off passengers at unauthorized locations. this modification crucially elevates the task's difficulty by eliminating the opportunity for the model to correct its mistakes after a wrong decision-a common allowance in the standard environment. we applied curriculum learning to dangeroustaxi. in the designed pickup curriculum, we assign a positive reward 20 and terminate the environment after the driver successfully pickup the passenger. in the dropoff stage, the pickup reward is retained, but the driver needs to further dropoff the passenger at destination to receive the full reward. alfworld. our study leverages alfworld @xcite , a multi-turn platform tailored for simulating household tasks by converting the graphical representation of a house into descriptive language. a robot in is required to complete certain tasks based on the descriptions. this benchmark has gained recognition to evaluate llm agents, with studies like arabzadeh et al. (2024) demonstrating its efficacy. our focus on the tomato picking task stems from its optimal mix of simplicity and representativeness. ## experimental results to verify our approach, we apply reflect-rl on gpt-2 xl @xcite . ## discussion and conclusion risk, impact, and responsible ai. in this study, we adhere to principles of responsible ai by ensuring transparency, efficiency, and security in both the training and evaluation stages. an exemplar of our commitment is the development of autoexploresandbox, designed to reduce the risk of security issues in the file system. recognizing the importance of ethical considerations and the social impact of our work, we pledge to engage in continuous evaluation of lms's performance in multi-step environment. limitations. our study, while comprehensive, acknowledges certain limitations. although alfworld benchmark is multimodal, this study primarily focued on the text representation, leaving the examination of multimodal models and cross-attention encoding of other modalities (such as images and audio) for future work. comparisons with commercial models is discussed in section 6, but the proprietary nature and potential biases (e.g., unknown training data) limit a fair comparison with open-source models. standard-ized benchmarks in the field are needed for further evaluation. lastly, the reflection data utilized in our study is generated by gpt-4, which may not fully capture the distribution of real human data. this indicates the importance of integrating more authentic humangenerated data in future evaluations. future direction. the primary goal of this study is to create an efficient online rl pipeline for lms to perform multi-step problem solving. building on this foundation, future research directions may explore the scalability of reflect-rl to develop larger foundation models, enabling them to adapt to previously unseen environments with out-of-domain generalization capabilities. the two-player design in our framework may naturally be extended to other multi-agent settings where language models can show their strengths. another future direction is to train the reflection model in rlft stage as we freeze it because of the interference with the policy model (appendix f.2), which will improve the reasoning ability of language models for decision-making tasks.