Re- R e ST : Reflection-Reinforced Self-Training for Language Agents (2024)

Abstract: Finetuning language agents with reasoning-action trajectories is effective, but obtaining these trajectories from human annotations or stronger models is costly and sometimes impractical. In this paper, we investigate the use of self-training in language agents, which can generate supervision from the agent itself, offering a promising alternative without relying on human or stronger model demonstrations. Self-training, however, requires high-quality model-generated samples, which are hard to obtain for challenging language agent tasks. To address this, we present Reflection-Reinforced Self-Training (Re-ReST), which uses a reflector to refine low-quality generated samples during self-training. The reflector takes the agent’s output and feedback from an external environment (e.g., unit test results in code generation) to produce improved samples. This technique enhances the quality of inferior samples and efficiently enriches the self-training dataset with higher-quality samples. We conduct extensive experiments on open-source language agents across tasks, including multi-hop question answering, sequential decision-making, code generation, visual question answering, and text-to-image generation. The results demonstrate the effectiveness of self-training and Re-ReST in language agent tasks, with self-training improving baselines by 7.6% on HotpotQA and 28.4% on AlfWorld, and Re-ReST further boosting performance by 2.0% and 14.1%, respectively. Our studies also confirm the efficiency of using a reflector to generate high-quality samples for self-training. Moreover, we demonstrate a method to employ reflection during inference without ground-truth feedback, addressing the limitation of previous reflection work.

## introduction large language models (llms) @xcite @xcite figure 1 : previous agent training methods @xcite distill knowledge from stronger models (e.g., gpt-4) to weaker ones (e.g., llama-2). in contrast, we adopt self-training and improve it with reflection to improve agents more autonomously, which reduces reliance on external propriety models and maintains a fully open-source framework. @xcite have demonstrated potential in interacting with external environments and addressing practical interactive tasks, resulting in a new class -language agents @xcite . finetuning llms for agentic tasks has proven effective, yet existing works rely on data generated by stronger models (e.g., gpt-4) @xcite , which are not always available (e.g., to improve the strongest model). among the potential techniques to improve agents @xcite @xcite , self-training holds promise for enhancing agent performance for challenging agentic tasks. the self-training process typically involves refining the model by generating samples, assessing their quality through rewards, and updating the model by training on high-quality samples. compared with existing agent training methods @xcite , selftraining can autonomously improve agents and reduce the discrepancy between the agent's training data and its original predictions. additionally, as in figure 1 , self-training can potentially allow for the development of performant agents within a fully open-source framework, without relying on closedsource, proprietary models. given these benefits, we propose to investigate the use of self-training in language agents in this paper. however, one significant challenge for applying self-training in language agent tasks lies in the acquisition of high-quality samples to achieve good performance. specifically, self-training requires a substantial amount of high-quality samples, while relying solely on model-generated samples can be inefficient, particularly for language agent tasks that demand multi-step reasoning and long-horizon planning. as a result, it is challenging to obtain good samples solely through sampling. moreover, the common practice of discarding low-quality samples neglects their potential for improvement and effective utilization, thus limiting the overall efficacy of self-training methods. to address these issues, we propose reflection-reinforced self-training (re-rest), which enhances the self-training algorithm using a reflection model. re-rest incorporates a reflector during self-training, which improves sample quality by utilizing environmental feedback such as execution successes and unit test outcomes. specifically, the reflector transforms lower-quality samples into higher-quality ones, leveraging the capability of llms to self-improve when provided with accurate ground-truth feedback @xcite . consequently, it enriches the training dataset, enabling more effective bootstrapping. after training, only the agent model is used for inference, ensuring no additional computational burden during testing. unlike existing self-reflection methods @xcite @xcite , re-rest only requires access to feedback during training, not during inference, making our setting more realistic and practical. we conduct extensive experiments with opensource llms across a wide range of tasks, including multi-hop question answering, sequential decision-making, code generation, visual question answering, and text-to-image generation. our results first demonstrate the potential of self-training in language agent tasks, showing improvements over few-shot baselines in long-horizon planning tasks, with gains of 7.6% on hotpotqa and 28.4% on alfworld. by incorporating re-rest, we further enhance performance significantly by 2.0% and 14.1% on hotpotqa and alfworld, respectively, achieving results better or comparable to models relying on commercial apis. ablation studies confirm the efficiency of the reflection model in generating high-quality self-training samples. furthermore, we explore using our reflection model during inference with self-consistency decoding, which improves the model performance while alleviating the need for ground-truth feedback required by previous work @xcite . additionally, we demonstrate the application of our method in preference optimization objectives. ## experiments we experiment with multi-hop reasoning, sequential decision-making, code generation, visual question answering, and text-to-image generation. we present the experimental settings and results for each task. in all our experiments, we advocate for the use of open-source models and aim to avoid black-box, closed-source commercial models whenever possible. ## related work in this section, we first overview the research progress in language agents, then briefly describe self-training and self-correction methods for improving language agents. we also summarize the major differences between our work and previous language agent methods in language agents. language agents refer to language models that interact with the world in general. it has been demonstrated that llms can perform actions by generating specific commands @xcite @xcite and calling external tool apis @xcite @xcite . by integrating the model reasoning and acting abilities, react @xcite asks an llm to first generate reasoning traces and then act accordingly, which is then improved by follow-up works through inference-time techniques such as reflection @xcite and planning @xcite . recently, finetuning agents @xcite have attracted attention from the research community. however, most of the existing works attempt to distill knowledge from a relatively strong llm (e.g., gpt-4) to a weaker llm (e.g., llama-2). by contrast, our work bootstraps a language agent's performance by utilizing its own reflective ability without using external models. self-training for language models. various self-training algorithms have been proposed to improve language models @xcite @xcite @xcite , with the general idea being to improve models with self-generated samples in an unsupervised or semi-supervised manner. @xcite is one early work in applying self-training to generative language models and points out the importance of introducing noises during pseudo-label generation to increase the sample diversity. in the large language model era, @xcite propose reinforced self-training (rest), where they use a scoring function to select self-generated samples and augment the training data. similarly, @xcite proposes self-rewarding that scores samples with the llm itself and trains the model with direct preference optimization (dpo) @xcite on the scored samples. self-training has also been employed to improve the chain-of-thought reason-ing @xcite ability of llms @xcite . for example, @xcite propose to ask an llm to generate rationales given questions and improve the llm with its own generated reasoning. re-rest falls under the self-training paradigm, and different from previous work, our aim is to generate useful samples efficiently for self-training. self-reflection/self-correction for language models. several works have used llms to reflect on their generations with internal or external feedback and correct their errors @xcite @xcite @xcite . a majority of this line of research is focused on improving llms during inference. for example, self-refine @xcite proposes to have llms iteratively evaluate their generations, based on which they improve their generations. similarly, @xcite use llm agents to reflect on its generations and their environment feedback, then guide the next generation with the generated verbal feedback. as pointed out by @xcite , high-quality external feedback is essential for these self-correction models, without which existing techniques actually decrease model performance. however, such high-quality feedback is often unavailable during the test time, thus we propose to use re-rest only during training and perform corrections with oracle feedback from environments, ensuring its effectiveness in correcting the model generations. in addition, during the test time, the corrected generations are distilled into the language model, thus directly generating the answer without introducing overhead during inference. ## conclusion our study studies the applications of self-training in language agents and improves it with reflection-reinforced self-training (re-rest), an approach that efficiently obtains high-quality samples for self-training with a reflector. our experiments demonstrate that re-rest outperforms selftraining methods across various tasks, confirming the efficiency and effectiveness of incorporating a reflection mechanism. within the proposed framework, in the future, we can improve the reflection mechanism and develop better training paradigms for the agent and reflector.