Retrospex: Language Agent Meets Offline Reinforcement Learning Critic (2024)

Abstract: Large language models (LLMs) possess extensive knowledge and commonsense reasoning capabilities, making them valuable for creating powerful agents. However, existing LLM agent frameworks have not fully utilized past experiences for improvement. This work introduces a new LLM-based agent framework called Retrospex, which addresses this challenge by analyzing past experiences in depth. Unlike previous approaches, Retrospex does not directly integrate experiences into the LLM’s context. Instead, it combines the LLM’s action likelihood with action values estimated by a Reinforcement Learning (RL) Critic, which is trained on past experiences through an offline “retrospection” process. Additionally, Retrospex employs a dynamic action rescoring mechanism that increases the importance of experience-based values for tasks that require more interaction with the environment. We evaluate Retrospex in ScienceWorld, ALFWorld and Webshop environments, demonstrating its advantages over strong baselines.

## introduction the emergence of llms has paved the way for the development of llm-based agents. these agents leverage the vast knowledge and commonsense reasoning capabilities within llms to tackle a wide range of tasks @xcite @xcite @xcite . despite their potential, a significant challenge arises from the dependence on general-purpose llms. specifically, these agents might not be sufficiently adapted to the specific environments, potentially hindering their task completion effectiveness. training llm-based agents for new environments poses significant challenges. a common approach is to fine-tune the llm using "correct" 1 https://github.com/yufei-xiang/retrospex demonstrations-sample trajectories that successfully complete the task @xcite @xcite . however, this approach focuses on correct behaviors, limiting the agent's ability to learn and recover from mistakes. recently, efforts have been made to leverage imperfect experiences for training llm agents. these methods fall into two categories: those that rely on working memory, like reflexion @xcite , and those that utilize cross-task experiences from long-term memory, such as rememberer @xcite . despite the progress, experiences are still not sufficiently used, as they are only integrated into the llm's context. due to the limited context length of llms, this constrains the inclusion of more comprehensive experiences. in this work, we propose a novel llm-based agent framework called retrospex, which collects cross-task experiences for training a reinforcement learning (rl) critic in a "retrospection" stage. the rl critic is then used to support the llm in decision making. unlike previous studies (see figure 1 ), retrospex does not directly integrate experiences into the context. instead, it exploits an action rescoring strategy to combine the likelihood of the actions provided by the llm and the action values estimated by the rl critic. in addition, retrospex dynamically increases the weight of action values from the rl critic for tasks that require more interaction steps with the environment, allowing experiences to gradually play a more important role in difficult tasks. retrospex has several advantages over previous approaches. first, compared to rl-based agents, retrospex can leverage the strength of llms for more effective decision making. second, compared to previous llm-based agents, retrospex can better utilize experiences without increasing the context length. third, retrospex is more flexible in controlling how much experience is needed at each step thanks to the dynamic scoring method. finally, llm context llm context action observation observation reflection llm context long-term memory retriever experiences observation llm context long-term memory rl critic top-k dynamic action rescoring act likelihood act value observation react reflexion rememberer retrospex action action action offline rl training added to long-term memory llm-based agents without long-term memory retrospex is general and can be adapted to various llms or rl methods. in this paper, we implement rl critic with a lightweight neural network, thus providing little inference overhead compared to using only llms for action selection. we evaluate retrospex in three text-based simulation environments: scienceworld @xcite , alfworld @xcite , and webshop @xcite . the experimental results demonstrate that integrating the rl critic and dynamic action scoring in retrospex enhances the performance of llm-based agents, leading to success rate improvements of 9% in scienceworld, 3.5% in alfworld, and up to 5% in webshop. our contributions are summarized as follows: ## methodology figure 2 demonstrates the training process of retrospex which involves a warm-up phase and a retrospection phase. during the warm-up phase, we finetune the llm based on expert (e.g. human) demonstrations and collect the working experiences of the llm agent. in the retrospection stage, we train an rl critic from the llm agent's experiences using implicit q-learning (iql), an offline rl method. by doing so, the rl critic is expected to learn from the llm agent's mistakes and assist in making better decisions in the future. ## experiments the experiments are conducted in three environments: scienceworld @xcite , alf-world @xcite and webshop @xcite . we use average score (as) and success rate (sr) to measure the performance foot_0 . both metrics (as and sr) are scaled to the range of @xcite in all three environments. the training and testing sets for the three environments are summarized in baselines we compare retrospex to baselines of different types: (1) llm-based agents react @xcite and reflexion @xcite ; (2) online rl agent drrn @xcite ; and (3) saycan @xcite which combines llm with an affordance function for action grounding. the details for saycan, react and reflexion are provided in swiftsage paper @xcite . it is noteworthy that we do not compare to swiftsage here as it exploits two large language models (gpt and il-t5) for inference, resulting in a somewhat unfair comparison. the results of gptp3.5-based react and drrn are produced by ourselves. ## conclusion this work introduces a novel llm-based agent framework, named retrospex, that addresses the limitations of prior approaches in leveraging experiences for decision-making. retrospex overcomes the context length restriction by separating experience analysis (through a reinforcement learning critic) from llm-based action selection. this enables the agent to effectively utilize past experiences while retaining the strengths of llms. additionally, the dynamic action rescoring method allows for flexible control over the influence of experiences based on task complexity. evaluations demonstrate that retrospex achieves significant performance improvements compared to strong baselines, highlighting its potential for real-world applications of llm agents.