Improving Retrospective Language Agents via Joint Policy Gradient Optimization (2025)

Abstract: In recent research advancements within the community, large language models (LLMs) have sparked great interest in creating autonomous agents. However, current prompt-based agents often heavily rely on large-scale LLMs. Meanwhile, although fine-tuning methods significantly enhance the capabilities of smaller LLMs, the fine-tuned agents often lack the potential for self-reflection and self-improvement. To address these challenges, we introduce a novel agent framework named RetroAct, which is a framework that jointly optimizes both task-planning and self-reflective evolution capabilities in language agents. Specifically, we develop a two-stage joint optimization process that integrates imitation learning and reinforcement learning, and design an off-policy joint policy gradient optimization algorithm with imitation learning regularization to enhance the data efficiency and training stability in agent tasks. RetroAct significantly improves the performance of open-source models, reduces dependency on closed-source LLMs, and enables fine-tuned agents to learn and evolve continuously. We conduct extensive experiments across various testing environments, demonstrating RetroAct has substantial improvements in task performance and decision-making processes.

## introduction achieving independent, autonomous agents capable of thinking, reasoning, and dynamically interacting with their environment has long been a fundamental goal for researchers in the field of artificial intelligence. in recent years, with the emergence of the powerful capabilities of large language models (llms) @xcite , researchers have started to utilize these models in building advanced intelligent agents @xcite @xcite . these the planner analyzes task requirements, calls external tools, and gathers feedback. if planning fails, the reflector intervenes to adjust the strategy until the issue is resolved. through joint strategy optimization, retroact continually enhances both the planner and reflector to tackle complex tasks more effectively. in this work, we propose a novel agent framework called retroact, which jointly optimizes the task-planning and self-reflection capabilities of open-source llms. this approach eliminates the dependency on closed-source models during inference while retaining the ability for continuous reflection and evolution. specifically, we developed a two-phase joint optimization process that integrates imitation learning and reinforcement learning. first, we use imitation learning to distill the planning and reflection capabilities of large-scale llms into smaller llms. then, we propose an off-policy joint policy gradient optimization algorithm with imitation learning regularization to enhance data efficiency and training stability in the reinforcement learning process. during the joint optimization process, the planner and reflector can mutually facilitate each other and collectively improve the overall performance of the agent, showcasing the unique advantages of joint optimization. we conduct experiments on three representative agent tasks: complex reasoning (hot-potqa @xcite ), embodied decision (alfworld @xcite ), and interactive programming (intercode @xcite ), based on the llama-7b and llama-13b @xcite . through joint optimization, our model achieves performance improvements ranging from 22.3% to 348.3% on these tasks, attaining performance comparable to or exceeding that of chat-gpt @xcite . moreover, we further validate the mutual promotion between the planner and reflector. we also demonstrate that a single model can concurrently learn both planning and reflection capabilities, albeit with a slight decrease in performance. finally, we conduct systematic ablation studies to demonstrate the importance of each module in our approach. to summarize, our contributions are shown in the following: ## approach existing methods for agent-tuning primarily rely on knowledge acquired during the fine-tuning process, lacking the ability for real-time self-reflection and self-improvement. to address these limitations, we propose retroact, a novel approach that jointly optimizes llms' task-planning and self-reflection capabilities. our approach significantly enhances the performance of open-source llms through a combination of imitation learning (il) and reinforcement learning (rl). this eliminates the dependency on large-scale llms in agent tasks and retains the potential to evolve in new environments without parameter updates. figure 2 illustrates the overall framework of our retroact. in this section, we frame the process of llm-based agent tasksolving as a markov decision process (mdp) and construct our method within this mdp framework. ## experiments in this section, we systematically explore the performance of our proposed method. our experimental design revolves around four key questions: q1: how does retroact perform compared with the existing prompt-based and fine-tuning method? what are the benefits of joint optimization in improving the learning process of both the planner and reflector? q2: how about integrating the planner and reflector within the same open-source model? can the model effectively learn both planning and selfreflection capabilities simultaneously? q3: how do the optimized planner and reflector respectively influence the behavior of retroact? q4&q5: how does the reinforcement learning process and imitation learning regularization impact retroact? ## conclusion in this work, we introduce retroact, an agent framework that utilizes imitation learning and offpolicy reinforcement learning to jointly optimize the task-planning and self-reflective capabilities of open-source llms. retroact significantly enhances the performance of these models, reducing the reliance on closed-source llms in agent tasks. we conduct extensive experiments across various agent environments, demonstrating the substantial improvements over existing baselines. future work may involve designing more complex reinforcement learning systems, such as training reward models for the planner and reflector to provide more fine-grained reward signals.