Watch Every Step! LLM Agent Learning via Iterative Step-level Process Refinement (2024)

Abstract: Large language model agents have exhibited exceptional performance across a range of complex interactive tasks. Recent approaches have utilized tuning with expert trajectories to enhance agent performance, yet they primarily concentrate on outcome rewards, which may lead to errors or suboptimal actions due to the absence of process supervision signals. In this paper, we introduce the **I**terative step-level **P**rocess **R**efinement **(IPR)** framework, which provides detailed step-by-step guidance to enhance agent training. Specifically, we adopt the Monte Carlo method to estimate step-level rewards. During each iteration, the agent explores along the expert trajectory and generates new actions. These actions are then evaluated against the corresponding step of expert trajectory using step-level rewards. Such comparison helps identify discrepancies, yielding contrastive action pairs that serve as training data for the agent. Our experiments on three complex agent tasks demonstrate that our framework outperforms a variety of strong baselines. Moreover, our analytical finds highlight the effectiveness of IPR in augmenting action efficiency and its applicability to diverse models.

## introduction the advancements in large language models (llms), such as gpt-3.5 @xcite , gpt-4 @xcite , llama @xcite have paved ways for llm-based agents to excel in handling complex interactive tasks, including online shopping @xcite and embodied housework @xcite . to accomplish these tasks, llm agents explore the environment step by step, achieving sub-goals along action trajectories @xcite . the efficacy of this task-solving process is pivotal to agent's overall performance. initial efforts in the task-solving process for agents involve generating trajectories by directly leveraging the planning ability of llms, such as react @xcite and reflexion @xcite . to further enhance llm agent abilities, several studies focus on trajectory tuning @xcite @xcite . @xcite and @xcite construct agent trajectory data from teacher agents (e.g., gpt-4) and fine-tune open-source llms for specific agent abilities, such as reasoning. @xcite employ a multi-task supervised fine-tuning (sft) approach, which does not significantly improve generalized agent capabilities. observing that the sft-based works predominantly rely on expert success trajectories (figure 1 (a)), @xcite utilize failure trajectories and propose the exploration-based trajectory opti-mization (eto) method to learn the task-solving process (figure 1(b) ). although these methods present a promising avenue for enhancing agent capabilities, they treat an entire trajectory as a single entity during training and prioritize the final reward of a trajectory over the process, thus overlooking the potentially exploitable information throughout interaction process. regarding agent trajectories, it is well-known that alongside those with correct outcomes, there are trial-and-error paths with detours and erroneous ones that achieve accidental success. step-level process supervision can offer granular guidance at each step hence is beneficial for task resolution @xcite . nevertheless, the application of step-level optimization to llm agents encounters two practical challenges. firstly, the majority of existing llm agent environments @xcite @xcite provide only final outcome feedback. even in cases where environments offer sub-goal level feedback @xcite , the information is often too sparse. secondly, the question of how to effectively utilize step rewards to enhance agent training, particularly for tasks with long trajectories and complex action spaces, remains unexplored. in this paper, we address these challenges by introducing the iterative step-level process refinement (ipr) framework ( § 3) , which encompasses two principal mechanisms: step-level reward acquisition ( § 3.2) and iterative agent optimization ( § 3.3). more specifically, to construct the step reward within the agent environment, we employ monte carlo (mc) method to estimate rewards via sampling. the iterative agent optimization component aims to refine the agent's actions through a cyclical process. during each cycle, the agent navigates the expert trajectory and generate new actions. these actions are then compared with the corresponding step of the expert trajectory using step-level rewards to pinpoint errors, resulting in contrastive step pairs. subsequently, we train the agent using an arrangement of outcome-level direct preference optimization (dpo), step-level dpo, and sft losses, thereby enhancing the agent's action capabilities at each step (figure 1(c) ). we assess our ipr framework on three representative benchmarks: online shopping environment webshop @xcite , interactive sql environment intercodesql @xcite and textual embodied environment alfworld @xcite . the experimental results, detailed in § 4.2, reveal that our method surpasses the current leading method by margins of 5.8%, 7.2% and 3.2% on webshop, intercodesql, and alfworld, respectively. moreover, we present a comprehensive analysis to substantiate the efficacy of our method from various perspectives. in summary, our contributions are as follows: the primary scope of this study is the task-solving of llm agents interacting with the environment and receiving feedback. following @xcite , we formulate the task as a partially observable markov decision process (pomdp) defined by the elements (u, s, a, o, t , r). here, u denotes the instruction space, s the state space, a the action space, o the observation space, t the transition function (t : s × @xmath0, and r the reward function (r : s × @xmath1, 1]). in the context of our llm-based agent, u, a, o are subsets of natural language space. at time step t, the llm agent π θ receives the observation o t-1 ∈ o from the environment and takes an action a @xmath2, where e t-1 = (u, a 1 , o 1 , ..., a t-1 , o t-1 ) represents the historical trajectory. the action leads to a change in the state space s @xmath3, and receives execution feedback as observation o @xmath4. the interaction loop continues until the task is completed or the maximum steps are reached. the final trajectory is e @xmath5, a 1 , o 1 , ..., a n , o n ), where n denotes the trajectory length, and the outcome reward is r o (u, e n ) ∈ [0, 1]. for the convenience of subsequent content, we define e t:@xmath6, o t , ..., a n , o n ) to represent the trajectory after time step t. ## method the overall architecture of our method is depicted in figure 2 . initially, we empower the language model with fundamental agent capabilities via supervised learning ( § 3.1). subsequently, we develop the mc method to estimate the step-wise rewards within the agent's environment ( § 3.2). in the final stage, we enhance the agent's performance through iterative optimization ( § 3.3): by constructing contrastive action pairs and executing mixture trajectory optimization. ## conclusion in this paper, we present ipr, a novel framework designed to elevate the capabilties of llm agents in complex interaction tasks. our approach integrates process-level supervision, enabling agents to learn from contrast action pairs. to provide finegrained guidance in environments where only outcome rewards are available, we use the mc method to automatically calculate step rewards. by employing iterative agent optimization, ipr provides an effective way to optimize agent decision-making trajectories. experiments on three benchmarks demonstrate that our framework consistently outperforms existing baselines. subsequent analyses validate the efficacy of each part of the framework and action efficiency. we believe the ipr framework can serve as a potent tool for enhancing agent performance at the action level, thereby catalyzing future progress in intelligent agent development.