ST e C a: Step-level Trajectory Calibration for LLM Agent Learning (2025)

Abstract: Large language model (LLM)-based agents have shown promise in tackling complex tasks by interacting dynamically with the environment. Existing work primarily focuses on behavior cloning from expert demonstrations or preference learning through exploratory trajectory sampling. However, these methods often struggle to address long-horizon tasks, where suboptimal actions accumulate step by step, causing agents to deviate from correct task trajectories.To address this, we highlight the importance of timely calibration and the need to automatically construct calibration trajectories for training agents. We propose Step-Level Trajectory Calibration (STeCa), a novel framework for LLM agent learning. Specifically, STeCa identifies suboptimal actions through a step-level reward comparison during exploration. It constructs calibrated trajectories using LLM-driven reflection, enabling agents to learn from improved decision-making processes. We finally leverage these calibrated trajectories with successful trajectories for reinforced training.Extensive experiments demonstrate that STeCa significantly outperforms existing methods. Further analysis highlights that timely calibration enables agents to complete tasks with greater robustness. Our code and data are available at https://github.com/WangHanLinHenry/STeCa.

## introduction large language models (llms) have recently demonstrated impressive capabilities in reasoning and planning, leading to the development of various llm-based agents that tackle real-world tasks such as household assistance @xcite , web browsing @xcite , and complex scientific reasoning @xcite . these agents typically engage in long-horizon interactions with en- † corresponding author. vironments, making sequential decisions to accomplish specific goals. despite their promise, llm agents still face significant challenges in generating high-quality task plans for complex scenarios @xcite . this highlights the need for more effective approaches to enhance their decision-making capabilities over time. previous work has investigated agent learning by leveraging augmented exploratory data @xcite @xcite . these methods primarily rely on behavior cloning from expert demonstrations, training agents exclusively on successful trajectories. however, this approach prevents agents from proactively self-correcting mistakes, leading to the accumulation of errors and ultimately suboptimal task performance @xcite . to address this limitation, another line of work focuses on preference learning @xcite and reinforcement learning @xcite , integrating failure trajectories additionally to refine decision-making. these approaches train llm agents using explicit error signals or reward functions. however, many long-horizon agentic tasks involve multi-turn interactions, where errors often only become evident at the terminal state @xcite . as a result, these methods fail to address early-stage deviations, which may not be immediately apparent but accumulate incrementally over time, ultimately leading to significant errors. to address the above limitations, we highlight the importance of timely calibration, which allows agents to adjust suboptimal actions as they occur, rather than deferring correction until after the entire exploration. as illustrated in figure 1 , when an early suboptimal action occurs, the subsequent actions are prone to deviate from the optimal trajectory, significantly increasing the risk of task failure. if an agent can engage in self-reflection and calibrate its behavior in real time, it stands a much better chance of successfully completing the task. however, implementing step-level calibrations presents significant challenges. 1) unlike in mathematical reasoning tasks @xcite , where well-defined rules simplify error detection, identifying deviations at each step in long-horizon agentic tasks is considerably more complex. this complexity stems from the dynamic and diverse nature of task execution in interactive environments. 2) as far as we know, the lack of step-level calibration trajectory data poses a major obstacle to training agents to effectively recognize and mitigate deviations. in this work, we propose step-level trajectory calibration (steca), a novel agent learning framework that enables llm agents to perform real-time calibration. steca operates by interacting with the environment to perform explorations and utilizes monte carlo (mc) sampling @xcite to estimate step reward for each action. by comparing the rewards of adjacent actions, steca effectively identifies deviated actions that lead to suboptimal performance. then, we utilize off-the-shelf llms for reflection, which revises a deviated action into its ground-truth counterpart while generating a reflective thought. the resulting action and its thought, along with subsequent expert trajectory, form a calibrated trajectory. all calibrated trajectories, combined with successful trajectories during exploration, are then used to reinforce the agent's training, optimizing its learning process. we evaluate steca on two widelyused agent benchmarks @xcite . extensive experimental results demonstrate that steca significantly outperforms existing methods, achieving higher success rates across a variety of tasks. in summary, our contributions are as follows: • we highlight the importance of timely calibration in interactive agentic tasks, a crucial aspect largely overlooked by previous methods. unlike existing approaches that rely on terminal-state error signals or reward functions, we emphasize the need for real-time adjustments to prevent the accumulation of deviations, which can lead to significant errors in long-horizon tasks. ## preliminaries task formulation. this work investigates how llm-based agents tackle long-horizon tasks within specific environments through interactions. following previous studies @xcite , we formalize these agentic tasks as a partially observable markov decision process (pomdp), which contains the key elements (u, s, a, o, t , r). here, u denotes the instruction space, s the state space, a the action space, o the observation space, t the transition function (t : s × @xmath0, and r the reward function (r : s × @xmath1, 1]). since the task planning capability of llm agents is our main focus, u, a, o are subsets of natural language space. given a task instruction @xmath2, the llm agent π θ at time step t takes an action a t ∼ π θ (•|u, e t-1 ) and receives the environmental feedback as the observation o @xmath3. e t-1 denotes the historical interaction trajectory (a 1 , o 1 , ..., a t-1 , o t-1 ). each action a t incurs the environment state to s @xmath4. the interaction loop terminates when either the agent completes the task or the maximum step is reached. the final trajectory is e @xmath5, a 1 , o 1 , ..., a m , o m ), where m denotes the trajectory length. the outcome reward r o (u, e m ) ∈ [0, 1] indicates the success or failure of the task. ## method in this section, we present step-level trajectory calibration (steca), a novel learning framework for llm agents. first, we warm up agent training with supervised fine-tuning ( §3.1), equipping llm agents with necessary task planning capabilities. then, we focus on calibration trajectory construction ( §3.2), which detects deviated actions for an explored trajectory through step-level reward comparison and calibrates them by reflection. finally, we utilize these calibrated trajectories as a crucial part of data for reinforced training ( §3.3). figure 2 illustrates the overview of steca. ## discussions and analyses in this section, we provide detailed analyses of the steca framework from the following aspects. ## conclusion in this paper, we introduce steca, a novel agent learning framework designed to enhance the performance of llm agents in long-horizon tasks. steca identifies deviated actions through step-level reward comparisons and constructs calibration trajectories via reflection. these trajectories serve as critical data for reinforced training. extensive experiments demonstrate that steca significantly outperforms baseline methods, with additional analyses underscoring its robust calibration capabilities.