W eb C o T : Enhancing Web Agent Reasoning by Reconstructing Chain-of-Thought in Reflection, Branching, and Rollback (2025)

Abstract: Web agents powered by Large Language Models (LLMs) show promise for next-generation AI, but their limited reasoning in uncertain, dynamic web environments hinders robust deployment. In this paper, we identify key reasoning skills essential for effective web agents, i.e., reflection & lookahead, branching, and rollback, and curate trajectory data that exemplifies these abilities by reconstructing the agent’s (inference-time) reasoning algorithms into chain-of-thought rationales. We conduct experiments in the agent self-improving benchmark, OpenWebVoyager, and demonstrate that distilling salient reasoning patterns into the backbone LLM via simple fine-tuning can substantially enhance its performance. Our approach yields significant improvements across multiple benchmarks, including WebVoyager, Mind2web-live, and SimpleQA (web search), highlighting the potential of targeted reasoning skill enhancement for web agents.

## introduction the rise of large language models (llms) has sparked significant interest in developing intelligent agents capable of interacting with the web through a browser, commonly referred to as web agents @xcite @xcite . however, despite recent advancements, even the best-performing web agents still lag far behind human performance-even when compared to users unfamiliar with a website's structure or functionality @xcite @xcite . this performance gap is primarily attributed to the limited reasoning abilities of current language models when applied to web agent workflows. despite recent advances in large reasoning models (lrm, e.g., deepseek-r1; deepseek-@xmath0, score: <@xmath1: 𝑎 ! (%) , simulated output: <res-ult2>, @xmath2, __ has the highest score and we should perform __. figure 1 : overview of our framework. we leverage a language model to translate inference-time processes, i.e., reflection and look-ahead, branching, and rollback, into natural language chain-of-thoughts, which are then used to train the agent language model. @xcite , these models primarily focus on arithmetic reasoning and are prone to overthinking and generating unnecessarily complex solutions for agent tasks @xcite @xcite . while directly applying reinforcement learning (rl) in agentic environments @xcite @xcite @xcite is a viable alternative, the resulting reasoning abilities are often unpredictable and lack structured priors. moreover, these approaches incur prohibitively high costs @xcite when conducting real-world rollouts. additionally, they often focus on static and deterministic environments such as webarena @xcite , whereas applying them to stochastic real-world open-domain web environments can be problematic due to the randomness inherent in rollouts. in contrast, distilling specific reasoning patterns into agents @xcite @xcite combines the adaptability of learned policies with the interpretability and taskaware heuristics of curated reasoning, mitigating both the overthinking problem and the exploration burden of pure rl. in this paper, we carefully examine and design the specific reasoning abilities required for effective web agents, and sample corresponding agent trajectories to conduct supervised fine-tuning (sft) on llms. in particular, we focus on three key components: (1) reflection & lookahead, the ability to reflect on previous failures and conjure precise long-horizon plans. (2) branching, the ability to sample multiple possible actions and have accurate awareness of the possible outcomes, selecting the most promising one; and (3) rollback, the ability to validate error, roll back to a previous state, and self-correct the agent's mistake. to study these abilities, we implement representative algorithms for each component: a novel reflectionand-lookahead module, webdreamer @xcite for branching, and agentrollback @xcite for rollback. simple illustrations of the three abilities are shown in figure 1 . for reflection and look-ahead, we analyze failed trajectories or redundant steps, distill the corrected trajectories after reflecting on errors, into key planning steps, and further refine them into structured chain-ofthought rationales using an llm. for branching, we sample multiple actions and select the best one using an llm, and then paraphrase this selection process into a paragraph of rationale. for rollback, we reference a successful trajectory and sample an intermediate state to intentionally generate incorrect branches, thereby constructing a goback action along with its corresponding rationale. to validate the effectiveness of the proposed method, experiments are conducted following the settings in the web agent self-improvement benchmark, openwebvoyager @xcite . specifically, we use llama-3.3 @xcite as the backbone llm for sampling web agent trajectories, and employ gpt-4o to perform rationale paraphrasing and action selection in reflection & look-ahead, branching, and rollback. the collected trajectories are then used to fine-tune llama-3.3 as our final agent model. our results demonstrate that distilling these reasoning patterns into llm chain-of-thought yields significant performance gains across multiple benchmarks, such as web-voyager @xcite , mind2web-live @xcite , and simpleqa @xcite . the approach substantially outperforms rejection sampling baselines and even exceeds distillation from more capable lrm like qwq-32b @xcite . overall, our research foot_0 underscores the importance of explicitly defining and instilling effective thinking and reasoning patterns for agent tasks. ## preliminary this section formalizes the web agent task and presents the foundational components of our agent optimization framework. ## webcot in this section, we introduce the details of the we-bcot reasoning patterns, reflection & lookahead, branching, and rollback, and how to perform cumulative training on top of the openwebvoyager self-improvement framework. an overview of the pipeline is shown in figure 2 . ## conclusion we presented webcot, a framework that enhances the reasoning ability of llms for webbased agent tasks through reflection & lookahead, branching, and rollback. by curating and finetuning reasoning trajectories, webcot can improve the efficiency and task completion accuracy by a large margin. our experiments across benchmarks such as webvoyager, mind2web-live, and simpleqa show significant performance gains over baselines, including gpt-4o and qwq-32b. these results highlight the value of targeted reasoning ability enhancements in bridging the gap between human and machine web agents.