PGPO : Enhancing Agent Reasoning via Pseudocode-style Planning Guided Preference Optimization (2025)

Abstract: Large Language Model (LLM) agents have demonstrated impressive capabilities in handling complex interactive problems. Existing LLM agents mainly generate natural language plans to guide reasoning, which is verbose and inefficient. NL plans are also tailored to specific tasks and restrict agents’ ability to generalize across similar tasks. To this end, we explore pseudocode-style plans (P-code Plan) to capture the structural logic of reasoning. We find that P-code Plan empowers LLM agents with stronger generalization ability and more efficiency. Inspired by this finding, we propose a pseudocode-style \underline{P} lanning \underline{G} uided \underline{P} reference \underline{O} ptimization method called PGPO for effective agent learning. With two planning-oriented rewards, PGPO further enhances LLM agents’ ability to generate high-quality P-code Plans and subsequent reasoning. Experiments show that PGPO achieves superior performance on representative agent benchmarks and outperforms the current leading baselines. Analyses reveal the advantage of PGPO in reducing action errors and omissions during reasoning.

## introduction recent advances in large language models have promoted the development of llm agents @xcite . planning serves as a critical component of agent reasoning @xcite , allowing agents to break down complex problems into manageable subtasks. by prompting strategies @xcite @xcite , task-specific fine-tuning @xcite or external classical planners @xcite , llm agents are equipped with basic planning abilities. due to to complete the task, first check nearby tables for the tissue box, then locate desklamp to view tissue box under it. thought: first, i need to locate the book. i will start by checking the nearby location … action: go to drawer 1 thought: first, i should go to the nearest table to find one tissue box ... action: go side prior work @xcite demonstrates the advantage of executable code as agent's action over text or json format. inspired by this, we explore using pseudocode to represent plan, given that plans can be considered as high-level abstraction of actions while pseudocode outlines code. our work starts by distilling pseudocodestyle plans (denoted as p-code plan) from existing react-style @xcite datasets, adhering to predefined format requirements. we observe that through fine-tuning with p-code plan incorporated, llm agents exhibit improved out-of-distribution generalization. as shown in figure 1 , abstract planning steps in p-code plan can capture generalizable task knowledge while nl plans focus on specific knowledge and may suffer from overfitting. moreover, agent's planning ability learned from concise p-code plan facilitates a more efficient reasoning process with fewer interactions. building upon these insights, we further propose a pseudocode-style planning guided preference optimization method named pgpo for agent capability enhancement. specifically, we first utilize supervised fine-tuning to build a base agent. then, pgpo contains two iterative phases: (1) the base agent performs exploration on expert trajectories to construct contrastive trajectory datasets based on two designed planning-oriented rewards; (2) direct preference optimization is employed to refine the base agent's pseudocode-style planning ability for task guidance. through experiments with four llms, pgpo outperforms various strong baselines by relative 11.6% performance gain averaged across three representative agent benchmarks. in summary, our contributions are as follows: ## pseudocode-style plan is beneficial in this section, we first define the structural representation of p-code plan and then introduce a plan generation pipeline, followed by preliminary experiments to show the advantage of p-code plan. ## p-code plan-guided agent learning despite widespread use in open llm agents @xcite @xcite , supervised fine-tuning approach has its drawback, that is, the limited generalization ability due to overreliance on expert trajectories @xcite . recent works focus on direct preference optimization (dpo) @xcite and its variants to develop llm agents with contrastive trajectory dataset @xcite @xcite . inspired by our findings in section 2.3, we propose a method for improving agents' pseudocodestyle planning ability, called pgpo, standing for planning guided preference optimization. the overview of our method is depicted in figure 4 .