Teaching LLM s to Plan, Not Just Solve: Plan Learning Boosts LLM s Generalization in Reasoning Tasks (2025)

Abstract: Reinforcement learning (RL) on self-generated data has emerged as a promising paradigm for improving reasoning in large language models (LLMs). However, RL relies on accurate reward signals, which are scarce in many domains, making it critical to train models that can generalize to unseen problems. Existing methods often focus on task-specific or domain-specific reasoning, lacking consideration for generalization and may degrade performance on other tasks. To address this, we distinguish between abstract plans, representing high-level problem-solving strategies, and concrete solutions, proposing that learning plans develops transferable general reasoning capabilities and promotes better generalization. Building on this insight, we propose PlanLearn, a framework that combines plan-based search with Step-level Advantage Preference Optimization (Step-APO) to optimize plan learning. Experimental results show that PlanLearn, trained exclusively on GSM8K and MATH, not only significantly improves in-domain performance but also enhances out-of-domain benchmarks, such as HumanEval (+12.2%), GPQA (+8.6%), ARC-C (+4.0%), MMLU-STEM (+2.2%), and BBH (+1.8%). The code is available at https://github.com/tianlwang/PlanLearn.

## introduction large language models (llms) have achieved significant success through scaling in pretraining @xcite . to further unblock the potential of llms, recent efforts extend this scaling paradigm to posttraining, where reinforcement learning (rl) on self-generated data has emerged as a new learning paradigm (e.g., star @xcite , rest @xcite ). notably, systems like openai's o1 @xcite demonstrate that large-scale rl can teach models to think more systematically, significantly enhancing their reasoning capabilities. additionally, recent research works @xcite @xcite leverage monte carlo tree search (mcts) @xcite to collect high-quality reasoning paths, which rl then uses for iterative self-improvement. however, rl requires accurate reward signals to guide the learning process; for example, ground truth answers are available to evaluate correctness in mathematical tasks, but obtaining such highquality supervision remains challenging in many domains. to build a general reasoner, it is crucial to enhance the model's generalization ability. most existing methods @xcite focus on enhancing task-specific or domainspecific reasoning capabilities, such as in mathematics or coding, often relying on task-specific fine-tuning, which hampers generalization and may degrade performance on other tasks @xcite . furthermore, unlike traditional rl, which operates in a limited action space, llms function ## methods in this section, we introduce our planlearn, which boosts model performance via an iterative process of plan-based search and step-level preference learning. we first introduce our plan-based mcts, which enables the llm to explore diverse plan strategies in the vast search space. next, we present our step-apo in detail to further explore the potential of step-level preference learning in multi-step reasoning tasks. finally, we describe how we iteratively optimize the policy and value models. ## experiments in this section, we conduct a comprehensive evaluation of planlearn across two in-domain datasets and five out-of-domain (ood) datasets to address the following research questions: • rq1 (subsection 3.2): when trained exclusively on mathematical reasoning data, does planlearn achieve superior performance compared to existing methods on both in-domain and out-of-domain reasoning tasks? • rq2 (subsection 3.3): does plan-based learning effectively enhance the generalization capabilities of models across diverse reasoning tasks? • rq3 (subsection 3.4): does step-apo provide a more effective optimization strategy for plan learning compared to existing approaches? furthermore, we explore the construction of steplevel preference data to enhance generalization capabilities (subsection 3.5) and demonstrate the generalizability of planlearn across different models (subsection 3.6). ## conclusion developing a general reasoner through rl remains an open and important research question. in this work, we propose searching and learning on highlevel abstract plans to enhance model generalization, rather than focusing on concrete solutions that often limit generalization. planlearn enhances transfer performance on diverse out-of-domain tasks, underscoring its value for future research in generalizable reasoning.