Unlocking Recursive Thinking of LLM s: Alignment via Refinement (2025)

Abstract: The OpenAI o1-series models have demonstrated that leveraging long-form Chain of Thought (CoT) can substantially enhance performance. However, the recursive thinking capabilities of Large Language Models (LLMs) remain limited, particularly in the absence of expert-curated data for distillation. In this paper, we propose AvR : Alignment via Refinement , a novel method aimed at unlocking the potential of LLMs for recursive reasoning through long-form CoT. AvR introduces a refinement process that integrates criticism and improvement actions, guided by differentiable learning techniques to optimize refinement-aware rewards . As a result, the synthesized multi-round data can be organized as a long refinement thought, further enabling test-time scaling. Experimental results show that AvR significantly outperforms conventional preference optimization methods. Notably, with only 3k synthetic samples, our method boosts the performance of the LLaMA-3-8B-Instruct model by over 20% in win rate on AlpacaEval 2.0. Our code is available at Github .

## introduction long-form cot @xcite plays a crucial role in test-time scaling @xcite , as it enables recursive reasoning akin to human cognitive processes when addressing query intent @xcite @xcite . however, most existing llms lack sequential revision capability, making it difficult to iteratively refine response quality through extended reasoning @xcite . traditional alignment methods optimize llm outputs solely based on final preference rewards, overlooking critical processes such as reflection and refinement of previously generated content. although recent approaches, such as process supervision and reinforcement learning (rl) without supervised fine-tuning @xcite , have shown notable gains in reasoning performance, the challenge of achieving effective testtime scaling remains an open research question. we begin by comparing how reward assignment is handled across various reinforcement learning from human feedback (rlhf) algorithms, as illustrated in figure 1 . traditional preference optimization methods fine-tune llm behavior to directly maximize a reward function based on the bradley-terry preference model @xcite . however, despite its efficiency, this approach fails to capture distinctions among generated responses, often leading to similar errors being repeated during parallel sampling. in contrast, o1-like models incorporate long-form cot reasoning to align models with outcome-level or process-level rewards @xcite . as a bonus, by leveraging rl algorithms and testtime computation, llms naturally learn to engage in recursive thinking behavior, incorporating selfverification, reflection. however, activating such capabilities incurs significant computational overhead @xcite , including the requirement for powerful backbone models, extensive sampling, and training costs. a central challenge is whether their complementary strengths can be integrated to enhance response quality with lower computational cost. in this work, we introduce a refinement-aware reward to unlock recursive thiking capability, which is both effective and efficient for alignment. specifically, our method aims to maximize the preference reward of refinement, incorporating criticism and improvement actions @xcite . this approach draws inspiration from differential learning @xcite , which ensures that the decisionmaking process can be effectively improved by optimizing the reward between different refinements. in implementation, we design a two-stage framework to synthesize long-form recursive thinking data and optimize llms to align with refinementdriven behavior. in stage 1, we leverage llms to perform criticism and improvement actions to refine initial responses via parallel sampling, then rank the refined outputs to generate paired preference data. then, we apply dpo @xcite to train llms to maximize the refinementaware reward, enabling the model to learn a policy that favors progressively improved outputs. in stage 2, we use the stage 1 model to synthesize high-quality cot trajectories via sequential revision, which serve as guidance to promote longterm refinement behavior, effectively encouraging models to optimize for long-horizon rewards. we demonstrate that our proposed avr model, trained on only 10k examples, significantly outperforms the baseline (llama-3-8b-instruct) and even surpasses rl-based methods trained on 60k samples, as evaluated on alpacaeval 2 and arena-hard v0.1. our contributions are as follows: ## related work test-time compute, which leverages inference time resources to refine model outputs, has shown promise in enhancing reasoning quality. to better understand how models can acquire and benefit from recursive thinking capabilities, we provide a brief review of recent approaches. self-correction while this process allows models to refine their outputs through feedback and can be further enhanced by cot synthesis, recent work has highlighted a key limitation: llms often struggle to correct their own errors in the absence of an external reward function @xcite @xcite . however, obtaining high-quality cot trajectories and reliable reward functions remains challenging in open-domain tasks. to mitigate the reliance on external reward functions in multi-turn selfimprovement, qu et al. (2024) proposed a majority voting mechanism, enabling the model to select improved responses through self-comparison. other approaches such as self-rewarding(yuan et al., 2024) and meta-rewarding(wu et al., 2024) utilize llms-as-a-judge to generate internal reward signals, facilitating iterative self-improvement. however, these methods primarily focus on enhancing llms' ability to evaluate or generate responses in multiple rounds, rather than enabling models to naturally engage in response refinement. ## problem formulation unlike the traditional definition of token-level markov decision processes (mdps) in language generation tasks, we define both the query and each response from llms as independent actions. we introduce a new action, refinement, which can be viewed as a combination of self-judgment and self-correction performed by the llms. formally, we define the multi-step mdp as a tuple {s, a, t , r, γ}, where s represents the state space, with the initial state s 0 sampled from the initial prompt distribution ρ 0 . the action space a consists of the possible sequences sampled by llms policy π, including both direct responses and refinement. the transition function t is typically deterministic for llms, modeled as p(s t+1 ) = [s t : a t ], where the next state is obtained by concatenating the previous state and the selected action. the reward function r provides immediate reward, while the discount factor γ balances short-term returns and long-term returns. most importantly, traditional token-level mdps are optimized to maximize rewards by directly generating the final response, with explicit reward signals available only at the final state. in contrast, our proposed mdp aims to maximize the cumulative reward over multiple steps: recursive thinking aims to encourage models to receive positive reward signals (i.e., @xmath0. for an initial response s 0 and a sequence of refinements {s 0 , s 1 , . . . , s t }, the refinement process can be naturally decomposed into two levels of optimization: 1) single-step optimization, where the goal is to ensure that a refinement action yields a higher reward than the original response, i.e., r(s 1 , s 0 ) > r(s 0 ). 2) multi-step optimization, which imposes a stronger constraint: each subsequent refinement step must produce a response with a progressively higher reward than the previous step, i.e., r(s t+1 , s t ) > r(s t , s t-1 ). an algorithm that satisfies these conditions would enable llms to acquire recursive thinking capabilities, empowering them to iteratively refine their outputs in a self-improving manner. in particular, we define a refinement-aware reward that enforces s t satisfy: each refinement must be better than the initial response s 0 ; each refinement must also be an improvement over the previous step. during optimization, we perform rejection sampling to discard any refinement trajectories that violate this condition: this ensures that the model learns from only effective and cumulatively beneficial refinement steps. ## method in this section, we propose a two-stage framework to unlock the recursive thinking capabilities of llms. as shown on the left side of figure 2 , the first stage explicitly guides the model to refine a previously generated response using carefully designed prompts. in contrast, the right side illustrates the second stage, where the model transitions to autonomous recursive thinking, performing multi-step refinement without relying on external instructions. the model constructs a recursive thinking trajectory by generating internal control statements such as "now, let's try to criticize this answer," "okay, let's improve the above answer based on the criticism," and "now it's almost done." these intermediate reasoning steps are enclosed within special tokens to distinguish them from the final response. ultimately, the model decides on its own when the refinement process is complete and proceeds to generate the final answer. ## conclusion in this paper, we propose a novel framework to unlock the recursive thinking capabilities of llms. by leveraging inference-time scaling and maximizing a refinement-aware reward, our method synthesizes high-quality data that supports long cot reasoning and enables self-improvement within the model itself. crucially, our two-stage training and inference framework significantly reduces the gpu memory footprint and training cost compared to rl-based methods. experimental results demonstrate that our approach substantially enhances both the generative and corrective abilities of the base model. remarkably, with only a small amount of training data, our method empowers an 8b model to dynamically refine its outputs during inference. furthermore, we observe that our model can effectively refine responses generated by more powerful models, offering new perspectives for improving llm performance through recursive selfimprovement.