Improving Preference Alignment of LLM with Inference-Free Self-Refinement (2025)

Abstract: Large language models (LLMs) develop the in-context learning capability through pretraining and instruction tuning, enabling task adaptation without parameter updates. Self-refinement is a manifestation of this capability, which allows LLMs to iteratively refine the output using self-generated feedback. However, empirical observations reveal Inference-Free Self-Refinement (IFSR) in preference alignment: LLMs generate preference-improved output via fixed instructions, requiring no specific feedback, even no initial responses. There are two key components of the IFSR in preference alignment. The refining instruction is a fixed instruction that constrains the output distribution from a preference-semantic perspective. During training, it facilitates joint learning of preference-related semantic representations and data distribution alignment. The pseudo reference response is constructed from paired preference data and serves as a demonstration to guide the output distribution. It mitigates off-policy distributional bias while enhancing token-level preference learning in training. Experiments across multiple datasets demonstrate that incorporating IFSR into preference alignment yields performance improvement over 10%. Further ablation studies reveal additional characteristics and potential principles of IFSR.

## introduction recent advancements in large language models (llms), trained on billions of tokens via unsupervised learning, demonstrate emergent capabilities including in-context learning, instruction following, and logical reasoning, and achieve impressive performance across tasks ranging from machine translation @xcite to code generation @xcite . building on this, instruction tuning and preference alignment @xcite further refine these capabilities by enhancing their ability * corresponding author. what runs around the whole yard without moving? please generate a better response. the answer to this riddle is a fence! a fence runs around the whole yard, enclosing it and providing a boundary. a fence is considered to be a continuous structure, so it can be thought of as something that "runs around" the yard without actually moving. a fence. to interpret human intent and generate outputs that are accurate, contextually coherent, and aligned with ethical constraints @xcite . early alignment approaches like reinforcement learning from human feedback (rlhf) @xcite relied on reward modeling and reinforcement learning, while subsequent methods such as direct preference optimization (dpo) @xcite eliminated the need for explicit reward models through supervised optimization. this evolution has spurred numerous refined techniques @xcite @xcite addressing diverse alignment challenges. the emergent in-context learning capabilities of llms enable self-refinement, whereby models iteratively generate feedback on initial responses and produce refined outputs. this mechanism has been widely adopted for tasks such as data augmentation @xcite and code optimization @xcite . in preference alignment scenarios, we observe a related yet distinct phenomenon. as illustrated in figure 1 , appending a simple fixed instruction (e.g., "please generate a better response") prompts the model to automatically enrich minimal initial responses with based on these findings, we propose a preference alignment enhancement method for llms using ifsr. this approach relies on the consistent contextual influence on generation distributions across both fine-tuning and inference stages, as they share identical forward propagation mechanisms. the model produces identical probability distributions for given contexts across stages, though their utilization differs: during fine-tuning, these distributions compute loss against labels, while during inference, they guide output sampling. by strategically applying ifsr components to reshape initial generation distributions during training, we effectively improve preference alignment. the two components collaboratively optimize pref- the training dynamics further amplify these effects through gradient-based updates. the instruction component facilitates semantic consistency across samples by establishing preference-semantic associations through instruction bridging. meanwhile, the reference response strengthens tokenlevel preference learning while directing model updates toward genuine preference patterns rather than distributional artifacts. notably, this approach is architecture-agnostic and algorithm-independent, enabling orthogonal integration with existing preference optimization methods. our contributions can be summarized as follows: ## analysis of ifsr to advance our investigation of inference-free selfrefinement (ifsr) in the preference alignment scenario, we conducted some experiments leveraging the llama3-8b-base (ai@meta, 2024) model and paired preference data from the openassistant conversations dataset (oasst) @xcite . the experimental design focus on quantifying the impact of the refining instruction i ↑ and the pseudo reference response ỹ on the model loss and generation probability distributions. specifically, i ↑ is 𝑥 𝑦 ! 𝑥 𝑦 # 𝑥 𝑦 ! 𝑦 " 𝑖 ↑ dpo or variants ifsr instruction fine-tuning inference preference optimization 𝑥 𝑦 ! 𝑦 " 𝑖 ↑ 𝑥 𝑦 ! 𝑦 " 𝑖 ↑ 𝑥 𝑦 ! 𝑥 𝑦 " 𝑥 prompt 𝑦 " 𝑦 ! chosen/rejected response 𝑖 ↑ refining instruction 𝑥 𝑖 ↑ 𝑦 # ↑ 𝑦 # generated response 𝑦 # 𝑦 # ↑ "please generate a better response", while ỹ is derived from paired responses in the dataset. our analysis reveals three critical insights. first, the inclusion of i ↑ and ỹ in the context reduces loss values, as shown in collectively, ifsr operates through dual mechanisms. the refining instruction establishes semantic relevance constraints that prune the learning space, while the pseudo reference response acts as an exemplar-driven regularizer to reduce the gap between the original and target distributions. ## proposed method in this section, we elaborate on the implementation of preference alignment with ifsr, which extends standard alignment pipelines as shown in figure 2 . the conventional alignment workflow first fine-tunes a pretrained base model π 0 by supervised finetuning (sft) to obtain π sf t , then optimizes it with a paired preference dataset @xmath0, y to produce the final aligned model π pre , where x denotes inputs, y + and y -represent chosen and rejected responses. ifsr extends this workflow by integrating the refining instruction i ↑ and the reference response across stages. with ifsr-augmented data, π sf t undergoes additional fine-tuning ( §3.1) and π pre is further refined ( §3.2), while the inference context also includes the instruction and the reference ( §3.3). these procedures reduce distribution mismatches between model generations and target responses while strengthening the semantic connection between preference descriptions and their textual expressions. ## case study to investigate how ifsr modifies llm outputs, we analyzed two representative examples targeting harmlessness and helpfulness: ## conclusion this paper studies the phenomenon of inferencefree self-refinement in llm preference alignment, establishing the critical roles of the refining instruction and the pseudo reference response in the context. building on these insights, we propose a method ifsr that systematically integrates these components into the fine-tuning, optimization, and inference stages of llm alignment. this approach enhances the utilization of paired preference data while reinforcing the model's focus on concrete preference expressions through abstract preference descriptions and reference examples, thereby significantly improving preference learning efficacy. extensive experiments across multiple datasets validate the effectiveness of the method, with analytical experiments and case studies further advancing the understanding of its operational principles.