Instantly Learning Preference Alignment via In-context DPO (2025)

Abstract: Human Preference Alignment (HPA) can assist large language models (LLMs) to generate safe content. Due to the heavy cost of fine-tuning, tuning-free methods have emerged, typically modifying LLM decoding via post-processing. In this paper, we propose a novel and effective approach for HPA in a tuning-free way, named In-Context Direct Preference Optimization (ICDPO). We first rethink the derivation procedures of DPO, based on which we conversely build an instant scorer using the states of the LLM before and after ICL. It enables LLMs to both generate and select the well-aligned response, which is precisely estimated by the aforementioned instant scorer, thereby enhancing the final performance. ICDPO can be further enhanced with a two-stage retriever and an upgraded scorer. Extensive experiments show its effectiveness, particularly in outperforming multiple tuning-free baselines, even competitiveness with SFT and DPO. We also conduct detailed analyses to offer comprehensive insights into ICDPO.

## introduction human preference alignment (hpa) is crucial within the llm industry as it prevents llms from generating content contrary to human values. presently, mainstream approaches to hpa heavily depend on fine-tuning, exemplified by rlhf @xcite @xcite , raft @xcite , rrhf @xcite , or dpo @xcite . nevertheless, the huge computational and annotation costs of fine-tuning are hard to ignore. as a response, tuning-free methods with external supervision in decoding have gained popularity. for instance, external scorers capable of distinguishing human preference can be involved to apply bestof-n selection for multiple candidates or enhance block selection in inference @xcite . in this work, we propose a novel and effective approach, named in-context direct preference optimization (icdpo). specifically, we rethink the derivation of dpo @xcite , which transforms the rlhf objective and bridges the relation between the provided reward model (rm) and expert policy π * , where the rm is in sync with the distributional disparity between π * and its reference model π 0 . conversely, given π * aligned with human preference, it can both empower response generation, as well as work with its reference model (amateur) to enhance the scoring of hpa for candidate responses. meanwhile, icdpo avoids fine-tuning by utilizing in-context learning (icl) to shift the distribution of base models in the part response, thus instantly acquiring π * . the superiority of icdpo is attributed to two points: (1) existing approaches focus on the postprocessing of token distribution in decoding, while icdpo employs icl to directly bring the hpa capability to llms, requiring just several good demonstrations wherever they come from. figure 1 shows its similarity with fine-tuning by parameter updates. (2) the proposed mechanism of contrastive scoring by the expert-amateur collaboration in figure 1 (b) provides a more reliable estimation than the process of independent decision. furthermore, we are inspired by the prevalent contrastive decoding to facilitate the formulation of the expert-amateur collaboration. in detail, we incorporate both chosen and rejected demonstrations by annotators, driving the initial π 0 to a favorable π + and unfavorable π -by icl, respectively, which amplifies the disparity between them. it works as further debiasing the final distribution of candidates to consolidate icdpo. on the other hand, since icdpo harnesses llms through contextual demonstrations, the selection and ordering of demonstrated samples become crucial. inspired by the nature of fine-tuning, where aligned distributions between training and test sets maximize effectiveness, we develop a two-stage retriever to identify demonstrations that are most similar to the test samples in both form and semantics, thereby improving the performance of icdpo. extensive experiments are conducted to evaluate the proposed icdpo, encompassing evaluations using both a reward model (rm) and gpt-4, along with an ablation study validating each module and comprehensive analyses to explore from fine-grained perspectives. the observations of this work are as follows: 1. with a novel formulation, icdpo can instantly endow base models with effective hpa through a generation-scoring workflow. it consistently outperforms multiple tuning-free baselines and even competes with techniques like sft/dpo plus lora @xcite . the proposed two-stage retriever r and upgraded scorer ŝ further enhance the effectiveness of icdpo, as shown in both rm and gpt-4 evaluations. 2. demonstrations and the capacity of base models are closely tied to the final performance. both better base models and larger/higher-quality demonstrations have positive impacts, while r enlarges the effect of demonstration quality. 3. regarding scoring, both s and ŝ in icdpo can offer reliable estimations of hpa degree. ## methodology in this section, we rethink the transformation from rlhf to dpo @xcite , an elegant supervised fine-tuning algorithm derived from the original rlhf objective t . we focus on the relation between a given rm and the corresponding optimal policy π * , and adapt it to llm inference in the manner of in-context learning (icl), which we term as icdpo. ## conclusion we propose an effective method icdpo, which equips llms with hpa without fine-tuning. it first optimizes llms instantly via just several contextual demonstrations, while a novel formulation from the derivation of dpo is utilized to build an expert-amateur collaboration for a reliable estimation for the selection of candidate responses. comprehensive experiments demonstrate the effectiveness of icdpo across various forms, encompassing both content generation and scoring. we hope this work to be a catalyst for further exploration of tuning-free methods towards hpa.