ORPO : Monolithic Preference Optimization without Reference Model (2024)

Abstract: While recent preference alignment algorithms for language models have demonstrated promising results, supervised fine-tuning (SFT) remains imperative for achieving successful convergence. In this paper, we revisit SFT in the context of preference alignment, emphasizing that a minor penalty for the disfavored style is sufficient for preference alignment. Building on this foundation, we introduce a straightforward reference model-free monolithic odds ratio preference optimization algorithm, ORPO, eliminating the need for an additional preference alignment phase. We demonstrate, both empirically and theoretically, that the odds ratio is a sensible choice for contrasting favored and disfavored styles during SFT across diverse sizes from 125M to 7B. Specifically, fine-tuning Phi-2 (2.7B), Llama-2 (7B), and Mistral (7B) with ORPO on the UltraFeedback alone surpasses the performance of state-of-the-art language models including Llama-2 Chat and Zephyr with more than 7B and 13B parameters: achieving up to 12.20% on AlpacaEval 2.0 (Figure 1), and 7.32 in MT-Bench (Table 2). We release code and model checkpoints for Mistral-ORPO- \alpha (7B) and Mistral-ORPO- \beta (7B).

## introduction pre-trained language models (plms) with vast training corpora such as web texts @xcite or textbooks @xcite have shown remarkable abilities in diverse natural language processing (nlp) tasks @xcite @xcite @xcite . however, the models must undergo further tuning to be usable in downstream applications, typically through processes such as instruction tuning and preference alignment. instruction-tuning @xcite @xcite ) trains 1 github: https://github.com/xfactlab/orpo 2 models: orpo collection 4.96 7.7 9.44 8.35 10.99 11.33 12.2 llama-2 mistral alpacaeval 2.0 llama (7b) llama (13b) llama-orpo (7b) zephyr-a zephyr-b mistral-orpo-a mistral-orpo-b 0.0 2.5 5.0 7.5 10.0 12.5 win rate (%) algorithm rlhf dpo orpo figure 1: alpacaeval 2.0 result of llama-2 (7b) and mistral (7b) fine-tuned with orpo (blue) in comparison to the state-of-the-art models. notably, mistral-orpo-α & β surpasses zephyr β and llama-2-chat (13b) with a single epoch training exclusively on the ultrafeedback. to further align these models with human values, additional training is required with pairwise preference data using techniques such as reinforcement learning with human feedback @xcite and direct preference optimization @xcite . existing preference alignment methods typically consist of a multi-stage process, as shown in figure 2 , typically requiring a second reference model and a separate warm-up phase with supervised fine-tuning (sft) @xcite @xcite , which adds additional resource overheads. alignment without reward model recently proposed techniques for preference alignment mitigate the need for reinforcement learning @xcite @xcite @xcite . rafailov et al. (2023) introduce direct preference optimization (dpo), which removes the reward modeling stage. azar et al. (2023) prevented potential overfitting problems in dpo through identity preference optimization (ipo). ethayarajh et al. (2024) and cai et al. (2023) proposed kahneman-tversky optimization (kto) and unified language model alignment (ulma) that does not require the pair-wise preference dataset, unlike rlhf and dpo. song et al. (2023) and xu et al. (2024) further suggest incorporation of the softmax value of the reference response set in the negative log-likelihood loss to merge the supervised finetuning and preference alignment. alignment with supervised fine-tuning there have been approaches to build human-aligned language models by conducting supervised finetuning (sft) only with filtered datasets @xcite @xcite @xcite @xcite . @xcite demonstrated that sft with a small amount of data with fine-grained curation could be sufficient for building helpful language model assistants. furthermore, @xcite and haggerty and chandra (2024) proposed an iterative process of fine-tuning the supervised fine-tuned language models with their own generations after fine-grained selection of aligned generations and @xcite suggested that a curated subset of preference dataset is sufficient for alignment. ## odds ratio preference optimization we introduce a novel preference alignment algorithm, odds ratio preference optimization (orpo), which incorporates an odds ratio-based penalty to the conventional supervised fine-tuning (sft) loss (i.e., negative log-likelihood (nll)) for differentiating the generation styles between favored and disfavored responses. we discuss the effects of sft on preference alignment in section 3.2 and explain the mechanism of orpo in section 3.3. ## experimental results first, we assess the general instruction-following abilities of the models by comparing the preference alignment algorithms in single-turn (section 5.1) and multi-turn (section 5.2) instruction following benchmarks. then, we compare orpo against other alignment methods in the controlled setting, using opt with various model sizes (section 5.3). ## conclusion in this paper, we introduced a reference-free monolithic preference alignment method, odds ratio preference optimization (orpo), by revisiting and understanding the value of the supervised fine-tuning (sft) phase in the context of preference alignment. orpo was consistently preferred by the fine-tuned reward model against sft and rlhf across the scale, and the win rate against dpo increased as the model size increased. furthermore, we validate the scalability of orpo with 2.7b and 7b pre-trained language models by exceeding the larger state-of-the-art instruction-following language models in alpacaeval. specifically, mistral-orpo-α and mistral-orpo-β achieved 11.33% and 12.20% in alpacaeval 2.0 , 7.23 and 7.32 in mt-bench, thereby underscoring the efficiency and effectiveness of orpo.