L i PO : Listwise Preference Optimization through Learning-to-Rank (2025)

Abstract: Aligning language models (LMs) with curated human feedback is critical to control their behaviors in real-world applications. Several recent policy optimization methods, such as DPO and SLiC, serve as promising alternatives to the traditional Reinforcement Learning from Human Feedback (RLHF) approach.In practice, human feedback often comes in a format of a ranked list over multiple responses to amortize the cost of reading prompt. Multiple responses can also be ranked by reward models or AI feedback. There lacks such a thorough study on directly fitting upon a list of responses. In this work, we formulate the LM alignment as a listwise ranking problem and describe the LiPO framework, where the policy can potentially learn more effectively from a ranked list of plausible responses given the prompt. This view draws an explicit connection to Learning-to-Rank (LTR), where most existing preference optimization work can be mapped to existing ranking objectives. Following this connection, we provide an examination of ranking objectives that are not well studied for LM alignment, with DPO and SLiC as special cases when list size is two. In particular, we highlight a specific method, LiPO- \lambda , which leverages a state-of-the-art listwise ranking objective and weights each preference pair in a more advanced manner. We show that LiPO- \lambda can outperform DPO variants and SLiC by a clear margin on several preference alignment tasks with both curated and real rankwise preference data.

## introduction recent large language models, such as gpt-4 (openai, 2023) and gemini @xcite , have unlocked unprecedented capabilities, witnessed by impressive performance on diverse tasks from conversational chatbot to programming. a * equal contribution key step to control the behavior of such language models (lms) is to align them with curated human feedback. reinforcement learning with human feedback (rlhf) @xcite was first introduced to improve the alignment of lms with human preferences @xcite . however, rlhf is a complex process, requiring substantial memory and hyperparamter tuning. several recent works resort to alternatives of rlhf, and noticeably converge to a pairwise ranking optimization paradigm. for example, dpo @xcite optimizes a pairwise logistic loss directly from pairwise human preference data, while avoiding an explicit reward model and rl-based optimization. similarly, slic @xcite optimizes a pairwise hinge loss objective on pairwise preference data directly from human or reward model ranked samples from the supervised fine-tuned (sft) policy. rrhf @xcite starts from listwise preference data labeled by a reward model, and optimizes a pairwise contrastive objective by comparing all pairs in the list, which is analogous to the slic objective. virtually all popular preference optimization frameworks do not go beyond pairwise preferences. however, in practice, human preference data can come as a ranked list to amortize the cost of reading prompt @xcite . it is also highly efficient for sampling a list of responses from a shared prompt @xcite . these motivate us to study preference optimization on listwise data. some recent work did study preference optimization using listwise ranking objectives. for example, the dpo paper @xcite briefly touches the listwise plackett-luce model @xcite for preference modeling without any experimental results. meanwhile, pro @xcite , studied the plackett-luce model more formally. however, existing work are usually ad-hoc, and there lacks a thorough study of the listwise ranking perspective for the lm preference optimization problem, which will bring in new insights and solutions as shown in this paper. in this work, we formally formulate lm alignment as a listwise ranking problem, where the lm can potentially learn alignment more effectively from listwise preferences. this for the first time draws an explicit connection to the rich learningto-rank (ltr) literature @xcite . noticeably, the ltr literature has shown that a direct listwise optimization can be more effective than pairwise alternatives for the listwise ranking problem, with a rich set of methods of various properties and performances. in particular, popular existing methods can be mapped to existing ranking objectives. as we will show, not every listwise objective is effective and thus it is desired to conduct an close examination to identify the most effective ones for lm alignment. in figure 1 , we can see that listwise preference data can benefit existing pairwise methods like dpo even though they treat all pairs from a list equally, a problem not well studied in the literature, while our new listwise method, inspired by our general formulation, called lipo-λ, can further benefit from listwise preferences. to this end, we provide the first comprehensive study of ranking objectives under the listwise preference optimization (lipo) framework, which allows us to compare popular and state-of-the-art ranking objectives for the lm preference optimization problem. in particular, existing popular methods, such as dpo, rrhf, and slic, can be mapped to existing pairwise ranking optimization objectives from listwise data (if only pairwise data is used, it is equivalent to list size being 2). on the other hands, existing listwise methods, such as pro, usually map to ineffective listwise objectives and lipo allows us to analyze their pitfalls. furthermore, inspired by the lipo framework, we show that a new method, lipo-λ, which leverages a state-of-theart ranking objective @xcite @xcite , can achieve very competitive performance. noticeably, lipoλ allows an intuitive interpretation: it leverages a sophisticated weighting paradigms that assigns listwise-aware weighting to sampled pairs, to optimize a well-founded ranking metric @xcite , in contrast to existing methods that assign uniform weights or use weighting schema that fails to consider various factors. by a comprehensive study of various ranking objectives on reddit tl;dr, anthropichh, and openassistant tasks, we show that lipo-λ is better than existing methods, including (pairwise and listwise) dpo @xcite , slic @xcite , pro @xcite , as well as other baselines motivated by the ltr literature. our contributions are summarized as follows: ## lipo-λ under the general lipo framework, a rich family of other optimization objectives can be explored from the ltr literature. in this section, we propose a specific instantiation, lipo-λ, which builds upon a state-of-the-art ranking objective, addresses the above pitfalls in a principled manner, and performs well empirically in the experiments. our lipo-λ is based on the lambdaloss method @xcite . specifically, the training objective of lipo-λ is: as the commonly used one, where τ (i) is the rank position of y i in the ranking permutation induced by s, thus it is a listwise method even though the formula can be written in terms of pairs. in other words, there are dependencies on other items in the same list for each pair. one has the flexibility to change the gain and discount functions and we resort to the original lambdaloss configuration for its simplicity and strong empirical performance. it has been shown that this loss function can optimize the dcg metric @xcite , comparisons. there are several interesting findings by comparing lipo-λ with other methods under the lipo framework. first, the gain function g considers label score values ψ, which is ignored in virtually all existing methods. second, comparing with the list mle loss, the permutation considered is induced by the model prediction scores, instead of the static labels. @xcite showed that considering such dynamic permutations based on model predictions during training can lead to smoother optimization landscapes to optimize the non-smooth ranking objectives, resulting in better empirical performance than using the static ranks from labels. last but not least, lipo-λ can be treated as a weighted version of dpo bt over all pairs of ranked list by comparing with eq 5. instead of treating each pair equally, lambda weight is a listwise permutation aware weighting mechanism. one intuition of lambda weight is to weight each pairs by the difference of ranking metrics when they are swapped in the list @xcite . ## other ranking losses other existing or future ranking objectives may be studied under the lipo framework. in experiments, in addition to existing methods that can be treated as specific instantiations of lipo, we also study two pointwise ranking objectives, the pointwise mean squared error (mse) loss: we also consider softmax cross entropy loss as in listnet @xcite ), which again maps to a recent work called nce @xcite method. for each task, we first train a t5-large (770m) @xcite ) sft policy on the sft dataset. we pick the best checkpoint with lowest perplexity on validation split. we also train a t5-xxl (11b) pairwise reward-ranking model @xcite on the human preference dataset (see appendix f for details). we pick the best checkpoint with the highest accuracy on validation split of the human preference dataset. then we sample @xmath0.7 and @xmath1. we conduct all pair comparisons using the pairwise reward-ranking model, resulting in a winning probability matrix ψ ∈ [0, 1] k×k , after which we compute ψ k as reward model is shown to be critical in the success of alignment @xcite . this aggregation and labeling schema is valid theoretically (shah and wainwright, 2018) and works well empirically @xcite in the ranking literature. for directly optimizing on human-ranked responses without a reward model, we approximate the winning probability as p(y i y j ) = 1(rank y i is higher than y j ). see section 5.4 for an experiment on openassisant dataset @xcite . to compute the normalized ranking scores s, we set β = 0.05. we consider three types of loss functions under the lipo framework: pointwise, pairwise, and listwise. we use the open-sourced rax library @xcite to compute losses in jax. see appendix b for example usages. for pairwise losses, we utilize all 64 pairs and show in section 5.2 that it can improve performance than sampling a single pair each time as commonly done, so this setting is not only fair but also benefit pairwise methods. we use batch size 32 and learning rate 2e-5 with adafactor optimizer @xcite , which takes about 1 day to run the calibration on 32 tpu-v3 chips. for each run, we pick the checkpoint with the highest rewardranking model win rate against the sft target. evaluation. our experiments use three different approaches to evaluate following existing protocols @xcite : proxy reward model, autosxs, and human evaluation. proxy reward model computes the relative win rate of generated response against sft target over the sft validation split on the trained t5-xxl pairwise reward-ranking model. autosxs computes the relative win rate of generated response against sft target over the sft validation split using large instruction tuned model (palm 2-l-it) @xcite via few-shot in-context learning (details in appendix c). human evaluation asks human raters to assign an absolute quality score on each response and determine the relatively best one among dpo bt , dpo pl (pro) and lipoλ (details in appendix d). we note these are standard evaluation protocols in related work. ## conclusion we describe the lipo framework for lm alignment with a list of responses for each prompt with connection to ltr techniques. we generalize recent preference optimization methods and analyze pitfalls of existing methods from the ltr perspective. with comprehensive studies over existing ltr losses, we highlight lipo-λ as the best approach that builds upon the state-of-the-art ranking objectives and shows competitive performance across several tasks.