K-order Ranking Preference Optimization for Large Language Models (2025)

Abstract: To adapt large language models (LLMs) to ranking tasks, existing list-wise methods, represented by list-wise Direct Preference Optimization (DPO), focus on optimizing partial-order or full-order list ranking consistency for LLMs to enhance their ranking abilities.However, we argue that optimizing top-K ranking consistency could be more appropriate for real-world applications. There are two main reasons: (1) users are typically concerned with only the top-K results, making top-K ranking more important, and (2) tail items often lack precise feedback, making top-K ranking more reliable. Based on this, we propose \textbf{K} -order Ranking \textbf{P} reference \textbf{O} ptimization (KPO) by extending the DPO’s Plackett-Luce model to accommodate top-K rankings. Additionally, recognizing that the number of important items can vary across queries, we extend KPO to dynamically determine appropriate K for different samples and introduce a curriculum learning strategy to boost training efficiency. Extensive experiments demonstrate the effectiveness of KPO, highlighting its high sample efficiency and robustness to noise. The code is available at https://github.com/Lanyu0303/KPO.

## introduction large language models (llms) have shown great potential in addressing a wide range of real-world tasks @xcite . by leveraging their semantic reasoning abilities and extensive world knowledge, llms can more effectively capture the nuanced relationships between queries and candidate items, making them also promising for ranking tasks @xcite -the core of many real-world applications such as product search @xcite and recommendation @xcite . however, as illustrated in fig .(1), ranking tasks extend beyond evaluating the †corresponding authors. relevance of individual candidates to a user query; they require ranking a list of candidates. yet, llms are not explicitly trained to optimize list-wise ranking preferences during pretraining. this limitation has sparked greater research efforts to enhance llms' list-wise ranking capabilities @xcite . among existing approaches, the list-wise direct preference optimization (dpo) method @xcite has emerged as a promising technique for optimizing llms to generate ranked outputs that align with human preferences directly at the list level. according to the ranking consistency optimized for a list, existing methods can be categorized into: • partial-order method (e.g., s-dpo @xcite ), which simply optimizes the ranking consistency where "the best item is better than all others," i.e., y 1 ≻ all others. this method focuses on the ranking of the best one, failing to optimize the fine-grained ranking consistency. towards the top-k order ranking preference alignment, we propose k-order ranking preference optimization (kpo). the core idea is to extend existing dpo methods' plackett-luce preference model @xcite , originally designed for full rankings, to accommodate top-k rankings. intuitively, kpo works by increasing the relative log probability of each top-k item over all its subsequent items, ensuring both the fine-grained order among the top-k items and the order between the top-k items and the others. as discussed, kpo is expected to outperform full-order methods due to its closer alignment with real-world scenarios. additionally, theoretical analysis demonstrates that kpo surpasses existing partial-order methods. taking it a step further, in real-world scenarios, the number of most relevant items can vary across queries. to address this, we extend kpo to handle varying k values across samples, incorporating a strategy to adaptively determine k based on llm confidence in assessing item relevance. furthermore, to accommodate varying k, we incorporate a curriculum learning strategy into kpo to simplify the learning process. specifically, we guide kpo to focus on k-order optimization progressively, starting from smaller k and gradually increasing to larger k. this approach is motivated by the fact that higher k introduces greater learning challenges, as it requires distinguishing more complete and fine-grained rankings. the main contributions of this work can be summarized as follows: ## related work in this section, we delve into related studies from two perspectives: llm-based ranking and preference alignment in llms. ## problem definition consider a ranking dataset d comprising querycandidate pairs, where each k-th instance (q (k) , c (k) ) ∈ d consists of: (1) a query q (k) representing an information need (e.g., search query, recommendation context). (2) a candidate set m } containing m items to be ranked. the llm takes as input a concatenated sequence x (k) = (q (k) , c (k) ) and aims to generate a permutation m }, where ∀y (k) @xmath0, the symbol ≻ represents a pair-wise preference relationship. when ambiguity is absent, we omit the superscript (k) for notational simplicity (e.g., y i instead of y (k) i ). we instantiate this framework through two representative ranking applications: • sequential recommendation: the query q ≜ [v 1 , v 2 , . . . , v m ] encodes a user's interaction history, where v j denotes the j-th consumed item. ## methodology we first review foundational work in preference modeling to establish the necessary background. then, we introduce the proposed model in detail. ## experiments in this section, we aim to answer the following research questions (rq): • rq1: how does kpo perform in the recommendation and product search tasks? • rq2: what are the effects of the key components and hyperparameters? • rq3: how does kpo perform in terms of sample efficiency and robustness to noisy logits? ## conclusion in this study, we propose a novel method called kpo, designed to address the limitations of existing approaches that rely on full-order or partialorder ranking but often neglect the significance of top-k ranking. in detail, we introduce the k-order ranking, which prioritizes fine-grained ranking consistency for the top-k items while disregarding less relevant ones. building on this foundation, we extend the pl model to accommodate top-k ranking and develop the corresponding kpo loss. additionally, we derive a theoretical formula for the optimal accuracy achievable by kpo, thereby theoretically demonstrating that kpo outperforms s-dpo. considering the varying number of relevant items across queries, we make kpo query-adaptive, enabling it to dynamically adjust k for each query. to further improve training efficiency and stability, we introduce k-aware curriculum learning, which allows llms to progressively learn from simpler to more complex data. extensive experiments show that kpo significantly outperforms existing preference alignment methods, highlighting not only the effectiveness of top-k ranking but also the critical role of query-adaptive k.