Multi-Conditional Ranking with Large Language Models (2025)

Abstract: Utilizing large language models (LLMs) to rank a set of items has become a common approach in recommendation and retrieval systems. Typically, these systems focus on ordering a substantial number of documents in a monotonic order based on a given query. However, real-world scenarios often present a different challenge: ranking a comparatively smaller set of items, but according to a variety of diverse and occasionally conflicting conditions. In this paper, we define and explore the task of multi-conditional ranking by introducing MCRank, a benchmark tailored for assessing multi-conditional ranking across various item types and conditions. Our analysis of LLMs using MCRank indicates a significant decrease in performance as the number and complexity of items and conditions grow. To overcome this limitation, we propose a novel decomposed reasoning method, consisting of EXtracting and Sorting the conditions, and then Iteratively Ranking the items (EXSIR). Our extensive experiments show that this decomposed reasoning method enhances LLMs’ performance significantly, achieving up to a 14.4% improvement over existing LLMs. We also provide a detailed analysis of LLMs performance across various condition categories, and examine the effectiveness of decomposition step. Furthermore, we compare our method with existing approaches such as Chain-of-Thought and existing ranking models, demonstrating the superiority of our approach and complexity of MCR task. We will make our dataset and code publicly available.

## introduction the rapid advancement of autoregressive large language models (llms) has significantly enhanced our ability to understand and solve nlp related tasks @xcite @xcite . among these tasks, document ranking plays a crucial role in recommendation and retrieval systems @xcite . while there has been a considerable advancement in ranking extensive document collections given a query @xcite @xcite , the nuanced task of ranking a smaller set of items based on multiple conditions-a critical requirement in numerous real-world applicationshas not been addressed in prior research. ranking a set of items according to multiple conditions has vast implications across various fields and applications. in recommendation systems, for instance, once the top candidates are shortlisted, the user experience can be significantly enhanced by offering the capability to re-rank these candidates based on specific conditions, such as genres and categories. in the realm of education, this task can be applied to the ranking of questions, enabling educators to prioritize and arrange questions effectively according to different criteria, such as subject matter. moreover, in competitive job markets, multi-conditional ranking is crucial for matching resumes to job postings, prioritizing factors like skills and experience. in this paper, we define and explore the task of multi-conditional ranking (mcr) by developing mcrank, a comprehensive benchmark consisting various item types and conditions for assessing mcr task. in addition, we also propose a novel decomposed reasoning based method, exsir, that beats strong baselines (including cot) by up to 14.4%. the new benchmark, mcrank, spans a various category of conditions, including positional, locational, temporal, trait-based, and reasoning types. we have designed mcrank to address scenarios involving one to three conditions and to assess sets of 3, 5, or 7 items. the benchmark distinguishes between two types of items: token-level items, which consist of only a few tokens, and paragraph-level items, which contain up to 150 tokens. an example of mcrank involving two conditions and three token-level items is presented in figure 1 . our initial investigations into the performance of existing llms on mcrank revealed a notable decline in accuracy as the number of items and conditions increased. specifically, we observe that the accuracy of investigated llms , i.e., openai o1-mini, gpt-4 (openai, 2023), chatgpt @xcite (both turbo versions), llama3.1-70b @xcite and mistral (7b) @xcite in correctly ranking items fell dramatically as the task scaled to three conditions and seven items, with accuracy approaching nearly 0%. to address the shortcomings of existing llms in mcr task, we introduce a novel method based on decomposed reasoning. rather than directly prompting llms to rank items based on the given conditions, our approach begins with extracting and sorting the conditions based on their priority. subsequently, we iteratively apply these sorted conditions to the item list. an illustration of our approach is provided in figure 1 . applying our method to mcrank, we observed a notable improvement, with up to a 14.4% increase in the llms' ranking accuracy. observing the impact of our approach in improving llms performance on mcrank, we conduct an in-depth analysis of the models' performance, dissecting the results based on the types of items and conditions involved. additionally, we exam-ine the accuracy of the decomposition step within the evaluated llms shedding light on observed behavior of llms on mcrank. to delve deeper into the significance of the decomposition process, we incorporate a zero-shot chain-of-thought @xcite approach, further underscoring the importance of segmenting the mcr task into multiple steps to achieve improved outcomes. finally, we employ sfr @xcite , colbert (khattab and zaharia, 2020) and rankgpt (based on gpt-4) @xcite , rankers renowned for their performance in document ranking @xcite , to represent existing rankers. our comparison illustrates that, despite their success in existing ranking tasks, they exhibit considerably inferior performance on mcrank. ## multi-conditional ranking the task of multi-conditional ranking is designed to shift the focus from traditional ranking tasks, which typically involve ordering a large set of items based on a single query. instead, this task concentrates on sorting a smaller, pre-selected set of items according to multiple conditions. these conditions may not only conflict with one another but also carry varying levels of priority, adding layers of complexity to the ranking task. moreover, each condition may specify a complete order for all items or only provide a partial ordering instructions for the after extracting items, we determine their golden ranking based on their corresponding labels. the conditions may specify a complete or partial order for item placement. ## experimental details models: we evaluate both commercial llms, openai o1-mini, gpt-4 (openai, 2023), chat-gpt @xcite (both turbo versions), as well as open-source llms, llama3.1-70b (instruct) @xcite and mistral (mistral-7b-instruct-v0.2) (jiang et al., 2023) on mcrank. to address the mcr task, in the base setting, we input the string of conditions along with the list of items into the prompt, instructing the llms to organize the list according to the conditions. for paragraph-level items, we assign a unique label, "item-k," to each item. the task for the model is then to rank the items but to output the sequence of sorted labels-item-k-instead of the items themselves. the details of all prompts utilized in this study are provided in the appendix. we designed the prompts to mirror previous ranking works, while avoiding framing the task as a standard document monotonic-ranking task. evaluation metric: given that the mcr task defined in this paper represents a broader and more complex variation of previously defined ranking tasks-where, unlike those tasks, the significance or relevance of items in the gold ranking doesn't necessarily diminish in a linear order-traditional ranking metrics like mrr or ndcg @xcite are not suitable for our context. consequently, we evaluate model performance on the mcr task using exact match accuracy, scoring 1 for a fully correct ranking and 0 for an incorrect one. additionally, we employ an averaged accuracy metric, measuring the mean number of correctly ranked items per sample for a more nuanced assessment of models. ## experiments we explore the effectiveness of llms and the impact of exsir on the mcr task using the mcrank benchmark, starting with an evaluation of models' performances. subsequently, we provide percategory breakdown of performances to assess the impact if each condition category. we assess the accuracy of the decomposition step for each model to gauge its impact on exsir's functionality. lastly, to underscore the importance of decomposed reasoning through multiple steps, we compare our method's performance against zeroshot cot prompting and existing ranking models. ## related work llms have achieved significant success in ranking tasks, but most efforts focus on ranking passages based on a single query, despite the broad realworld applications of ranking. ranking with llms: in recent years, llms have become pivotal in addressing ranking related tasks. initially focusing on encoder-based rankers @xcite , the rapid advancement of autoregressive llms has led to the development of methodologies that utilize these models as rankers, achieving unparalleled performance across various benchmarks @xcite . however, despite these advances, the majority of llm-based ranking efforts have concentrated on ordering extensive lists of passages based on a query, often overlooking the diverse applications of ranking in real-world scenarios. our work closely aligns with developments in recommendation systems, such as the conditional methods proposed by @xcite , which only considers a limited concept of condi-tion in regard to variety and complexity compared to our notion of multi-conditional ranking. decomposed reasoning with llms: as llms grow stronger, decomposed reasoning has emerged as a fundamental strategy to enhance their capabilities by segmenting complex tasks into smaller, more manageable components. this decomposition can be straightforward, utilizing a single llm, as seen in approaches like chain-of-thought @xcite , tree-of-thought @xcite , and self-verification @xcite . alternatively, it can involve more complex interactions among multiple models within a multi-agent systems @xcite . prior research has successfully integrated decomposed reasoning into various tasks, including question answering @xcite , retrieval-augmented generation (rag) @xcite , and mathematical reasoning @xcite . ## conclusion we explore the multi-conditional ranking task, an essential but underexplored aspect of ranking in real-world applications. introducing mcrank benchmark, we have highlighted the challenges llms face when ranking a small set of items under a variety of complex and sometimes conflicting conditions. our investigation reveals a significant performance drop in llms as the number of conditions and items increases. to address this, we introduce exsir, a novel decomposed reasoning method that improves llm performance by up to 14.4% in accuracy. we also analyze the performance of llms across various condition categories and the effectiveness of the decomposition step in enhancing accuracy. finally, by contrasting our approach with other existing methodologies such as cot and existing rankers, we demonstrate its effectiveness and the complexity of the mcr task.