Consolidating Ranking and Relevance Predictions of Large Language Models through Post-Processing (2024)

Abstract: The powerful generative abilities of large language models (LLMs) show potential in generating relevance labels for search applications. Previous work has found that directly asking about relevancy, such as "*How relevant is document A to query Q?*”, results in suboptimal ranking. Instead, the pairwise-ranking prompting (PRP) approach produces promising ranking performance through asking about pairwise comparisons, e.g., "*Is document A more relevant than document B to query Q?*”. Thus, while LLMs are effective at their ranking ability, this is not reflected in their relevance label generation.In this work, we propose a post-processing method to consolidate the relevance labels generated by an LLM with its powerful ranking abilities. Our method takes both LLM generated relevance labels and pairwise preferences. The labels are then altered to satisfy the pairwise preferences of the LLM, while staying as close to the original values as possible. Our experimental results indicate that our approach effectively balances label accuracy and ranking performance. Thereby, our work shows it is possible to combine both the ranking and labeling abilities of LLMs through post-processing.

## introduction generative large language models (llms) have shown significant potential on question answering and other conversation-based tasks @xcite owing to their extraordinary generative abilities and natural language understanding capabilities. naturally, previous work has further investigated the application of llms to other areas, including search and recommendation tasks @xcite . the goal here is to rank items according to their relevance to a certain query. generally, existing approaches have applied llms to this task in two different ways: first, as pseudo-raters, llms are asked to simulate human raters by generating a relevance label for each query-document pair @xcite , for example, through prompts such as "how relevant is document a to query q?" secondly, an llm can also be asked directly about the ordering of documents for a query. for example, the pairwise-ranking-prompting (prp) method @xcite uses a prompt like "is document a more relevant than document b to query q?" alternatively, llms can be asked to generate the entire ranking through a prompt like "rank the following documents by their relevance to query q: document a, document b, document c, etc." @xcite thus, there are several distinct modes by which llms can be used for ranking purposes, which provide different kinds of output. each mode of applying llms to ranking tasks offers distinct advantages in terms of performance and efficiency. the pseudo-rater mode is currently favored in llm applications within ranking systems due to its simplicity and high efficiency @xcite @xcite . given the high costs associated with deploying or training llms for high-throughput applications like search and recommendations, it is, so far, only efficiently feasible to use llms as pseudo-raters to label a fraction of raw data in zero-shot or few-shot fashion as a replacement of more expensive human raters. however, the general llms are not tuned to generate meaningful ranking scores, as a result, there is still an apparent gap between state of the art (sota) ranking performance and the performance reached when leveraging llm pseudo-labels for model training @xcite . in parallel to exploring the costly fine-tuning of llms as ranking specialists @xcite , previous work has also investigated the direct ranking modes of llms, where no finetuning is involved. some of these direct ranking modes, such as prp @xcite , can reach sota ranking performance that is on-par with llms finetuned for ranking. moreover, prp enables open-source (oss) llms to outperform the largest commercial models like @xcite . however, document scoring by prp solely considers the resulting order of the candidate list, and thus, the absolute values of scores are meaningless. this makes prp results unsuitable to be directly used as pseudo-labels. for example, the prp ranking score of a fair candidate in the list of only poor candidates would be comparable to that of a good candidate in the list of strong competing candidates, (see example in figure 1 ). how to effectively combine these direct ranking modes with the pseudo-rater mode to consolidate ranking and relevance predictions of llms remains an essential challenge in applying llms to real world main stream applications. in this work, we study post-processing methods to do the consolidation, especially for the case when we have no human labelled data. we first define the problem in llm ranking in section 3, and propose our post-processing methods to consolidate llm predictions for unlabelled data in section 4. we discuss our experiments on public ranking datasets in section 5 and show our methods could approach the state of the art ranking performance with minimal tradeoff in relevance prediction performance in section 6. our contributions include: ## related work the strong capability of llms in textual understanding has motivated numerous studies leveraging llm-based approaches for textual information retrieval @xcite @xcite . before the generative llm era, the focus was more on finetuning pre-trained language models (plms) such as t5 @xcite or bert @xcite for the supervised learning to rank problem @xcite , which becomes less feasible with larger generative llms. two popular methods--relevance generation @xcite and query generation (sachan et al., 2022)--aim to generate per-document relevance scores or retrieval queries using generative llms. these methods are also termed pointwise approaches for ranking. more recent works @xcite @xcite explore listwise ranking generation approaches by directly inserting the query and a list of documents into a prompt. pairwise order generation through pairwise prompts @xcite turns out to be very effective for ranking purposes, especially for moderated-sized llms. however, none of these ranking approaches using generative llms attempt to consolidate the results with relevance generation. previous works on non-llm neural rankers @xcite focus on balancing or aligning regression with ranking objectives during the model training, which is unfortunately not feasible for llms using zero-shot or few-shot prompting. post-processing methods that calibrate model predictions using some validation data could be potentially applicable. originally developed for classification model calibration @xcite , these methods include parametric approaches like platt scaling @xcite for binary classification; piecewise linear transformation @xcite for regression; and non-parametric approaches like isotonic regression @xcite , histogram binning, and bayesian binning @xcite . but how effectively these post-processing approaches could be extended to llm-based ranking and relevance predictions has not been well studied in existing literature. ## problem formulation we formulate the problem of consolidating ranking and relevance predictions within this framework. given a set of queries, for each query q, we have a set of corresponding candidate documents {d} q , and their ground truth labels, {y} q , as their relevance evaluations, such as graded relevance. our first goal is to predict the relevance labels based on the content of each corresponding candidate. our second goal is to predict a ranked list of candidates, and we use {r} q to denote the rank of each candidate in this predicted ranking. the predicted ranking is optimal when the ranks align with the order of the relevance labels: r @xmath0, d j ) belonging to the same query q. taken together, our overall task is to optimize llm predictions for both relevance estimation and ranking performance. ## the methods this section presents our post-processing methods to consolidate the ranking scores ŝ as well as the pairwise preferences from the llm ranker mode and the relevance estimation ŷ from the pseudorater mode, aiming to optimally balance ranking and relevance prediction performance. to make a fair comparison with previous llm rankers, we stick to zero-shot prompting results with no training or finetuning. specifically, we introduce a constrained regression method to find minimal perturbations of the relevance predictions ŷ such that the resulting ranking matches the the pairwise preference predictions of prp. additionally, we also introduce an efficient version of our constrained regression method that avoids querying an llm to construct the complete quadratic number of pairwise constraints by selecting a linear-complexity subset of pairwise comparisons. finally, with the constrained regres-sion to consolidate, we propose a ranking-aware pseudo-rater pipeline that leverages both rating and ranking capabilities of llms to make high-quality ratings for search. ## experiment setup we conduct experiments using several public ranking datasets to answer the following research questions: ## conclusion in this work, we have studied the problem of consolidating ranking and relevance predictions of llms. we have found that the direct scores from the zeroshot pairwise ranking prompting (prp) poorly correlate with ground truth labels. to leverage the superior ranking ability of prp while aligning closely with the ground truth labels, we have investigated post-processing methods and proposed a class of constrained regression methods that combine pointwise ratings from the llm raters and pairwise constraints from the prp rankers to take advantage of the two. we have demonstrated with experiments on public ranking datasets that our methods are efficient and effective, offering competitive or superior ranking performance compared to the prp baseline and relevance prediction performance akin to the pointwise llm rater. last but not least, we have proposed a novel framework on how to effectively use generative llms to generate ranking-aware ratings, foundation for llm-powered search ranking.