Beyond Yes and No: Improving Zero-Shot LLM Rankers via Scoring Fine-Grained Relevance Labels (2024)

Abstract: Zero-shot text rankers powered by recent LLMs achieve remarkable ranking performance by simply prompting. Existing prompts for pointwise LLM rankers mostly ask the model to choose from binary relevance labels like “Yes” and “No”. However, the lack of intermediate relevance label options may cause the LLM to provide noisy or biased answers for documents that are partially relevant to the query. We propose to incorporate fine-grained relevance labels into the prompt for LLM rankers, enabling them to better differentiate among documents with different levels of relevance to the query and thus derive a more accurate ranking. We study two variants of the prompt template, coupled with different numbers of relevance levels. Our experiments on 8 BEIR data sets show that adding fine-grained relevance labels significantly improves the performance of LLM rankers.

## introduction large language models (llms) such as @xcite and palm 2 @xcite have demonstrated impressive zero-shot performance on a variety of nlp tasks. recently, there has been a growing interest in applying llms to zero-shot text ranking, with remarkably impressive results. the earliest zero-shot llm rankers are pointwise @xcite , which score one query and one document at each time and rank the documents based on the scores. lately, pairwise @xcite and listwise @xcite llm rankers also show strong performance, but they cannot scale to long lists and still largely rely on a high-quality first-stage ranking. a typical category of pointwise llm rankers is relevance generation @xcite . in this method, the llm is prompted to answer whether a document is relevant to the query. existing pointwise llm rankers mostly ask the llm to answer "yes" or "no" and use their likelihood to derive a ranking score. nevertheless, some documents cannot always be accurately classified into these two categories as they may not directly answer the query but still contain helpful information. studies on human subjects show that using binary options sometimes leads to biased answers @xcite . instead, providing reasonably fine-grained options can lead to more reliable results @xcite @xcite . actually, in information retrieval data sets, the annotation guidelines for human annotators often employ multiple relevance levels, like the 3level scale used in trec-covid @xcite and trec-robust @xcite , as well as the 4-level scale used in trec-dl @xcite . we believe that a zero-shot llm ranker might share the same behavior pattern with human annotators. therefore, we propose to explicitly provide finegrained relevance labels in the prompt to zero-shot llm rankers. instead of asking the llm to choose between two options, we provide the llm with fine-grained relevance labels, such as "highly relevant", "somewhat relevant" and "not relevant" and collect their likelihood scores from llm predictions to derive the ranking score. the intuition is that the intermediate relevance labels in the prompt serve as a "cue" to the llm to distinguish partially relevant documents from fully relevant or fully irrelevant ones. our evaluation on 8 beir @xcite ) datasets demonstrates that simply adding intermediate relevance labels significantly boosts llm ranking performance across different datasets, regardless of the actual ground-truth label granularity. an in-depth analysis reveals that the proposed new prompt enables llm rankers to distinguish documents previously indistinguishable with the binary-option prompt. zero-shot llm assessors. another related research area @xcite employs llms as assessors, where finegrained relevance labels are also provided in the prompt. however, these methods do not use the likelihood scores of fine-grained relevance labels. the goal of llm assessors is to provide a relevance label for every query-document pairs that aligns with the ground-truth relevance label, potentially created by human assessors. llm assessors are usually used to create an evaluation data set, which can be used to reliably evaluate different ranking models. this is different from llm rankers, which typically only need to ensure that the relative order of the top-ranked documents are accurate. ## conclusion we explore pointwise zero-shot llm rankers which score fine-grained relevance labels (e.g., "somewhat relevant") instead of binary labels. we propose to either provide intermediate relevance labels such as "somewhat relevant" as additional choices for the llm or ask the llm to rate the relevance between query-document pairs using a rating scale. then we aggregate the llm likelihood scores of different relevance labels into ranking scores to rank the documents. further experiments illustrate that the performance gains are not solely attributable to more precise relevance estimation by using more samples, as only using the log-likelihood of one relevance labels can also achieve similar performance gain. instead, it is possible that the inclusion of fine-grained relevance labels in the prompt may guide llms to better differentiate documents, especially those ranked at the top. we believe that this approach can be extended beyond information retrieval to many other applications @xcite . for example, the same method can be applied for recommendation @xcite , where the llm is asked to rate how likely a user would buy an item.