Few-shot Prompting for Pairwise Ranking: An Effective Non-Parametric Retrieval Model (2024)

Abstract: A supervised ranking model, despite its effectiveness over traditional approaches, usually involves complex processing - typically multiple stages of task-specific pre-training and fine-tuning. This has motivated researchers to explore simpler pipelines leveraging large language models (LLMs) that can work in a zero-shot manner. However, since zero-shot inference does not make use of a training set of pairs of queries and their relevant documents, its performance is mostly worse than that of supervised models, which are trained on such example pairs. Motivated by the existing findings that training examples generally improve zero-shot performance, in our work, we explore if this also applies to ranking models. More specifically, given a query and a pair of documents, the preference prediction task is improved by augmenting examples of preferences for similar queries from a training set. Our proposed pairwise few-shot ranker demonstrates consistent improvements over the zero-shot baseline on both in-domain (TREC DL) and out-domain (BEIR subset) retrieval benchmarks.

## introduction development of novel neural architectures and training methodologies @xcite @xcite @xcite have substantially outperformed the unsupervised approaches. commonly, these neural approaches involve deep interactions between embedded representations of queries and documents @xcite thus overcoming the vocabulary mismatch problem of discrete term representations. however, to achieve good performance, not only do these deep neural models require a large number of training data in the form of pairs of example queries and their relevant documents @xcite , but the effectiveness of these models also depends on a number of ad-hoc decision choices, e.g., the following. with such a large number of decision choices available, it is difficult to converge on a set of 'best practices' for designing the pipeline of a supervised ranker. instead, in this paper, we present a relatively simple approach of leveraging the information from a training set of examples of query and relevant document pairs without requiring any parametric training. the motivation for this simple yet effective pipeline stems from the recent developments in large language models (llms), which exhibit emergent capabilities of modeling semantics @xcite . by including a task definition and, optionally, examples of labelled or unlabelled data, these models can perform competitively on unseen tasks without fine-tuning @xcite . as such, they offer a compelling alternative to the highly data-driven fine-tuning currently applied in the field of neural retrieval. although recent work has employed llms for ranking, these approaches either simply use a zeroshot approach, thus not leveraging the benefits of a non-parametric memory (i.e., of a training set) @xcite , or they employ zero-shot inference only as an initial step for training data augmentation prior to train a supervised ranking model @xcite . in contrast, our work, while on the one hand, employs an unsupervised approach (i.e., with no parametric training involved), it is able to make use of the training data of query-relevance examples via few-shot prompting on the other. more specifically, as outlined in figure 1 , in our approach, following the methodology of @xcite we input a query and a pair of documents seeking to estimate the relative preferential order between the pair. however, in contrast to the prp (pairwise rank prompting) method of @xcite , we input a set of queries (from an available training set) that are related to the current query in terms of an abstract similarity measure. this is motivated from the well-known cognitive bias attribute substitution effect in psychology, where to answer an unknown question, human brains recollect known answers to related questions and eventually process information from them to answer the unknown one @xcite . in our work, we emulate this behavior of attribute substitution heuristics on llms, where in additional context @xcite , we provide examples of related queries and their relevant documents. the hypothesis is that llms, with their inherent language processing abilities, should be able to make a more informed judgment of pairwise relevance by processing the examples provided. this proposed workflow requires investigating a number of research questions and challenges including: "how to retrieve an effective set of related queries for a given input?", and "what is the downstream effect of the similarity of the information needs of the related queries". our experiments show that an embedding-based neighborhood to retrieve related queries yields better downstream effectiveness than a lexical model, and we also show that a small number of examples can, in fact, lead to consistent improvements over the zero-shot prp approach @xcite . our experiments also demonstrate the potential of this unsupervised method for out-of-domain generalization. more specifically, we show that examples queries from ms-marco lead to improvements on trec covid and scifact test collections. the source code of our proposed llm-based few-shot ranker foot_0 is made available for research purposes. in-context learning (icl). in-context learning or few-shot learning is an inference strategy that differs from the standard notion of supervised learning in the sense that labeled examples are appended to a model instruction improving effectiveness in an out-of-domain downstream task @xcite . though initially considered to guide sequence generation in tasks such as question answering and abstractive summarization @xcite , icl has been shown to be effective in classification-style tasks @xcite and, therefore, could be effective in a cross-encoder setting for ranking. in terms of example selection for icl, prior work has found that conditioning chosen examples on the current test instance is effective @xcite . ## conclusions and future work we proposed a novel example selection process inspired by neural retrieval training processes, which improves unsupervised performance in a pair-wise ranking setting by exploiting in-context learning and is adaptable beyond a target domain. this non-parametric approach helps eliminate several decision choices involved in a supervised learning-to-rank pipeline, e.g., the architecture, the pretraining, index construction, negative sampling, distillation, etc. despite the simplicity, our experiments confirm that the few-shot prp not only significantly outperforms the zero-shot prp on indomain but also either statistically outperforms monot5 (covid dataset) or is statistically indistinguishable from it (scifact dataset). as future work, we plan to explore ways of selecting a variable number of examples on a perquery basis @xcite or consider an opendomain icl approach of using unlabelled data as contexts, @xcite , e.g., information from wikipedia, for improving the ranking task further.