Incorporating Relevance Feedback for Information-Seeking Retrieval using Few-Shot Document Re-Ranking (2022)

Abstract: Pairing a lexical retriever with a neural re-ranking model has set state-of-the-art performance on large-scale information retrieval datasets. This pipeline covers scenarios like question answering or navigational queries, however, for information-seeking scenarios, users often provide information on whether a document is relevant to their query in form of clicks or explicit feedback. Therefore, in this work, we explore how relevance feedback can be directly integrated into neural re-ranking models by adopting few-shot and parameter-efficient learning techniques. Specifically, we introduce a kNN approach that re-ranks documents based on their similarity with the query and the documents the user considers relevant. Further, we explore Cross-Encoder models that we pre-train using meta-learning and subsequently fine-tune for each query, training only on the feedback documents. To evaluate our different integration strategies, we transform four existing information retrieval datasets into the relevance feedback scenario. Extensive experiments demonstrate that integrating relevance feedback directly in neural re-ranking models improves their performance, and fusing lexical ranking with our best performing neural re-ranker outperforms all other methods by 5.2% nDCG@20.

## introduction user queries can be categorized as navigational (retrieving a specific document), transactional (retrieving a website to perform a particular action) or informational @xcite . for informationseeking queries, users might want to learn about a new topic or might be unfamiliar with the search domain. therefore they potentially do not use common keywords of the domain which decreases performance @xcite . furthermore, they might want to find complementary information from diverse sources or consider different aspects of a topic @xcite . lastly, informationseeking queries can also be used to keep up with the latest developments on a topic. concretely, these queries are encountered during scientific literature review @xcite , when looking for news and background information @xcite , during argument retrieval, @xcite or in the legal context for case law retrieval @xcite . formulating effective queries to satisfy the complex information need in these scenarios is difficult. on the contrary, a user can easily judge whether a document is relevant to their query. therefore, information obtained from the user when interacting with the search results, known as relevance feedback, can be used in the search. this can be obtained implicitly from click logs @xcite or explicitly by asking users whether a document is relevant @xcite . we focus on explicit feedback because it is clean compared to implicit feedback and existing information retrieval datasets can be transformed into this scenario. in both settings, the amount of feedback is limited, since users will provide feedback only on a few documents. incorporating relevance feedback in information retrieval (ir) systems is well-established for lexical retrieval @xcite @xcite . these systems incorporate the feedback by expanding the query with terms extracted from relevance feedback documents. while these approaches can alleviate the lexical gap, they inherently struggle with semantics because they represent text as a bag of words. additionally, lexical query expansion methods have the disadvantage that their latency increases with the number of query terms @xcite . to mitigate these issues, neural retrieval and reranking methods have been proposed and recently outperformed lexical retrieval @xcite @xcite . state-of-the-art retrieval results are obtained in a two-stage setup: first, an efficient and recall-optimized retrieval method (e.g. dense or lexical retrieval) retrieves an initial set of documents. subsequently, a neural re-ranker optimizes the rank of the documents. however, there exists no neural re-ranking model that directly incorporates relevance feedback. to this end, we explore how relevance feedback can directly be integrated into neural re-ranking models. this is difficult because state-of-the-art models have millions of parameters and require a large amount of training data, while only a limited amount of relevance feedback per query is available. we make use of recent advances in parameterefficient fine-tuning @xcite and few-shot learning @xcite to address the challenges of model re-usability and learning from limited data. concretely, we present a knn approach that re-ranks documents based on their similarity to the feedback documents. we further propose to finetune a re-ranking model from only the relevance feedback for each query. we explore the effectiveness of our approach with a varying number of feedback documents and evaluate its computational efficiency. to evaluate our models, we transform four existing ir datasets into the re-ranking with relevance feedback setup. our final model combines the strengths of lexical and neural re-ranking using reciprocal rank fusion @xcite . in summary, our contributions are as follows: ## datasets large-scale ir datasets mostly target use cases where a user has a less complex information need, e.g., looking for a factoid answer @xcite @xcite . these are usually sparsely annotated, i.e. there is only a single (or few) judged relevant documents per query. however, for our information-seeking use case, we are interested in queries where many relevant documents exist. therefore, we select datasets where a large set of relevant documents per query are judged. further, the datasets should target suitable use cases containing queries that have a diverse set of relevant documents. for example, the query "what is the origin of covid-19" from trec-covid, has relevant documents about the geographical location of the first cases, the genetic origins of the virus, and animals that likely have transmitted the disease to humans. specifically, we consider robust04 @xcite , trec-covid @xcite , trec-news @xcite , and webis-touché @xcite . an overview of all datasets with their statistics is provided in for our experiments, we create training, validation and test splits in a 3:1:1 ratio, by randomly assigning each query to one set. we further conduct three random shuffles over the assignment of a query into the training, validation, and test set. we report the averaged results over the shuffles. ## task setup to incorporate relevance feedback in any retrieval process, a multi-stage approach is required. we propose a multi-phase task setup which is visualized in figure 1 . in phase 1, the relevance feedback 2 more details on the datasets in appendix a. the query q and feedback documents r are used for query expansion and the second round of retrieval. further, a re-ranking model is fine-tuned using the user-selected feedback documents. phase 3: the documents are re-ranked using the finetuned re-ranker, obtaining the final document ordering. to improve performance, the ranking from the re-ranker and the second phase are be fused. specifically, in phase 1, an initial retrieval is conducted with the query q against the document collection. for lexical retrieval, we use bm25 as it is robust in a zero-shot setting on a diverse set of domains @xcite . next, we select the top @xmath0, 4, 8} relevant and non-relevant documents from the first-stage retrieval according to the judgments in the dataset, i.e., there are 2k documents selected per query. we refer to these documents as feedback documents r. by selecting the top judged and retrieved documents, this process simulates a user provid-ing relevance feedback. 3 for our evaluation, we remove the feedback documents from the relevance judgments (i.e. we use the residual collection, @xcite in order to evaluate the ability of the models to rank non-selected documents higher. in phase 2, we use the 2k feedback documents for query expansion and a second retrieval step. we extract e terms per relevant feedback document and append them to the query resulting in the expanded query for second-stage retrieval. 4 furthermore, the feedback documents are used to finetune a re-ranking model. starting from a common base model, a new model is fine-tuned for every query. to exploit the small number of feedback documents most effectively, we employ few-shot learning when fine-tuning the re-ranker. finally, in phase 3, the documents are scored using the query-specific re-ranker from the second phase. additionally, the ranking from bm25-qe can be incorporated in the final document ranking. we experiment with different models and settings, details are described in §5. the re-ranking could also be performed on the documents from the first-stage retrieval. however, since the feedback documents are available and query expansion generally improves recall (which is important for the re-ranking performance), we chose to not experiment with re-ranking the firststage retrieval documents. this also improves the evaluation, because all models re-rank the same set of documents. evaluation metrics. to measure the ranking performance we use ndcg@20 @xcite implemented by pytrec_eval (gysel and de rijke, 2018). this metric considers graded relevance labels. we chose the cut-off at 20 to take the large number of relevant documents per query into account. beyond ranking performance, we also focus on retrieval/re-ranking latency and parameter efficiency. the response time of ir systems is generally crucial for user satisfaction @xcite . therefore, we evaluate the time for retrieval, query expansion, fine-tuning, and re-ranking. since we fine-tune a model per query, the memory footprint of the model should be small. this allows keeping many models in memory at the same time or quickly reloading a model whenever a user revisits a query. ## conclusion in this work, we introduced a few-shot learning task for incorporating relevance feedback in neural re-ranking models. we further transformed existing ir datasets into the few-shot setting. most importantly, we have introduced different methods for incorporating relevance feedback directly into neural re-ranking models. the proposed knn approach is particularly computationally efficient, however, by itself, it cannot outperform bm25 with query expansion. since the knn method does not add significant latency to the re-ranking, it can be combined with bm25 query expansion, which outperforms the latter by 2.6% ndcg@20. our second re-ranking method based on a cross-encoder model performs on par with bm25 with query expansion. regarding its latency, we show that finetuning on a query basis is feasible since a majority of the time is spent on re-ranking and not fine-tuning. similar to knn, performing rank fusion of the two approaches yields a high performance gain of 5.2% ndcg@20. advantageously, reciprocal rank fusion is very stable in our setting, mitigating weaknesses of individual model-task combinations.