Few-shot Reranking for Multi-hop QA via Language Model Prompting (2023)

Abstract: We study few-shot reranking for multi-hop QA (MQA) with open-domain questions. To alleviate the need for a large number of labeled question-document pairs for retriever training, we propose PromptRank, which relies on language model prompting for multi-hop path reranking. PromptRank first constructs an instruction-based prompt that includes a candidate document path and then computes the relevance score between a given question and the path based on the conditional likelihood of the question given the path prompt according to a language model. PromptRank yields strong retrieval performance on HotpotQA with only 128 training examples compared to state-of-the-art methods trained on thousands of examples — 73.6 recall@10 by PromptRank vs. 77.8 by PathRetriever and 77.5 by multi-hop dense retrieval.

## introduction many information-seeking queries are in the form of multi-hop questions. for instance, to answer the question "what 1988 christmas comedy film did brian-doyle murray star in?", we need to (i) search for movies starring brian murray, then (ii) identify which of them were released in 1988 during christmas. evidence required to answer such questions is often dispersed in different documents, requiring sequential, multi-step reasoning to reach the answer @xcite , typically referred to as multi-hop question answering (mqa). given a multi-hop question and a large document corpus, existing mqa systems largely follow a retrieve-then-read pipeline, where a retriever module first identifies relevant documents from the corpus, and a reader module produces the answer based on the retrieved output @xcite promptrank, using only 128 examples, outperforms drkit @xcite and performs closely to multi-hop dense retrieval @xcite . both fully supervised models are trained on ~90k examples. 2020; @xcite @xcite . the retriever module is trained to predict the ground-truth evidence document(s) given the question @xcite @xcite . however, curating large datasets of question-document pairs is expensive, especially for low-resource languages or domains that require unique expertise (e.g., medical or legal documents), thus creating a bottleneck for building qa pipelines @xcite . moreover, resorting to heuristics for data labeling can lead to incorrect annotation @xcite . this difficulty is further exacerbated in the case of multi-hop questions, as they need to be annotated with multiple support documents. the majority of existing data-efficient retrieval and reranking methods are restricted to singlehop qa, and it is unclear how to extend them to the multi-hop setting. for instance, @xcite proposed "recurrent span retrieval" to obtain psuedo question-document pairs in an unsupervised way for single-hop qa. however, in the multi-hop case, it is less likely that we can retrieve recurrent spans from multiple documents that follow a valid reasoning trajectory. moreover, their method requires intensive pretraining on the obtained corpus. @xcite focus on weakly supervised multi-hop qa retrieval, yet their method uses corpus-specific (e.g., wikipedia) heuristics and also requires pretraining. this motivates the need for data-efficient multi-hop retrieval methods that (i) work out-of-the-box without requiring additional (pre)training, and (ii) do not rely on hand-designed heuristics for data collection and annotation. to this end, we present promptrank, which leverages the power of large language models (llms) for few-shot multi-hop retrieval. promptrank combines a simple unsupervised retrieval method i.e., tf-idf similarity, with an llm reranker that scores the relevance of document paths to a question based on the conditional likelihood of generating the question given the path. our approach makes use of instruction-based prompting @xcite to steer the llm towards assigning higher scores to more relevant support document chains. 2 to calibrate the model's reranking scores and alleviate prompt sensitivity @xcite , we borrow techniques from the literature such as temperature scaling @xcite and instruction ensembling @xcite . we also utilize demonstration ensembling to leverage more examples than what can fit into the context of transformer llms by combining reranking probabilities computed with different demonstrations. we evaluate few-shot promptrank on hot-potqa @xcite , a standard mqa benchmark, and show that it compares favorably against state-of-the-art models while using orders of magnitude fewer examples. more precisely, with only 128 training examples, promptrank outperforms drkit @xcite and is only 4.2 recall@10 points lower than multi-hop dense retrieval (mdr) @xcite (see figure 1 ). we also showcase promptrank as part of a qa pipeline, again, displaying close qa performance to fully-supervised retrievers-only 4.1 f 1 points lower than mdr. in summary, our contributions in this paper are: ## promptrank an overview of the full retrieval system is displayed in figure 2 : given a question q, the system expands sequences of supporting documents into paths of length h, which are used to answer the question. at each step, we first use tf-idf similarity to ob-tain an initial set of supporting document paths. 3 we then use promptrank to rerank the current document chains based on their relevance to the question ( §2.1). concretely, we start with retrieving f candidate documents using tf-idf for the 'first hop'. these '1-hop' candidates are scored by promptrank and k 1 top-ranked documents are kept and further expanded based on their hyperlinks to obtain 2-hop reasoning paths. 4 these 2-hop reasoning chains are again reranked and the most promising k 2 candidates are further expanded. the process repeats until we obtain paths of length h, where h can be a hyperparameter. 5 as the document graph can have a high branching factor, we only keep the top-l hyperlinks as reranking candidates based on tf-idf similarity between the hyperlink document and the question. we have found this pruning step to improve efficiency without much performance drop. this process is shown in figure 2(a) . ## related work multi-hop retrieval the majority of approaches for multi-hop question answering rely on two main components: a retriever and a reader. the retriever component can be a sparse index or heuristic-based such as tf-idf or bm25 @xcite or dense index @xcite @xcite . other approaches aimed to improve the retriever with an additional reranking step on top of a simple retriever @xcite @xcite . @xcite combined tf-idf retriever with a recurrent graph retriever and used the reader module to rerank paths based on answer confidence. @xcite used a single transformer model to perform retrieval, reranking and reading in an iterative fashion. however, the good performance of previous work comes mainly from training on a large number of examples and is likely to fail in low-data settings. to treat this issue, @xcite proposed to pretrain mdr @xcite on a large number of weakly-supervised examples of questions and the corresponding document paths. although promising in low-data settings, their pretraining is computationally expensive as it is done on millions of examples. on the other hand, our approach requires no task-specific pretraining. language models prompting prompt-based learning aims to construct better inputs, i.e., prompts to language models to elicit better zeroor few-shot performance @xcite . recently, instruction tuning, where a language model is trained to follow natural language instruction, has shown impressive zero-shot performance on unseen tasks @xcite . in our work, we use instructions to guide to model toward assigning better scores to more relevant document paths. lm-based reranking our scoring function is related to query likelihood retrieval @xcite and is in line with previous work that employed generative language models for passage reranking @xcite . dos santos et al. (2020) performed single-hop reranking using question likelihood given the passage, but their setting was limited to fully-supervised, single-hop qa. concurrent with our work is @xcite , where the authors leverage llms for unsupervised passage reranking for qa. while their focus is on single passages, we study the reranking of multi-passage paths, which is more challenging. moreover, their exploration of prompting is limited to a single manual instruction, whereas we provide an in-depth analysis of the effect of different prompting aspects on the recall such as instruction importance, location in the prompt, and manual vs. automated.