Joint Passage Ranking for Diverse Multi-Answer Retrieval (2021)

Abstract: We study multi-answer retrieval, an under-explored problem that requires retrieving passages to cover multiple distinct answers for a given question. This task requires joint modeling of retrieved passages, as models should not repeatedly retrieve passages containing the same answer at the cost of missing a different valid answer. Prior work focusing on single-answer retrieval is limited as it cannot reason about the set of passages jointly. In this paper, we introduce JPR, a joint passage retrieval model focusing on reranking. To model the joint probability of the retrieved passages, JPR makes use of an autoregressive reranker that selects a sequence of passages, equipped with novel training and decoding algorithms. Compared to prior approaches, JPR achieves significantly better answer coverage on three multi-answer datasets. When combined with downstream question answering, the improved retrieval enables larger answer generation models since they need to consider fewer passages, establishing a new state-of-the-art.

## introduction passage retrieval is the problem of retrieving a set of passages relevant to a natural language question from a large text corpus. most prior work focuses on single-answer retrieval, which scores passages independently from each other according to their relevance to the given question, assuming there is a single answer @xcite @xcite . however, questions posed by humans are often open-ended and ambiguous, leading to multiple valid answers @xcite . for example, for the question in figure 1 , "what was eli whitney's job?", an ideal retrieval system should provide passages covering all professions of eli whitney. this introduces the problem of multianswer retrieval-retrieval of multiple passages * work done while interning at google. ## jpr: joint passage retrieval we propose jpr (joint passage retrieval), which models the joint probability p (p 1 ...p k |q) for multi-answer retrieval. jpr uses an approach consisting of first-stage retrieval followed by reranking: the first-stage retrieval obtains candidate passages b from the corpus c, and a reranker processes b to output {p 1 ...p k } ⊂ b. we refer to section 4.2 for the first-stage retrieval, and focus on the reranking component of the model, which allows (1) efficiently modeling the joint probability p (p 1 ...p k |q), and (2) processing candidate passages with a more expressive model. the overview of jpr is illustrated in figure 2 . the reranker of jpr leverages the encoder-decoder architecture for an autoregressive generation of passage references (section 3.1). unlike typical use cases of the encoder-decoder, (1) the ordering of passages to retrieve is not given as supervision, and (2) it is important to balance between exploring passages about new answers and finding passages that may cover previously selected answers. to this end, we introduce a new training method (section 3.2) and a tree-based decoding algorithm (section 3.3). ## experimental setup we compare jpr with multiple baselines in a range of multi-answer qa datasets. we first present an intrinsic evaluation of passage retrieval by reporting mrecall based on answer coverage in the retrieved passages (section 5.1). we then present an extrinsic evaluation through experiments in downstream question answering (section 5.2). ## related work we refer to section 2 for related work focusing on single-answer retrieval. diverse retrieval studies on diverse retrieval in the context of information retrieval (ir) requires finding documents covering many different subtopics to a query topic @xcite . questions are typically underspecified, and many documents (e.g. up to 56 in @xcite ) are considered relevant. in their problem space, effective models post-hoc increase the distances between output passages during inference @xcite . our problem is closely related to diverse retrieval in ir, with two important differences. first, since questions represent more specific information needs, controlling the trade-off between relevance and diversity is harder, and simply increasing the distances between retrieved passages does not help. 7 second, multi-answer retrieval uses a clear notion of "answers"; "sub-topics" in diverse ir are more subjective and hard to enumerate fully. ## conclusion we introduce jpr, an autoregressive passage reranker designed to address the multi-answer retrieval problem. on three multi-answer datasets, jpr significantly outperforms a range of baselines in both retrieval recall and downstream qa accuracy, establishing a new state-of-the-art. future work could extend the scope of the problem to other tasks that exhibit specific information need while requiring diversity.