Diverse Multi-Answer Retrieval with Determinantal Point Processes (2022)

Abstract: Often questions provided to open-domain question answering systems are ambiguous. Traditional QA systems that provide a single answer are incapable of answering ambiguous questions since the question may be interpreted in several ways and may have multiple distinct answers. In this paper, we address multi-answer retrieval which entails retrieving passages that can capture majority of the diverse answers to the question. We propose a re-ranking based approach using Determinantal point processes utilizing BERT as kernels. Our method jointly considers query-passage relevance and passage-passage correlation to retrieve passages that are both query-relevant and diverse. Results demonstrate that our re-ranking technique outperforms state-of-the-art method on the AmbigQA dataset.

## introduction the objective of open-domain question answering is to provide answers to queries utilising a large collection of documents from the world wide web, wikipedia etc. more than 50% of questions present in a widely used open-domain qa dataset (natural questions @xcite ) comprise of ambiguous questions @xcite ). questions that are ambiguous may be interpreted in a number of ways and as a result, they need various answers. in this paper, we concentrate on questions with multiple distinct answers. open domain question-answering systems are designed to generate answers from several data sources. since similar information can be present across multiple data sources, it introduces a significant amount of redundancy. traditional opendomain qa @xcite ) systems comprise of a retriever, which retrieves passages relevant to the question. a passage retriever is primarily concerned with retrieving passages that are relevant * equal contribution to the query, and it does not address redundancy in the passages during retrieval. to be able to produce diverse answers to the question, the passages retrieved must be both relevant to the question and distinct from one another. after the retrieval stage, we introduce a novel re-ranking approach to handle redundant passages. as a result, the re-ranked passages would capture most of the diverse answers to the question. in this paper, we investigate the multi-answer retrieval task, which entails retrieving passages that can cover the distinct answers. re-ranking methods have been employed previously to improve the question answering accuracy significantly @xcite ; @xcite ; @xcite ; @xcite ). @xcite tackles diverse multianswer retrieval by proposing a re-ranker based on an auto-regressive framework in which each passage selected is dependent on the passages chosen at a previous time step. determinantal point processes (dpp) @xcite ) are probabilistic models that are effective at identifying diverse subsets of elements from a collection while preserving quality. dpp methods have proven effective in natural language processing tasks where there is a need for diverseness. @xcite al. (2019), li et al. (2019), and cho et al. (2020), sharghi et al. (2017) have used dpps to perform summarization by choosing salient but also diverse items to be included in the summaries. in this paper, we propose an unsupervised re-ranking technique for multi-answer retrieval utilising determinantal point processes and bert to model the kernels. our contributions can be summarized as follows: 1) we propose a re-ranking method based on determinantal point processes that focuses on diverse passage retrieval. 2) since our approach is unsupervised, our method does not require a large amount of data unlike prior re-ranking methods @xcite . instead, we rely on dpp to identify the most relevant passages to the question that are distinct from one another. 3) we demonstrate that our technique outperforms the state-of-the-art method on the ambigqa dataset using mrecall @ k metrics. ## related work many open domain question answering systems @xcite al. (2017); yang et al. (2019); izacard and grave (2021); guu et al. (2020); lee et al. (2019) ) adopt the retriever-reader method by retrieving the relevant documents and later applying neural techniques to predict the answer. the retriever-reader method was first proposed by @xcite . drqa( chen et al. (2017) ) uses wikipedia as knowledge source and employs a sparse retrieval method using tf-idf and a recurrent neural network to identify the answer spans. while @xcite adopts anserini retriever @xcite ) using bm25 as the ranking function and bert model @xcite ) as the reader. sparse retrieval based methods, such as tf-idf and bm25, face challenges when retrieving relevant passages that do not match the question's exact terms. dense retrieval-based approaches, on the other hand, overcome this problem by mapping each word into a vector space in which words with similar meanings tend to be closer together. orqa @xcite ) and dpr @xcite ) employ a question and passage encoder based on bert and compute a relevance score. using this relevance score, the retriever retrieves the most relevant documents from the corpus. ## determinantal point processes for re-ranking re-ranker acts as a filter to pick a limited number of passages that can be used as input to generate answers to the questions. we formulate the task of passage re-ranking as a subset selection problem. our objective is to choose a subset of passages (y ) of size k from the ground set y comprising n passages that covers all of the answers to a given question q . dpp models a distribution on all the subsets of the ground set y jointly considering the quality and diversity. a subset y is drawn according to the probability distribution p . dpp focuses on two measures -quality and similarity ( fig 1 ). quality score q(i, q) measures how salient the passage i is and whether it contains an answer to the question q. similarity score s(i, j) is computed between two passages i and j to incorporate diversity in the passages. dpp assigns a probability to a set y proportional to the determinant of l-ensemble which may be interpreted geometrically as the volume of the parallepiped covered by the quality and similarity measures @xcite ). a diverse passage subset occupies more volume than a subset of similar passages, therefore dpp assigns higher probability to diverse and relevant passages rather than the most relevant and similar passages. if passages are relevant and diverse, then the passages can cover multiple distinct answers to the question. ## discussion impact on qa system's performance: an open domain question answering system's pipeline consists of three stages. 1) retrieval 2) re-ranking followed by 3) answer extraction. improvements in any of these stages significantly improve the overall system's ability to answer a question. nogueira and cho (2019), min et al. (2021) have shown that the use of a re-ranker has led to end-to-end qa improvements. based on the results presented in ## conclusion in this paper, we propose a dpp-based approach to improve the diverseness of the retrieved passages. we compare our method to the state-ofthe-art method and outperform it by 3% (top 5), 8% (top 10) for single-answer questions, and 18% (top5) and 21% (top10) for multi-answer retrieval on ambigqa dataset.