Double Retrieval and Ranking for Accurate Question Answering (2023)

Abstract: Recent work has shown that an answer verification step introduced in Transformer-based answer selection models can significantly improve the state of the art in Question Answering. This step is performed by aggregating the embeddings of top k answer candidates to support the verification of a target answer. Although the approach is intuitive and sound, it still shows two limitations: (i) the supporting candidates are ranked only according to the relevancy with the question and not with the answer, and (ii) the support provided by the other answer candidates is suboptimal as these are retrieved independently of the target answer. In this paper, we address both drawbacks by proposing (i) a double reranking model, which, for each target answer, selects the best support; and (ii) a second neural retrieval stage designed to encode question and answer pair as the query, which finds more specific verification information. The results on well-known datasets for Answer Sentence Selection show significant improvement over the state of the art.

## introduction in recent years, automated question answering (qa) research has received a renewed attention thanks to the diffusion of virtual assistants. for example, google home, siri and alexa provide general information inquiry services, while many other systems serve customer requests in different application domains. retrieval-based qa is enabled by two main tasks: (i) answer sentence selection (as2), which, given a question and a set of answer-sentence candidates, consists in selecting sentences (e.g., retrieved by a search engine) that correctly answer the question; and (ii) machine reading (mr), e.g., @xcite , which, given a question and a reference text, finds an exact text span that answers the question. deploying mr q: what causes heart disease? c1: cardiovascular disease (also called heart disease) is a class of diseases that involve the heart or blood vessels (arteries, capillaries, and veins). c2: the causes of cardiovascular disease are diverse but atherosclerosis and/or hypertension are the most common. c3: cardiovascular disease refers to any disease that affects the cardiovascular system , principally cardiac disease, vascular diseases of the brain and kidney , and peripheral arterial disease. more specifically, given a question q, and a target answer, t, to be verified, which is taken from a ranked set of answer candidates (c 1 , .., c k ), asr concatenates transformer-based embeddings of (q, c i ) with the max-pooling vector produced by the top k embeddings of (t, c i ), where the c i are selected by an initial answer reranking model (e.g., tanda). for example, asr reduced the error of tanda by 10% (relative), both on wikiqa and trec-qa datasets. however, asr shows two important limitations: first, when attempting the verification step of t, the k candidates, used in the max-pooling operation, are ranked only based on the question, i.e., independently of t. second, the support for each t is provided by other answer candidates, which again are retrieved independently of t, i.e., t is not part of the query used for searching relevant documents. in this paper, we provide new answer verification models, which are more efficient and accurate than asr. we introduce a new architecture, double answer reranking (dar), which uses two models for reranking target answers and supporting candidates, respectively. given t, the first, support reranker (sr), sorts (q, t, c i ) triplets with respect to i, in order to find the best support for t, i.e., s @xmath1, while the second, answer reranker (ar), orders (q, t, s t ) triplets with respect to t, thus ranking all target answers. additionally, we improve the quality of supports using a second retrieval stage that searches for passages relevant to (q, t). this is important as standard answer candidates provide only information relevant to q, thus they not necessarily provide useful context for assessing t. as formulating an effective query for retrieving a question/answer pair is a new problem, and can be challenging, we exploit deep passage retrieval (dpr) @xcite . this enables us to automatically produce embeddings for (q, t) as the target query of a neural retrieval model. as dar is efficient, it can process many candidates from dpr, making double retrieval (dr) effective. the results derived on three well-known as2 datasets, wikiqa @xcite , trec @xcite , and selqa @xcite and a popular multi-hop qa dataset, hotpotqa @xcite , show consistent and significant improvement over the state of the art. for example, dar improves tanda by 13.6% (relative error reduction), achieving the same accuracy of the computational expensive asr verification approach (84.36%) while dar-dr improves the as2 state of the art, reducing the error by an additional 8%. we will release the datasets augmented with dpr retrieval (support candidates) for each (q, a) of each of the datasets above. ## related work we focus our research on qa systems based on information retrieval. since early versions, e.g., trec qa tracks @xcite , these systems have been based on a search engine, which retrieves documents relevant to the asked questions, followed efficient and accurate passage rerankers to select text that most likely contains the answer. this research was revived introducing the task of answer sentence reranking @xcite . in recent work, the probability, p(q, c i ), for a passage/sentence, c i , to be correct for q is estimated using neural networks, e.g., encoding q and c i text, separately with a cnn @xcite . also designing attention mechanisms, e.g., compare-aggregate @xcite , interweighted alignment networks @xcite . the state of the art is achieved with pre-trained transformers, e.g., @xcite . a number of researchers has proposed more than one candidate for the inference stage, e.g., using pairwise model, i.e., binary classifiers of the form χ(q, c i , c j ), which determine the partial rank between c i and c j , for example, @xcite @xcite use a pairwise loss and encoding. however, these methods have been largely outperformed by the pointwise models based on transformers. bonadiman and moschitti (2020) designed several joint models that improved early neural models for as2 but failed to improve transformer-based models. @xcite used the relation between candidates in multi-task learning approach for as2 but as they did not exploit transformer models, their results are rather lower than the state of the art. very recently, @xcite proposed asr, a model based on a pointwise reranker fed with the embeddings refined by a pairwise approach. this significantly improved the state of the art, there-fore, we analyzed asr and specifically compare our models with it. very different approaches to qa systems than above use mr to extract answers from entire documents. as they have been mainly developed to find answers in a paragraph or in a text of limited size, they are rather inefficient at processing hundreds of documents, while as2 methods can do this with high efficiency. @xcite al. (2017); hu et al. (2019) ; @xcite proposed solutions for reliably performing inference with mr models on multiple documents. still, the efficiency drawback was not solved. finally, multihop qa uses multiple retrieval stages @xcite but the answers are just entities. ## baseline models for as2 a general problem formulation for as2 is the following: given a question q, a subset of its topk ranked answer candidates, and a target answer @xmath2, train a function, f : q × c @xmath3, t, c 1 , .., c k-1 ) provides the probability of t to be correct. in this section, we describe our re-implementation of baselines, and the state-ofthe-art model for as2, namely, asr @xcite . more complex models are built on top of simpler ones, thus providing an ablation study. ## double reranking and retrieval asr is the state of the art for joint modeling candidates. however, it suffers from three main limitations: (i) it needs to limit k otherwise the complexity may be too high, this means that it may not able to process all available supporting candidates, (ii) the top k candidates are the best answer ranked by tanda, which does not guarantee that these are also the best supports, and (iii) answer candidates may be good supports but they were not retrieved for this purpose. we address the above drawbacks proposing: (i) double reranking functions, which can efficiently rank supports as well as the best target answers, and (ii) a second stage of retrieval that performs support retrieval using a representation of the target answer and question pairs. ## experiments we compare our models with several baselines we implemented from previous work, and asr, which is the current state of the art for as2. for the evaluation, we used three different datasets traditionally used for as2. finally, we provide error analysis and model discussion.