Joint Models for Answer Verification in Question Answering Systems (2021)

Abstract: This paper studies joint models for selecting correct answer sentences among the top k provided by answer sentence selection (AS2) modules, which are core components of retrieval-based Question Answering (QA) systems. Our work shows that a critical step to effectively exploiting an answer set regards modeling the interrelated information between pair of answers. For this purpose, we build a three-way multi-classifier, which decides if an answer supports, refutes, or is neutral with respect to another one. More specifically, our neural architecture integrates a state-of-the-art AS2 module with the multi-classifier, and a joint layer connecting all components. We tested our models on WikiQA, TREC-QA, and a real-world dataset. The results show that our models obtain the new state of the art in AS2.

## introduction automated question answering (qa) research has received a renewed attention thanks to the diffusion of virtual assistants. among the different types of methods to implement qa systems, we focus on answer sentence selection (as2) research, originated from trec-qa track @xcite , as it proposes efficient models that are more suitable for a production setting, e.g., they are more efficient than those developed in machine reading (mr) work @xcite . @xcite proposed the tanda approach based on pre-trained transformer models, obtaining impressive improvement over the state of the art for as2, measured on the two most used datasets, wikiqa @xcite and trec-qa @xcite . however, tanda was applied only to pointwise rerankers (pr), e.g., simple binary classifiers. bonadiman and moschitti a more structured approach to building joint models over sentences can instead be observed in fact verification systems, e.g., the methods developed in the fever challenge @xcite . such systems take a claim, e.g., joe walsh was inducted in 2001, as input (see tab. 1), and verify if it is valid, using related sentences called evidences (typically retrieved by a search engine). for example, ev 1 , as a member of the eagles, walsh was inducted into the rock and roll hall of fame in 1998, and into the vocal group hall of fame in 2001, and ev 3 , walsh was awarded with the vocal group hall of fame in 2001, support the veracity of the claim. in contrast, ev 2 is neutral as it describes who joe walsh is but does not contribute to establish the induction. we conjecture that supporting evidence for answer correctness in as2 task can be modeled with a similar rationale. in this paper, we design joint models for as2 based on the assumption that, given q and a target answer candidate t, the other answer candidates, (c 1 , ..c k ) can provide positive, negative, or neutral support to decide the correctness of t. our first approach exploits fact checking research: we adapted a state-of-the-art fever system, kgat @xcite , for as2. we defined a claim as a pair constituted of the question and one target answer, while considering all the other answers as evidences. we re-trained and rebuilt all its embeddings for the as2 task. our second method, answer support-based reranker (asr), is completely new, it is based on the representation of the pair, (q, t), generated by state-of-the-art as2 models, concatenated with the representation of all the pairs (t, c i ). the latter summarizes the contribution of each c i to t using a maxpooling operation. c i can be unrelated to (q, t) since the candidates are automatically retrieved, thus it may introduce just noise. to mitigate this problem, we use an answer support classifier (asc) to learn the relatedness between t and c i by classifying their embedding, which we obtain by applying a transformer network to their concatenated text. asc tunes the (t, c i ) embedding parameters according to the evidence that c i provides to t. our answer support-based reranker (asr) significantly improves the state of the art, and is also simpler than our approach based on kgat. our third method is an extension of asr. it should be noted that, although asr exploits the information from the k candidates, it still produces a score for a target t without knowing the scores produced for the other target answers. thus, we jointly model the representation obtained for each target in a multi-asr (masr) architecture, which can then carry out a complete global reasoning over all target answers. we experimented with our models over three datasets, wikiqa, trec-qa and wqa, where the latter is an internal dataset built on anonymized customer questions. the results show that: finally, it is interesting to mention that masr improvement is also due to the use of fever data for pre-fine-tuning asc, suggesting that the fact verification inference and the answer support inference are similar. ## problem definition and related work we consider retrieval-based qa systems, which are mainly constituted by (i) a search engine, retrieving documents related to the questions; and (ii) an as2 model, which reranks passages/sentences extracted from the documents. the top sentence is typically used as final answer for the users. ## baseline models for as2 in this section, we describe our baseline models, which are constituted by pointwise, pairwise, and listwise strategies. ## joint answer support models for as2 we proposed the answer support reranker (asr), which uses an answer pair classifier to provide evidence to a target answer t. given a question q, and a subset of its top-k+1 ranked answer candidates, a (reranked by an as2 model), we build a function, σ : q × c × c @xmath0, t, a \ {t}) provides the probability of t to be correct, where c is the set of sentence-candidates. we also design a multi-classifier masr, which combines k asr models, one for each different target answer. ## experiments in these experiments, we compare our models: kgat, asr and masr with pointwise models, which are the state of the art for as2. we also compare them with our joint model baselines (pairwise and listwise). finally, we provide an error analysis. ## conclusion we have proposed new joint models for as2. asr encodes the relation between the target answer and all the other candidates, using an additional transformer model, and an answer support classifier, while masr jointly models the asr representations for all target answers. we extensively tested kgat, asr, masr, and other joint model baselines we designed. the results show that our models can outperform the state of the art. most interestingly, asr constantly outperforms all the models (but masr-fp), on all datasets, through all measures, and for both base and large transformers. for example, asr q: what kind of colors are in the rainbow? c1: red, yellow, and blue are called the primary colors. c2: the order of the colors in the rainbow goes: red, orange, yellow, green, blue, indigo and violet. c3: the colors in all rainbows are present in the same order: red, orange, yellow, green, blue, indigo, and violet. c4: a rainbow occurs when white light bends and separates into red, orange, yellow, green blue, indigo and violet.