A Study on Efficiency, Accuracy and Document Structure for Answer Sentence Selection (2020)

Abstract: An essential task of most Question Answering (QA) systems is to re-rank the set of answer candidates, i.e., Answer Sentence Selection (AS2). These candidates are typically sentences either extracted from one or more documents preserving their natural order or retrieved by a search engine. Most state-of-the-art approaches to the task use huge neural models, such as BERT, or complex attentive architectures. In this paper, we argue that by exploiting the intrinsic structure of the original rank together with an effective word-relatedness encoder, we achieve the highest accuracy among the cost-efficient models, with two orders of magnitude fewer parameters than the current state of the art. Our model takes 9.5 seconds to train on the WikiQA dataset, i.e., very fast in comparison with the 18 minutes required by a standard BERT-base fine-tuning.

## introduction in recent years, there has been a renewed interest in question answering (qa) led by industrial needs, e.g., the development of personal assistants and academic research on neural networks. regarding the latter, as2 @xcite and machine reading comprehension (mrc) @xcite have been largely explored. as2 consists of selecting sentences that are answers to a target question from documents or paragraphs retrieved from the web by a search engine. mrc regards the extraction of an exact text span from a document answering the question, where the document is usually provided with the target question. even though mrc is gaining more and more popularity, as2 is more suitable for a production scenario, where a combination of a retrieval engine and an automatic sentence selector can already constitute a qa system. in contrast, mrc has been mainly developed to find answers in a paragraph or a text of limited size. several models for adapting mrc to an end-to-end retrieval setting have been proposed, e.g., @xcite and @xcite , the deployment of mrc systems in production is challenged by two key factors: the lack of datasets for training mrc with realistic retrieval data, and the large volume of content needed to be processed, i.e., mrc cannot efficiently process a large amount of retrieved data. in contrast, as2 research originated from the trec competitions @xcite ; thus, it has targeted large databases of unstructured text from the beginning of its development. neural models have significantly contributed to as2 with new techniques, e.g., @xcite , @xcite and @xcite . recently, new approaches for pre-training neural language models on large amount of data, e.g., elmo @xcite , gpt @xcite , bert @xcite , roberta @xcite , have led to major advancements in several nlp subfields. these pre-training techniques allow for creating models that automatically capture dependencies between the sentence compounds. interestingly, the resulting models can be easily adapted to different tasks by fine-tuning them on the target training data. unfortunately, all the models above (especially transformer-based architectures) require a considerable number of parameters (up to 340 million for bert large). this poses three critical challenges for having how long was i love lucy on the air ? ## as2: answer sentence selection the task of answer sentence selection (as2) can be formalized as follows: given a question q and a set of answer sentence candidates @xmath0, c 2 , ..., c n }, assign a score s i for each candidate c i such that the sentence receiving the highest score is the one that most likely contains the answer, i.e., the answer sentence. it is interesting to note that as2 can be, in fact, modeled as a re-ranking task. although re-ranking is a structured output problem, most state-of-the-art approaches treat it as pointwise classification, i.e., classifying answer sentences as positive and all the others as negative. this design bias prevents as2 models from capturing the underlying structure of the original rank. however, in this paper, we argue that building systems capable of capturing such information is crucial for improving the performance of efficient as2 models. ## related work despite the importance of global structure, most state-of-the-art models @xcite @xcite @xcite do not take the global structure of candidates into account. they use a point-wise approach to maximize the score of positive candidates, i.e., those containing the answer, and minimize the score of those not containing the answer. most models treat ranking as a binary classification problem. nevertheless, other methods have been studied, e.g., contrastive pair-wise and, more recently, list-wise approaches. in the case of contrastive pair-wise training @xcite , given a question q, the loss maximizes the score of the model for a positive question candidate pair (q, cˆ+) in contrast to the score of the negative pair (q, cˆ-). this approach intrinsically balances the distribution of positive and negative examples in training and can improve the overall results. in particular, when paired with a hard negative sampling strategy. however, the comparison between the answer candidates is just performed at the score level; therefore, the model is still point-wise in terms of internal representation. the list-wise approach was proposed in a recent paper by @xcite . the approach predicts the score for each question candidate pair individually, but it applies a softmax function on top of the set of scores, s 1 , s 2 , ..., s @xmath1, c 1 ), φ(q, c 2 ), ..., φ(q, c n )). this helps in providing stability in the training process. however, the model is agnostic of the global structure of the rank. most as2 models have a way to identify the lexical overlap and, in particular, the semantic word-overlap between question and answer. the first neural models developed to solve the task @xcite directly added the lexical overlap feature in the model by concatenating it to the question-candidate representation or as a feature concatenated to the word embeddings of convolutional neural network (cnn). subsequent approaches, e.g., @xcite , use a word-level attention mechanism to identify the semantic overlap between each word in the question and each word in the answer candidate. despite obtaining better results than previous approaches, the computational cost of performing word-level attention and the aggregation steps to leverage the information extracted by the attention mechanism increases the computational cost of previous methods. more recent models, e.g., @xcite @xcite , leverage contextualized word representation, e.g., pre-trained using bert, elmo, roberta, etc. these approaches achieve state-of-the-art results for as2, but they require significant computational power for both pre-training, fine-tuning, and testing on the final task. ## efficient model for as2 to build an efficient yet accurate model for as2, it is crucial to leverage all the strong signals present in the dataset without increasing the complexity of the model itself. for this reason, we design a model as follows: (i) we build an efficient encoder to capture the question-candidate pair's lexical-overlap, i.e., our cosinet. (ii) we add a recursive neural network on top of the question-candidate pairs to capture the original rank's global structure. (iii) we apply a global, list-wise, optimization approach to rank all the candidate pairs jointly. ## experiments in these experiments, we first report the results of simple baselines and the state of the art for efficient and expensive models. then, we present the results of our models carrying our comparison with the previous results. ## conclusions in this paper, we argue that by exploiting the intrinsic structure of the original rank and an effective word-relatedness encoder, we can achieve competitive results in comparison with state of the art, while retaining high efficiency. we first analyzed the structure of standard datasets, highlighting the importance of the original rank's global structure. capitalizing on this, we propose a model that both exploits (i) the rank structure using a simple rnn and (ii) the standard word-relatedness features while preserving high efficiency. the model uses around 1m parameters depending on the configuration and achieves better results than the previous work that does not make use of computationally expensive pre-trained language models. our model takes 9.5 seconds to train on the wikiqa dataset, i.e., very fast compared to the ∼ 18 minutes required by a standard bert-base fine-tuning. on squad, this difference is even higher, i.e., minutes vs. hours required by transformer-based models. it should be stressed the fact that our very fast and accurate models are essential for at least two real-world application scenarios: (i) when the efficiency and computation cost does not allow for using pre-trained transformer models; and (ii) in case of sequential re-ranking, our model can be positioned earlier in the re-ranking pipeline. recent work has also been devoted to distill cheaper versions of transformer models @xcite . these inevitably lose accuracy with respect to the original larger models. our joint models can be potentially applied to them to limit their accuracy drop. additionally, our cosinet is still competitive in terms of speed with any distilled models, as none of them can ever have such a low number of parameters. this means that cosinet can always be valuable in the first position of a sequential reranker, possibly followed by distilled versions of transformer models.