Modeling Context in Answer Sentence Selection Systems on a Latency Budget (2021)

Abstract: Answer Sentence Selection (AS2) is an efficient approach for the design of open-domain Question Answering (QA) systems. In order to achieve low latency, traditional AS2 models score question-answer pairs individually, ignoring any information from the document each potential answer was extracted from. In contrast, more computationally expensive models designed for machine reading comprehension tasks typically receive one or more passages as input, which often results in better accuracy. In this work, we present an approach to efficiently incorporate contextual information in AS2 models. For each answer candidate, we first use unsupervised similarity techniques to extract relevant sentences from its source document, which we then feed into an efficient transformer architecture fine-tuned for AS2. Our best approach, which leverages a multi-way attention architecture to efficiently encode context, improves 6% to 11% over non-contextual state of the art in AS2 with minimal impact on system latency. All experiments in this work were conducted in English.

## introduction as2 models for open-domain qa typically consider sentences from webpages as independent candidate answers for a given question. for any webpage containing potential answer candidates for a question, as2 models first extract individual sentences, then independently estimate their likelihood of being correct answers; this approach enable highly efficient processing of entire documents. however, under this framework, context information from the entire webpage (global context), which could be crucial for selecting correct answers, is ignored. conversely, current systems in machine reading (mr) @xcite figure 1 : candidate sentence (in orange) for question "how many digits are in pi?". local context is shown in green, while global context is shown in red. @xcite @xcite uses a much larger context from the retrieved documents. mr models receive a question and one or more passages retrieved through a search engine as input; they then select one or more spans from the input passages to return as answer. while being potentially more accurate, mr models typically have higher computational requirements (and thus higher latency) than as2 models. that is because mr models need to process passages in their entirety before an answer can be extracted; conversely, as2 systems break down paragraphs in candidate sentences, and evaluate them all at once in parallel. therefore, in many practical applications, mr models are only used to examine 10 to 50 candidate passages; in contrast, as2 approaches can potentially process hundreds of documents, e.g., @xcite . in this work, we study techniques that can combine the efficacy of mr models with the efficiency of as2 approaches, while keeping a single sentence as target answer, as in related as2 works foot_0 . in particular, we focus our efforts on improving accuracy of as2 systems without affecting their latency. early neural models for retrieval-based qa focused on incorporated neighboring sentences (local context) to improve performance. for example, @xcite proposed a neural architecture based on gated recurrent units to encode question, answer, and local context; their approach, while effective at the time, shows a significant gap to the current state-of-the-art models @xcite . @xcite studied neural efficient models for mr by optimizing answer candidate extraction. more recently, researchers have focused in including source document information in transformer models. for example, @xcite proposed a contextualized model for mr that augments named entities in candidate passages with snippets extracted from wikipedia pages. their approach, while interesting, is limited to entitiesbased context, and specific to wikipedia and mr domain. for as2, @xcite proposed a model that uses local context as defined by the preceding and following sentences of the target answer. they also introduced a simple bagof-words representation of documents as global context, which did not show significant improvement over non-contextual as2 models. unlike previous efforts, our approaches consider both local context (that is, the sentences immediately preceding or succeeding a candidate answer), as well as global context (phrases in documents that represent the overall topics of a page), as they can both uniquely contribute to the process of selecting the right answer. as shown in the example in figure 1 , local context can help disambiguate cases where crucial entities are not present in the candidate answer (there's no mention of "pi" in "[c]urrently, there are more than 22.4 trillion known digits"); conversely, global context can help reaffirm the relevance of a candidate answer in cases where noisy information is extracted as local context (in the example, "[f]urther reading: pi and pie" does not contain any relevant information). the contributions of this work are: (i) first, we introduce two effective techniques to extract relevant local and global contexts for a given question and candidate answer; (ii) then, we propose three different methods for combining contextual information for as2 tasks; (iii) finally, we evaluate our approaches on two as2 datasets: asnq @xcite and a benchmark dataset we built to evaluate real-world qa systems. results show that our most efficient system, which lever-question: "where did the potter's wheel first develop" corrent answer: "tournettes, in use around 4500 bc in the near east, were turned slowly by hand or by foot while coiling a pot" sentence selected by n-grams: "in the iron age, the potter 's wheel in common use had a turning platform about one metre (3 feet) above the floor, connected by a long axle to a heavy flywheel at ground level. use of the potter's wheel became widespread throughout the old world but was unknown in the pre-columbian new world, where pottery was handmade by methods that included coiling and beating." question: "where do pineapples come from in the world" correct answer: "in 2016, costa rica, brazil, and the philippines accounted for nearly one-third of the world's production of pineapple." sentence selected by cosine similarity: "the plant is indigenous to south america and is said to originate from the area between southern brazil and paraguay" ## methodology our approach to ranking candidate answer consists of two components: the first (section 2.1) is responsible for extracting context for each candidate answer, while the second (section 2.2) encodes information from local and global contexts to score each question / candidate answer pair. ## conclusions for efficiency reasons, traditional as2 models are designed to estimate answer relevancy by only comparing question and candidates. in this work, we described and evaluate several techniques to incorporate local and global contexts in the answer selection process. the results of our experiments show that our proposed methods significantly outperform non-contextual approaches; further, we empirically demonstrate that local and global contexts can be effectively combined to further improve ranking performance.