Answer Span Correction in Machine Reading Comprehension (2020)

Abstract: Answer validation in machine reading comprehension (MRC) consists of verifying an extracted answer against an input context and question pair. Previous work has looked at re-assessing the “answerability” of the question given the extracted answer. Here we address a different problem: the tendency of existing MRC systems to produce partially correct answers when presented with answerable questions. We explore the nature of such errors and propose a post-processing correction method that yields statistically significant performance improvements over state-of-the-art MRC systems in both monolingual and multilingual evaluation.

## introduction extractive machine reading comprehension (mrc) has seen unprecedented progress in recent years @xcite @xcite . nevertheless, existing mrc systems-readers, henceforth-extract only partially correct answers in many cases. at the time of this writing, for example, the top systems on leaderboards like squad @xcite , hotpotqa @xcite and quoref @xcite all have a difference of 5-13 points between their exact match (em) and f1 scores, which are measures of full and partial overlap with the ground truth answer(s), respectively. figure 1 shows three examples of such errors that we observed in a state-of-the-art (sota) roberta-large @xcite model on the recently released natural questions (nq) @xcite dataset. in this paper, we investigate the nature of such partial match errors in mrc and also their post hoc correction in context. q: what type of pasta goes in italian wedding soup gt: usually cavatelli, acini di pepe, pastina, orzo, etc. prediction: acini di pepe q: when does precipitate form in a chemical reaction gt: when the reaction occurs in a liquid solution prediction: precipitation is the creation of a solid from a solution. when the reaction occurs in a liquid solution q: what is most likely cause of algal blooms gt: an excess of nutrients, particularly some phosphates prediction: freshwater algal blooms are the result of an excess of nutrients recent work on answer validation @xcite has focused on improving the prediction of the answerability of a question given an already extracted answer. @xcite look for support of the extracted answer in local entailments between the answer sentence and the question. @xcite propose an attention-based model that explicitly checks if the candidate answer satisfies all the conditions in the question. @xcite use a two-stage reading process: a sketchy reader produces a preliminary judgment on answerability and an intensive reader extracts candidate answer spans to verify the answerability. here we address the related problem of improving the answer span, and present a correction model that re-examines the extracted answer in context to suggest corrections. specifically, we mark the extracted answer with special delimiter tokens and show that a corrector with architecture similar to that of the original reader can be trained to produce a new accurate prediction. our main contributions are as follows: 2 partial match in mrc short-answer extractive mrc only extracts short sub-sentence answer spans, but locating the best span can still be hard. for example, the answer may contain complex substructures including multiitem lists or question-specific qualifications and contextualizations of the main answer entity. this section analyzes the distribution of broad categories of errors that neural readers make when they fail to pinpoint the exact ground truth span (gt) despite making a partially correct prediction. to investigate, we evaluate a roberta-large reader (details in section 3) on the nq dev set and identify 587 examples where the predicted span has only a partial match (@xmath0, @xmath1. since most existing mrc readers are trained to produce single spans, we discard examples where the nq annotators provided multi-span answers consisting of multiple non-contiguous subsequences of the context. after discarding such multi-span gt examples, we retain 67% of the 587 originally identified samples. there are three broad categories of partial match errors: in many cases, these predictions lack syntactic structure and semantic coherence as a textual unit. 3. prediction ∩ @xmath2: this final category consists of cases similar to the last example of figure 1 , where the prediction partially overlaps with the gt. (we slightly abuse the set notation for conciseness.) such predictions generally exhibit both verbosity and inadequacy. ## method in this section, we describe our approach to correcting partial-match predictions of the reader. ## conclusion we describe a novel method for answer span correction in machine reading comprehension. the proposed method operates by marking an original, possibly incorrect, answer prediction in context and then making a new prediction using a corrector model. we show that this method corrects the predictions of a state-of-the-art english-language reader in different error categories. in our experiments, the approach also generalizes well to multilingual and cross-lingual mrc in seven languages. future work will explore joint answer span correction and validation of the answerability of the question, re-using the original reader's output representations in the correction model and architectural changes enabling parameter sharing between the reader and the corrector.