No Answer is Better Than Wrong Answer: A Reflection Model for Document Level Machine Reading Comprehension (2020)

Abstract: The Natural Questions (NQ) benchmark set brings new challenges to Machine Reading Comprehension: the answers are not only at different levels of granularity (long and short), but also of richer types (including no-answer, yes/no, single-span and multi-span). In this paper, we target at this challenge and handle all answer types systematically. In particular, we propose a novel approach called Reflection Net which leverages a two-step training procedure to identify the no-answer and wrong-answer cases. Extensive experiments are conducted to verify the effectiveness of our approach. At the time of paper writing (May. 20, 2020), our approach achieved the top 1 on both long and short answer leaderboard, with F1 scores of 77.2 and 64.1, respectively.

## introduction deep neural network models, such as @xcite @xcite @xcite @xcite , have greatly advanced the state-of-the-arts of machine reading comprehension (mrc). natural questions (nq) @xcite is a new question answering benchmark released by google, which brings new challenges to the mrc area. one challenge is that the answers are provided at two-level granularity, i.e., long answer (e.g., a paragraph in the document) and short answer (e.g., an entity or entities in a paragraph). therefore, the task requires the models to search for answers at both document level and passage level. moreover, there are richer answer types in the nq task. in addition to indicating textual answer spans (long and short), the several works have been proposed to address the challenge of providing both long and short answers. @xcite adopts a pipeline approach, where a long answer is first identified from the document and then a short answer is extracted from the long answer. although this approach is reasonable, it may lose the inherent correlation between the long and the short answer, since they are modeled separately. several other works propose to model the context of the whole document and jointly train the long and short answers. for example, @xcite split a document into multiple training instances using sliding windows, and leverages the overlapped tokens between windows for context modeling. a mrc model based on bert @xcite is applied to model long and short answer span jointly. while previ- transformer 2 [cls] p4 p3 p2 p1 qn q1 ••• ••• [sep] [cls] p4 p3 p2 p1 qn q1 ••• ••• [sep] ans_type start end [cls] head features hidden confidence transformer 1 [cls] p4 p3 p2 p1 qn q1 ••• ••• [sep] [cls] p4 p3 p2 p1 qn q1 ••• ••• [sep] [cls] p4 p3 p2 p1 qn q1 ••• ••• [sep] ous approaches have proved effective to improve the performance on the nq task, few works focus on the challenge of rich answer types in this qa set. we note that 51% of the questions have no answer in the nq set, therefore, it is critical for the model to accurately predict when to output the answer. for other answer types, such as multi-span short answer or yes/no answer, although they have a small percentage in the nq set, they should not be ignored. instead, a systematic design which can handle all kinds of answer types well would be more preferred in practice. in this paper, we target the challenge of rich answer types, and particularly for no-answer. in particular, we first train an all types handling mrc model. then, we leverage the trained mrc model to inference all the training data, train a second model, called the reflection model takes as inputs the predicted answer, its context and mrc head features to predict a more accurate confidence score which distinguish the right answer from the wrong ones. there are three reasons of applying a secondphase reflection model. firstly, the common practice of mrc confidence computing is based on heuristics of logits, which isn't normalized and isn't very comparable between different questions. @xcite secondly, when training long document mrc model, the negative instances are down sampled by a large magnitude because they are too many compared with positive ones (see section 2.1). but when predicting, mrc model should inference all the instances. this data distribution discrepancy of train and predict result in that mrc model may be puzzled by some negative instance and predict a wrong answer with a high confidence score. thirdly, mrc model learns the representation towards the relation between the question, its type, and the answer which isn't aware of the correctness of the predicted answer. our second-phase model addresses these three issues and is similar to a reflection process which become the source of its name. to the best of our knowledge, this is the first work to model all answer types in nq task. we conducted extensive experiments to verify the effectiveness of our approach. our model achieved top 1 performance on both long and short answer leaderboard of nq challenge at the time of paper writing (may. 20, 2020). the f1 scores of our model were 77.2 and 64.1, respectively, improving over the previous best result by 1.1 and 2.7. ## our approach we propose reflection net (see figure 1 ), which consists of a mrc model for answer prediction and a reflection model for answer confidence. ## experiments we perform the experiments on nq @xcite dataset which consists of 307,373 training examples, 7,830 development examples and 7,842 blind test examples used for leaderboard. the evaluation metrics are separated for long and short answers, each containing precision (p), recall (r), f-measure (f1) and recall at fixed precision (r@@xmath0, r@@xmath1, r@@xmath2. for each question, the system should provide both answer and its confidence score. during evaluation, the official evaluation script will calculate the optimal threshold which maximizes the f1, if answer score is higher than this threshold, the answer is triggered otherwise no-triggered. our dataset preprocessing method is similar to @xcite : firstly, we tokenize the text according to different pretrained models, e.g. wordpiece for bert, bpe § we transform head features by scale to [0, 1], sqrt, log, minus mean then divided by standard deviation. for roberta; then use sliding window approach to slice document into instances as described in section 2.1. for nq, since the document is quite long, we add special atomic markup tokens to indicate which part of the document the model is reading. ## conclusion in this paper, we propose a systematic approach to handle rich answer types in mrc. in particular, we develop a reflection model to address the no-answer/wrong-answer cases. the key idea is to train a second phase model and predict the confidence score of a predicted answer based on its content, context and the state of mrc model. experiments show that our approach achieves the state-of-the-art results on the nq set. measured by f1 and r@@xmath3, and on both long and short answer, our method surpasses the previous top systems with a large margin. ablation studies also confirm the effectiveness of our approach.