MRC Examples Answerable by BERT without a Question Are Less Effective in MRC Model Training (2020)

Abstract: Models developed for Machine Reading Comprehension (MRC) are asked to predict an answer from a question and its related context. However, there exist cases that can be correctly answered by an MRC model using BERT, where only the context is provided without including the question. In this paper, these types of examples are referred to as “easy to answer”, while others are as “hard to answer”, i.e., unanswerable by an MRC model using BERT without being provided the question. Based on classifying examples as answerable or unanswerable by BERT without the given question, we propose a method based on BERT that splits the training examples from the MRC dataset SQuAD1.1 into those that are “easy to answer” or “hard to answer”. Experimental evaluation from a comparison of two models, one trained only with “easy to answer” examples and the other with “hard to answer” examples demonstrates that the latter outperforms the former.

## introduction the machine reading comprehension (mrc) task locates the best corresponding natural language answer when provided a question and its related context. in recent years, mrc models using neural networks have been proposed for squad @xcite , which is a large-scale, high-quality english mrc dataset. most recent neural network based mrc models have outperformed human performance @xcite . among those existing work, to analyze the difficulty of several popular mrc benchmarks such as babi @xcite , squad @xcite , cbt @xcite , cnn @xcite and who-did-what @xcite , @xcite established sensible baselines for these datasets, and found that question-only and context-only (which is called passage-only in @xcite ) models often performs surprisingly well. in particular, contextonly models achieve over 50% accuracy on 14 out of 20 babi tasks, and as for cbt, only the last one of the 20 sentences provided as a context is necessary to achieve a comparable accuracy. they also indicated that squad is designed more carefully than other datasets and achieved f1 scores of only 4% and 14.8% respectively on questiononly and context-only models, which are relatively lower. @xcite demonstrated that published mrc datasets should characterize the level of difficulty, and specifically, the extent to which questions and contexts are essential. moreover, @xcite also claimed that follow-up papers reporting improvements ought to report performance both on the full task and variations omitting questions and contexts. in view of the point demonstrated in @xcite , we concentrate more on the difficulty of every single mrc example, and aim to split the examples into easy ones and hard ones. given the mrc dataset squad1.1 (where each mrc example denoted as the tuple ⟨q, c, a⟩ of the question q, the context c, and the answer a) and the fine-tuned mrc model using bert @xcite , there exist contextonly examples that can be correctly answered, where only the context is provided without including the question. by focusing on this fact, this paper proposes a method that splits the mrc examples into binary classes of "easy to answer" or "hard to answer". a 10-fold cross-validation was applied on approximately 87,600 squad1.1 training examples comprised of 12,500 "easy to answer" and 75,000 "hard to answer" classes. from the comparison of the two classes, the followings are two significant findings. (1) based on the performance of training the bert mrc models with 12,500 "easy to answer" and "hard to answer" examples each, the model trained with "hard to answer" examples outperformed the other. (2) an analysis of the position distribution of answers a within the context c, answers from the "easy to answer" mrc example class tend to be located around the beginning of the context compared with those from the "hard to answer" mrc example class. ## machine reading comprehension using neural networks figure 1 shows the framework of mrc models that use neural networks. in the mrc model, when a question and context are input, the starting and ending positions of the answer with respect to the question within the context are predicted. let ts be a set of test examples with each example denoted as s(∈ ts). here, s is represented as @xmath0, c, a⟩. also, if a set of examples for training mrc models is denoted as tr, then the corresponding model is represented as m(tr). then, the answer a predicted from an input test example s with the trained mrc model m(tr) is denoted as @xmath1, s a boolean predicate answerable classifies if the given test example s is "answerable" or "unanswerable" by the trained mrc model m(tr), and is defined according to if the predicted answer a is the same as the reference answer a as answerable m(tr), @xmath2, the mrc models are trained and tested with 87,600 training examples from the mrc dataset squad1.1. the fine-tuning module for machine reading comprehension foot_0 was applied to the pre-trained multilingual cased model 2 of bert. 3 4 mrc examples answerable without a question first, we evaluate the bert mrc model trained with the 87,600 training examples against 10,570 development examples from of squad1.1. as shown in figure 2, we compare the two cases of with or without the question for each of these development examples. let s denote one of the 10,570 development examples represented as @xmath3, c, a⟩, as before, which are each evaluated with the bert mrc model trained with the 87,600 examples. from these results, 82% are correctly answered and classified as "answerable" with the remaining 18% incorrectly answered and classified as "unanswerable." next, let s ′ denote an mrc example obtained by replacing the question q from each example s of the development examples with an empty question q ′ = ∅ represented as s ′ = ⟨q ′ = ∅, c, a⟩. after evaluating the trained bert mrc model with each example s ′ , as shown on the right of figure 2, 6.1% are training set questions used tr i questions not used tr i @xmath4, 668 |a 2 | = 11, 498 |ua 1 | = 26, 931 |ua 2 | = 76, 101 questions not used s @xmath5, 682 |a 4 | = 10, 008 |ua 3 | = 83, 917 |ua 4 | = 77, 591 following the procedure for detecting mrc examples as answerable without a question demonstraed in the previous section, we similarly split the 87,600 squad1.1 training examples into "easy to answer" and "hard to answer" classes. as illustrated in figure 3 , the process designates 10% of the examples as "easy to answer" and "hard to answer" classes for testing through one fold of 10fold cross-validation, which is repeated ten times, resulting in 12,500 "easy to answer" and 75,100 "hard to answer" classes. from this, we obtain the following three types of evaluation results. detailed procedure the squad1.1 dataset is composed of approximately 100,000 mrc examples that use 23,215 paragraphs extracted from 536 wikipedia articles as context. with these contexts, questions and answers are annotated through crowdsourcing to generate the complete 100,000 mrc example set. from these examples, we apply n -fold crossvalidation (@xmath6. before the n -fold cross-validation, we first divide the 442 wikipedia articles into disjoint n subsets. from the i-th (@xmath7, . . . , n ) subset of wikipedia articles, we obtain the i-th test set ts i of the mrc examples, and the i-th training set of the mrc examples is obtained as the set tr i of the remaining mrc examples. then, the set u of the complete squad1.1 training example set is represented as @xmath8,...,n as shown in figure 6 compares the performance of the five mrc training examples, and figure 7 presents the learning curves with the training examples of the sets e and h sml of also, although we omit the detailed evaluation results, in addition to bert, we also applied span-bert @xcite 7 (base & cased) and xlnet @xcite foot_6 (xlnet-large, cased) and obtained the similar results regarding both of (1) the model trained with "hard to answer" examples outperformed that trained with "easy to answer" ones, and (2) answers from the "easy to answer" mrc example class tend to be located around the beginning of the context compared with those from the "hard to answer" mrc example class. in the task of recognizing textual entailment that classifies the relation between a pair of two sentence as a premise and hypothesis, tsuchiya (2018) compared two of the "recognizing tex-tual entailment" datasets, sick @xcite and snli @xcite . tsuchiya reported that the cases of snli had the correct textual entailment labels predicted when only the hypothesis sentence was provided and without the premise sentence. however, tsuchiya (2018) also pointed out that, a hidden bias in the snli corpus caused much of the high accuracy achieved by the neural network based models that were trained with snli. developing machine reading comprehension datasets requires an expensive and timeconsuming effort to manually create questions from paragraphs and extract spans of text from each paragraph to represent the answer to each question. the approach of active learning, in which the key idea is that a machine learning algorithm can achieve greater accuracy with fewer training labels if it is allowed to choose the data from which it learns @xcite , could be applied to reduce the cost of developing mrc datasets. while there exists no previous study that applies the active learning technique for machine reading comprehension task, other work applied the technique to reduce the cost of developing datasets for other nlp tasks @xcite , image classification @xcite , as well as other machine learning tasks, such as predicting molecular energetics in the field of chemistry @xcite . ## conclusion we proposed a method based on bert @xcite that splits the training examples from the mrc dataset squad1.1 into classes of "easy to answer" and "hard to answer." experimental evaluations of comparing the two models, one of which is trained only with the "easy to answer" examples and the other with the "hard to answer" examples, demonstrate that the latter outperformed the former. future work includes applying the analysis procedure of this paper to several popular mrc benchmark datasets other than squad @xcite and investigating whether the similar results are obtained. we also work on deeper analysis of the characteristics of "easy to answer" / "hard to answer" examples to find out features that are related to the disparity of training effectiveness.