Enhancing Multiple-choice Machine Reading Comprehension by Punishing Illogical Interpretations (2021)

Abstract: Machine Reading Comprehension (MRC), which requires a machine to answer questions given the relevant documents, is an important way to test machines’ ability to understand human language. Multiple-choice MRC is one of the most studied tasks in MRC due to the convenience of evaluation and the flexibility of answer format. Post-hoc interpretation aims to explain a trained model and reveal how the model arrives at the prediction. One of the most important interpretation forms is to attribute model decisions to input features. Based on post-hoc interpretation methods, we assess attributions of paragraphs in multiple-choice MRC and improve the model by punishing the illogical attributions. Our method can improve model performance without any external information and model structure change. Furthermore, we also analyze how and why such a self-training method works.

## introduction machine reading comprehension (mrc), which requires a machine to answer questions according to given documents, is an important way to test the ability of intelligence systems to understand human language @xcite . as with other tasks in natural language processing (nlp), deep models have achieved great success on mrc. at the same time, deep models' opaqueness grows in tandem with their power @xcite , which has motivated efforts to interpret how these black-box models work. post-hoc interpretation aims to explain a trained model and reveal how the model arrives at the prediction @xcite , as shown in figure 1 . this goal is usually approached with attribution method, which assesses attributions of inputs to model predictions @xcite @xcite . in nlp, interpretations are usually given by assess-ing attributions of words, phrases, sentences and paragraphs @xcite @xcite @xcite , in which positive attributions mean support to the prediction and negative ones mean opposition. it is well known that the strong fit ability of deep models can cause incredibly high performance on the training set. the correct prediction of an mrc model on the training set can't reflect the model has understood the sample and used a suitable way to predict. since post-hoc interpretations can provide insights into how the model arrives at the prediction, we argue that we can use these insights to explore problems which predictions can't reflect and to improve model performance. in this work, we interpret multiple-choice mrc models by assessing attributions of paragraphs and improve model performance by punishing the illogical parts of these attributions. the illogical attributions here mean the positive ones to the wrong choices and negative ones to the right, reflecting paragraphs' support to the wrong choices and opposition to the right in the model reasoning process. figure 2 shows two specific examples, both of which are from the training set. numbers on the right are model predictions and numbers on the left are attributions of paragraphs. in example1, if we only observe model predictions, the model makes the right choice: b and does not appear to be distracted. however, the attributions show strong support of para3 to distractors a and d, which overlap words in it. these attributions show the model's strong dependency on word-overlap form, which is not suitable for answering this question. we constrain the model from such dependency by punishing these attributions. attributions in this example reflect problems predictions fail to, and we take advantage of this to improve the model. it is worth noting that we don't simply constrain the model from certain forms. on the contrary, we let the model learn differences between different circumstances. for example, in example2, attributions show strong support of para4 to choice c, reflecting the model's dependency on the wordoverlap form. however, we will not constrain the model from such dependency as in example1 because the choice c is the right choice and the attribution here is logical. this way, we let the model learn which circumstance is suitable for using such forms and which one isn't. compared to existing work @xcite @xcite , our method does not need any external information and model structure change. we simply train a new model after getting attributions of the original model. we demonstrate the effectiveness of our method through experiments on three representative datasets: race @xcite , multirc @xcite and dream @xcite . the main attributions of this paper are summarized as follows: ## conclusion and future work in this work, we improve multiple-choice mrc resort to attribution interpretations. experimental results show that our method can remarkably improves model performance on three representative datasets. we believe using post-hoc interpretations to improve nlp models is a promising research field. the future work contains two aspects: