Multi-Strategy Knowledge Distillation Based Teacher-Student Framework for Machine Reading Comprehension (2021)

Abstract: The irrelevant information in documents poses a great challenge for machine reading compre-hension (MRC). To deal with such a challenge current MRC models generally fall into twoseparate parts: evidence extraction and answer prediction where the former extracts the key evi-dence corresponding to the question and the latter predicts the answer based on those sentences. However such pipeline paradigms tend to accumulate errors i.e. extracting the incorrect evi-dence results in predicting the wrong answer. In order to address this problem we propose aMulti-Strategy Knowledge Distillation based Teacher-Student framework (MSKDTS) for ma-chine reading comprehension. In our approach we first take evidence and document respec-tively as the input reference information to build a teacher model and a student model. Then the multi-strategy knowledge distillation method transfers the knowledge from the teacher model to the student model at both feature and prediction level through knowledge distillation approach. Therefore in the testing phase the enhanced student model can predict answer similar to the teacher model without being aware of which sentence is the corresponding evidence in the docu-ment. Experimental results on the ReCO dataset demonstrate the effectiveness of our approachand further ablation studies prove the effectiveness of both knowledge distillation strategies.

## introduction machine reading comprehension (mrc) is a task that enables machines to read and understand natural language documents to answer questions. since it well indicates the ability of machines in interpreting natural language as well as having a wide range of application scenarios, it has attracted extensive attention from academia and industry over the recent years. prevailing mrc datasets define their tasks as either extracting spans from reference documents to answer questions, such as squad @xcite and coqa @xcite , or inferring answers based on pieces of evidence from a given document, which is also referred to as non-extractive mrc, including multiple-choice mrc @xcite , open domain question answering @xcite and so on. current mrc faces the significant challenge of the irrelevant information in documents causing negative impact on answer predicting. therefore, our aim is to engage the model to focus on evidence sentences in documents and using them to answer corresponding questions accurately. to illustrate, consider the example shown in figure 1 (adapted from the reco dataset @xcite ). in this figure 1 : example of multiple-choice machine reading comprehension. the sentence in green is the evidence sentence for answering the given question in this document, which is of great importance. other sentences contain irrelevant information, while potentially negatively affecting the answer prediction. the sentence in blue is the evidence obtained by manual annotation (summarized or paraphrased by the annotator). previous attempts mainly focused on the pipeline (coarse-to-fine) paradigm @xcite : first locating or generating the evidence sentences corresponding to the question by an evidence extractor or generator, then the answer is predicted based on it. unfortunately, such a pipeline paradigm suffers from the problem of error accumulation. besides, in real-world scenarios, the evidence supporting the answer to the question is often implicitly present in the document and thus not easily extracted or generated. for instance, 46% of the evidence sentences could not be explicitly found in the documents in the reco dataset. once the evidence extractor or generator gets incorrect evidence, the result obtained by the answer predictor is bound to be wrong. in this paper, we attempt to engage the model to focus more on the evidence sentences in the document rather than extracting them out. thus we propose a multi-strategy knowledge distillation based teacher-student framework (mskdts). in the training phase, we first take evidence and document as the reference information to pretrain a teacher model and a student model, respectively. then, we incorporate multi-strategy knowledge distillation into the teacher-student framework, which is the student model attempts to produce teacher-like features and predicted answers through feature knowledge distillation and prediction knowledge distillation. subsequently, in the testing phase, the enhanced student model predicts the answer with only the document (unaware of the evidence sentences). hence, the whole process obviates the process of explicitly evidence extraction, which naturally circumvents the accumulation of errors in the conventional pipeline paradigm. our contributions are summarized as follows: ## methods figure 2 demonstrates the overall framework of mskdts, which aims to cope with the irrelevant information in documents. mskdts is composed of three major parts, namely, the teacher model, the student model and the knowledge distillation strategies. documents and evidence sentences are concatenated with queries and candidate answers respectively as the input to the teacher model and the student c c l 2 0 2 1 candidate answers: 可以/不可以/无法确定 query: 孕妇可以喝红糖水吗？ evidence: 孕妇喝红糖水可能会导致胎儿畸形， 有概率得妊娠糖尿病从而发展成为糖尿病。 candidate answers: 可以/不可以/无法确定 query: 孕妇可以喝红糖水吗？ document: ……这样可以补充营养和身体的所 需，那么孕妇能可以红糖水吗？红糖含很多的 蔗糖，太多的糖可能会导致高血糖，吃点红 枣，炖点乌鸡枸杞汤也是可以的，一般在孕期 多喝些白开水就好。孕妇喝红糖水可能导致妊 娠糖尿病加重，胎儿畸形…… e tea e stu d c tea c stu k e tea or e stu ? ① teacher model ③ feature knowledge distillation fine-tuning ④ prediction knowledge distillation fine-tuning ② student model fine-tuning figure 2 : the overall framework of mskdts. the model is composed of six component: the teacher encoder e tea , the student encoder e stu , the discriminator d, the teacher classifier c tea , the student classifier c stu and the prediction knowledge distillation component k. in the training phase, we first take evidence and document as the input reference information to pretrain the teacher model and the student model. next, e stu and d compete with each other through adversarial imitation strategy. in addition, the probabilities of the answers predicted by c stu and c tea are aligned by a prediction knowledge distillation approach k. in the final testing phase, documents are used as input to the enhanced e stu and c stu for answer prediction. ## conclusion we propose a multi-strategy knowledge distillation based teacher-student framework (mskdts) for mrc to address the challenges posed by irrelevant information in documents for answer prediction. the teacher-student framework naturally circumvents the error accumulation problem in the traditional pipeline paradigm and the knowledge distillation strategies enhance the model capability at the feature and prediction levels. experiments on the reco dataset demonstrate the effectiveness of our approach. c c l 2 0 2 1