Enhancing Answer Boundary Detection for Multilingual Machine Reading Comprehension (2020)

Abstract: Multilingual pre-trained models could leverage the training data from a rich source language (such as English) to improve performance on low resource languages. However, the transfer quality for multilingual Machine Reading Comprehension (MRC) is significantly worse than sentence classification tasks mainly due to the requirement of MRC to detect the word level answer boundary. In this paper, we propose two auxiliary tasks in the fine-tuning stage to create additional phrase boundary supervision: (1) A mixed MRC task, which translates the question or passage to other languages and builds cross-lingual question-passage pairs; (2) A language-agnostic knowledge masking task by leveraging knowledge phrases mined from web. Besides, extensive experiments on two cross-lingual MRC datasets show the effectiveness of our proposed approach.

## introduction machine reading comprehension (mrc) plays a critical role in the assessment of how well a machine could understand natural language. among various types of mrc tasks, the span extractive reading comprehension task (like squad @xcite ) has been become very popular. promising achievements have been made with neural network based approaches @xcite @xcite @xcite , especially those built on pre-trained language models such as bert @xcite , due to the availability of large-scale annotated corpora @xcite @xcite . however, these large-scale annotated corpora are language mrc nli em gap to en acc gap to en en 62.4 -85.0 es 49.8 -12.6 78.9 -6.1 de 47.6 -14.8 77.8 -7.2 ar 36.3 -26.1 73.1 -11.9 hi 27.3 -35.1 69.6 -15.4 vi 41.8 -20.6 76.1 -8.9 zh 39.6 -22.8 76.5 -8. 5 to alleviate the scarcity of training data for multilingual mrc, the translation based data augmentation approaches were firstly proposed. for example, (question q, passage p, answer a) in english squad can be translated into (q , p , a ) in other languages @xcite to enrich the non-english mrc training data. however, these approaches are limited by the quality of the translators, especially for those low resource languages. most recently, approaches based on multilingual/cross-lingual pre-trained models @xcite @xcite have proved very effective on several crosslingual nlu tasks. these approaches learn language-agnostic features and align language representations in vector space during multilingual pre-training process @xcite @xcite [question]: who were the kings of the southern kingdom [passage]: in the southern kingdom there was only one dynasty, that of king david, except usurper athaliah from the northern kingdom, who by marriage, [. . . ] [answer -ground truth]: king david [answer -model predication:] david, except usurper athaliah @xcite : what is the suggested initial does dosage of chlordiazepoxide @xcite : if the drug is administered orally, the suggested initial dose is 50 to 100 mg, to be followed by repeated doses as needed until agitation is controlledup to 300 mg per day. [. . . ] [answer -ground truth]: 50 to 100 mg [answer -model predication:] 100 mg for extractive mrc, the em metric is very critical since it indicates the answer boundary detection capability, i.e. the accuracy for extractive answer spans. in motivated by the above observations, we propose two auxiliary tasks to enhance boundary detection for multilingual mrc, especially for low-resource languages. first, we design a cross-lingual mrc task with mixed-languages question, passage pairs to better align the language representation. we then propose a knowledge phrase masking task as well as a languageagnostic method to generate per-language knowledge phrases from the web. extensive experiments on two multilingual mrc datasets show that our proposed tasks could substantially boost the model performance on answer span boundary detection. the main contributions of our paper can be summarized as follows. ## approach in this section, we first introduce the overall training procedure, and then introduce two new tasks, namely, mixed machine reading comprehension (mixmrc) and language-agnostic knowledge phrase masking (lakm), respectively. the overview of our training procedure is shown at figure 1 . our approach is built on top of popular multilingual pre-trained models (such as multilingual bert and xlm). we concatenate passage, question (optional) together with special tokens representation is used for all three tasks introduced as following. the first task, also our main task, is multilingual mrc, which aims to extract answers spans from the context passage according to the question. in this task, each language has its own data. however, only english has human labeled training data, and the other languages use machine translated training data from english. during training, the mrc training data in all languages will be used together for fine-tuning. in the following, we introduce our new proposed tasks which will jointly train with our main task to boost multilingual mrc performance. ## experiments in this section, we firstly describe the dataset and evaluation in section 4.1; then introduce the baseline models in section 4.2 and experiment setting in section 4.3; thirdly the experimental results are shown in section 4.4. ## analysis in this section, we ablate important components in lakm to explicitly demonstrate its effectiveness. ## conclusion this paper proposes two auxiliary tasks (mixmrc and lakm) in the multilingual mrc fine-tuning stage to enhance answer boundary detection especially for low resource languages. extensive experiments on two multilingual mrc datasets have been conducted to prove the effective of our proposed approach. meanwhile, we further analyze the model performance on fine-grained answer types, which shows interesting insights.