Cross-lingual Machine Reading Comprehension with Language Branch Knowledge Distillation (2020)

Abstract: Cross-lingual Machine Reading Comprehension (CLMRC) remains a challenging problem due to the lack of large-scale annotated datasets in low-source languages, such as Arabic, Hindi, and Vietnamese. Many previous approaches use translation data by translating from a rich-source language, such as English, to low-source languages as auxiliary supervision. However, how to effectively leverage translation data and reduce the impact of noise introduced by translation remains onerous. In this paper, we tackle this challenge and enhance the cross-lingual transferring performance by a novel augmentation approach named Language Branch Machine Reading Comprehension (LBMRC). A language branch is a group of passages in one single language paired with questions in all target languages. We train multiple machine reading comprehension (MRC) models proficient in individual language based on LBMRC. Then, we devise a multilingual distillation approach to amalgamate knowledge from multiple language branch models to a single model for all target languages. Combining the LBMRC and multilingual distillation can be more robust to the data noises, therefore, improving the model’s cross-lingual ability. Meanwhile, the produced single multilingual model can apply to all target languages, which saves the cost of training, inference, and maintenance for multiple models. Extensive experiments on two CLMRC benchmarks clearly show the effectiveness of our proposed method.

## introduction machine reading comprehension (mrc) is a central task in natural language understanding (nlu) with many applications, such as information retrieval and dialogue generation. given a query and a text paragraph, mrc extracts the span of the correct answer from the paragraph. recently, as a series of largescale annotated datasets become available, such as squad @xcite and triviaqa @xcite , the performance of mrc systems has been improved dramatically @xcite @xcite @xcite . nevertheless, those large-scale, high-quality annotated datasets often only exist in rich-resource languages, such as english, french and german. correspondingly, the improvement of mrc quality can only benefit those rich-source languages. annotating a large mrc dataset with high quality for every language is very costly and may even be infeasible @xcite . mrc in low-resource languages still suffers from the lack of large amounts of high-quality training data. besides, in real business scenarios, it is not practical to train separate mrc models for each language given that there are thousands of languages existed in the world. thus multi-lingual mrc (single model for multiple languages) is of strong practical value by greatly reducing the model training, serving and maintenance costs. to tackle the challenges of mrc in low-resource languages, cross-lingual mrc (clmrc) is proposed, where translation systems are used to translate datasets from rich-source languages to enrich training data for low-resource languages @xcite . however, clmrc is severely restricted by translation quality @xcite . recently, large-scale pre-trained language models (plm) @xcite @xcite are shown effective in nlu related tasks. inspired by the success of plm, multilingual plm @xcite @xcite are developed by leveraging large-scale multilingual corpuses for cross-lingual pre-training. those powerful multilingual plm are capable of zero-shot or few-shot learning @xcite , and are effective to transfer from rich-resource languages to low-resource languages. although those methods gain significant improvements on sentence-level tasks, such as sentence classification @xcite , there is still a big gap between the performance of clmrc in rich-resource languages and that in low-resource languages, since clmrc requires high quality fine-grained representation at the phaselevel @xcite . several studies combine multilingual plm with translation data to improve the clmrc performance by either data augmentation using translation @xcite or auxiliary tasks @xcite (see section 2 for some details). those studies take two alternative approaches. first, they may just leverage translated data in target languages as new training data to directly train target language models @xcite . the performance of such models is still limited by the translation issues (i.e, the noise introduced by the translation processing). second, they may strongly rely on language-specific external corpuses, which are not widely or easily accessible @xcite . according to the generalized cross-lingual transfer result @xcite , the best cross-lingual performance is often constrained by the passage language, rather than the question language. in other words, the passage language plays an important role in clmrc. the intuition is that the goal of mrc to pinpoint the exact answer boundary in passage, thus the language of passage has stronger influence on the performance than the question language. motivated by this intuition, in this paper, we propose a new cross-lingual training approach based on knowledge distillation for clmrc. we group the translated dataset (i.e., both questions and passages are translated into all target languages) into several groups. a group, called a language branch, contains all passages in one single language paired with questions in all target languages. for each language branch, a separate teacher model is trained. those language branch specific models are taken as teacher models to jointly distill a single multilingual student model using a novel multilingual distillation framework. with this framework, our method can amalgamate multiple language diversity knowledge from language branch specific models to a single multilingual model and can be more robust to defeat the noises in the translated dataset, which obtains better crosslingual performance. we make the following technical contributions. first, on top of translation, we propose a novel language branch training approach by training several language specific models as teachers to provide finegrained supervisions. second, based on those teacher models, we propose a novel multilingual multiteacher distillation framework to transfer the capabilities of the language teacher models to a unified clmrc model. last, we conduct extensive experiments on two popular clmrc benchmark datasets in 9 languages under both translation and zero-shot conditions. our model achieves state-of-the-art results on all languages for both datasets without using any external large-scale corpus. the rest of the paper is organized as follows. we review related work in section 2, and present our method in section 3. we report experimental results in section 4 and conclude the paper in section 5. ## related work our study is mostly related to the existing work on clmrc and knowledge distillation. we briefly review some most related studies here. assuming only annotated data in another source language is available, clmrc reads one context passage in a target language and extracts the span of an answer to a given question. translation based approaches use a translation system to translate labeled data in a source language to a low-resource target language. based on the translated data, @xcite devise a run-time neural machine translation based multilingual extracted question answering method. @xcite propose a data augmenta- ## methodology we formulate the clmrc problem as follows. given a labeled mrc dataset d @xmath0, q src , a src )} in a rich-resource language src, where p src , q src and a src are a passage, a question, and an answer to q src , respectively, the goal is to train a mrc model m for the rich-resource language src and another low-resource language tgt. for an input passage p tgt and question q tgt in tgt, m can predict the answer span a @xmath1, a tgt e ), where a tgt s and a tgt e are the starting and ending indexes of the answer location in passage p tgt , respectively. model m is expected to have good performance not only in the rich-resource language src, but also in the low-resource language tgt. we first propose a new data augmentation based training strategy, language branch machine reading comprehension (lbmrc), to train separate models for each language branch. a language branch is a group that contains passages in one single language accompanied with questions in all target languages. under this setting, we can construct a language branch dataset for each language. using each language branch dataset, we train a separate mrc model proficient in the language. then, the branch-specific mrc models are taken as multiple teacher models to train a single multilingual mrc student model using a novel multilingual language branch knowledge distillation framework. the overview of our approach is illustrated in figure 1 . ## experiments extensive experiments of our proposed method are conducted on two public cross-lingual mrc datasets. in the following sections, we describe our experimental settings, results, and analyze the performance. ## conclusions in this paper, we propose a novel language branch data augmentation based training strategy (lbmrc) and a novel multilingual multi-teacher distillation framework to boost the performance of cross-lingual mrc in low-resource languages. extensive experiments on two multilingual mrc benchmarks verify the effectiveness of our proposed method either in translation or zero-shot settings. we further analyze the reason why combine the lbmrc and multilingual distillation can gain better cross-lingual performance, which shows that our method is more effective to use the translation dataset and more robust to the noise hidden in the translated data. in addition, our distillation framework produces a single multilingual model applicable to all target languages, which is more practical to deploy multilingual serves.