Sharing, Teaching and Aligning: Knowledgeable Transfer Learning for Cross-Lingual Machine Reading Comprehension (2023)

Abstract: In cross-lingual language understanding, machine translation is often utilized to enhance the transferability of models across languages, either by translating the training data from the source language to the target, or from the target to the source to aid inference. However, in cross-lingual machine reading comprehension (MRC), it is difficult to perform a deep level of assistance to enhance cross-lingual transfer because of the variation of answer span positions in different languages. In this paper, we propose X-STA, a new approach for cross-lingual MRC. Specifically, we leverage an attentive teacher to subtly transfer the answer spans of the source language to the answer output space of the target. A Gradient-Disentangled Knowledge Sharing technique is proposed as an improved cross-attention block. In addition, we force the model to learn semantic alignments from multiple granularities and calibrate the model outputs with teacher guidance to enhance cross-lingual transferability. Experiments on three multi-lingual MRC datasets show the effectiveness of our method, outperforming state-of-the-art approaches.

## introduction recently, significant progress has been made in nlp by pre-trained language models (plms) @xcite @xcite . yet, these models often require a sufficient amount of training data to perform well, which is difficult to achieve in cross-lingual low-resource adaptation. although many cross-lingual plms have been proposed to learn generic feature representations @xcite @xcite @xcite , the performance gap between figure 1 : machine translation as an aid for cross-lingual transfer. above is a natural language inference (nli) task. the probability distribution of the source language can be fitted by kl divergence for teaching lowresource languages; during inference, the target language can be translated into the source language with its output used for calibration. below is an mrc task, where the knowledge is difficult to transfer directly. one of the most significant challenges in crosslingual mrc is the lack of annotated datasets in low-resource languages, which are difficult to obtain. as seen, most of the current mrc datasets are in english @xcite . another challenge is the linguistic and cultural variations that exist across different languages, which exhibit different sentence structures, word orders and morphological features. for instance, languages such as japanese, chinese, hindi and arabic have different writing systems and a more complicated grammatical system than english, making it challenging for mrc models to comprehend the texts. in the literature, machine translation-based data augmentation is often employed to translate the dataset of the source language into each target language for model training @xcite @xcite . as shown in figure 1 , it is relatively easy to enhance cross-lingual transferability of simple sequential classification tasks by directly fitting the output probability distribution of the source language via kullback-leibler divergence @xcite @xcite . however, for mrc, it is not possible to use the output distribution of the source language directly to teach the target language, due to the answer span shift caused by translation. motivated by this, we propose x-sta, a new approach for cross-lingual mrc that follows three principles: sharing, teaching and aligning. for sharing, we propose the gradient-disentangled knowledge sharing (gdks) technique, which uses parallel language pairs as model inputs and extracts knowledge from the source language. it enhances the understanding of the target language while avoiding degradation of the source language representations. for teaching, our approach leverages an attention mechanism by finding answers span from the target language's context that are semantically similar to the source language's output answers to calibrate the output answers. for aligning, alignments at multi-granularity are utilized to further enhance the cross-lingual transferability of the mrc model. in this way, we can enhance the language understanding of the model for different languages through knowledge sharing, teacherguided calibration and multi-granularity alignment. in summary, the main contributions of this study are as follows: ## related work in this section, we summarize the related work in the following three aspects. ## x-sta: proposed approach in this section, we present the detailed techniques of x-sta for cross-lingual mrc. ## conclusion in this paper, we propose x-sta, which addresses the challenges of cross-lingual mrc in effectively utilizing translation data and the linguistic and cultural differences. our work follows three principles: sharing, teaching and aligning. experimental results on three datasets show that our approach obtains the state-of-the-art performance compared to strong baselines. we further analyze the effec- passage: after leaving edison's company tesla partnered with two businessmen in 1886, robert lane and benjamin vail, who agreed to finance an electric lighting company in tesla's name, tesla electric light & manufacturing. the company installed electrical arc light based illumination systems designed by tesla and also had designs for dynamo electric machine commutators, the first patents issued to tesla in the us. question: what was produced at tesla's company? ours: electrical arc light based illumination systems chatgpt: tesla electric light & manufacturing produced electrical arc light based illumination systems and dynamo electric machine commutators. passage: ए�डसन क� कं पनी छोड़ने के बाद टे स्ला ने दो व्यापा�रय� रॉबटर् ले न और ब� जा�मन वे ल के साथ 1886, भागीदार� क�, जो टे स्ला के नाम पर एक इले िक्ट्रक लाइ�टं ग कं पनी टे स्ला इले िक्ट्रक लाइट एं ड मै न्यु फै क्च�रं ग को फाइन� स करने के �लए सहमत ह ु ए। कं पनी ने टे स्ला द्वारा �डज़ाइन �कए गए इले िक्ट्रकल आकर् लाइट आधा�रत रोशनी प्रणाल� इनस्टॉल �कए और उनके पास डायनामो इले िक्ट्रक मशीन कम्यू टे टर के �लए �डजाइन भी थे , अमे �रका म� टे स्ला को �दया गया पहला पे ट� ट था। question: टे स्ला क� कं पनी म� �कसका उत्पादन �कया गया था? ours w/o. atgc: डायनामो इले िक्ट्रक मशीन कम्यू टे टर attentive teacher output: इले िक्ट्रकल आकर् लाइट आधा�रत रोशनी प्रणाल� ours: इले िक्ट्रकल आकर् लाइट आधा�रत रोशनी प्रणाल� chatgpt: टे स्ला क� कं पनी ने इले िक्ट्रक लाइ�टं ग के �लए उत्पादन �कया जाने वाले आइटम जै से इले िक्ट्रक लाइट्स, इले िक्ट्रक मोटसर् , डायनामो इले िक्ट्रक मशीन, कम्यू टे टर आधा�रत रोशनी प्रणाल� आ�द का उत्पादन �कया था। tiveness of each component. in the future, we will extend our work to other cross-lingual nlp tasks for low-resource languages.