Multi-level Distillation of Semantic Knowledge for Pre-training Multilingual Language Model (2022)

Abstract: Pre-trained multilingual language models play an important role in cross-lingual natural language understanding tasks. However, existing methods did not focus on learning the semantic structure of representation, and thus could not optimize their performance. In this paper, we propose Multi-level Multilingual Knowledge Distillation (MMKD), a novel method for improving multilingual language models. Specifically, we employ a teacher-student framework to adopt rich semantic representation knowledge in English BERT. We propose token-, word-, sentence-, and structure-level alignment objectives to encourage multiple levels of consistency between source-target pairs and correlation similarity between teacher and student models. We conduct experiments on cross-lingual evaluation benchmarks including XNLI, PAWS-X, and XQuAD. Experimental results show that MMKD outperforms other baseline models of similar size on XNLI and XQuAD and obtains comparable performance on PAWS-X. Especially, MMKD obtains significant performance gains on low-resource languages.

## introduction pre-training a large-scale language model and finetuning it on downstream tasks has shown great success in natural language processing. most works @xcite @xcite focused on english, since it is easy to get a large amount of training data for english. this paradigm has recently emerged as a promising means for cross-lingual tasks. some multilingual language models @xcite trained on monolingual data from over 100 languages using masked language modeling (mlm) objective performed surprisingly well without any explicit alignment. on the other hand, xlm @xcite extended mlm objective to a parallel corpus version -translation language modeling (tlm), and achieved impressive results. this inspired researchers to develop alignment methods using parallel corpora. follow-up works of xlm @xcite @xcite leveraged various of training objectives to align parallel sentences at different granularity. these works were usually trained using both large amounts of monolingual data and additional parallel corpora, which are time-consuming and require considerable computational resources. another line of research @xcite @xcite only used limited parallel data to improve existing pre-trained language models rather than training new models from scratch. they depended on new alignment methods for parallel pairs of words and sentences, which could further achieve performance gains over current state-of-the-art pre-trained language models. however, these approaches neglected vector space properties when aligning across languages, and thus generating sub-optimal results. we hypothesize that a large-scale english corpus can provide more semantic and structural information than most other languages used to train multilingual language models. moreover, bert @xcite , which is trained from a vast amount of english wikipedia and bookscorpus @xcite , can capture this information properly and guide the training procedure of other languages, especially for those with limited resources. in this work, we employ a teacher-student framework to adopt vector space properties in english, and transfer its rich knowledge to our multilingual language model. we use bert-base as the teacher model and multilingual bert (mbert; @xcite as the student model. we propose a multi-level multilingual knowledge distillation (mmkd) method to align semantically similar sentences in parallel corpora to improve mbert. specifically, we propose a cross-lingual word-aware contrastive learning (xwcl) to encourage word representation similarity between teacher and student networks. we also adopt tlm objective in the student network to take advantage of corresponding context information in the target languages of masked tokens. we present a new sentence-level alignment objective to imitate english sentence projections from the teacher network. moreover, we propose a structure-level alignment objective to transfer relationships between sentences in bert vector space. we conduct experiments on zero-shot cross-lingual natural language understanding tasks, including natural language inference, paraphrase identification, and question answering. experimental results show that mmkd significantly improves mbert and outperforms baseline models of similar size. the analysis demonstrates the cross-lingual transferability of mmkd on low-resource languages. mmkd provides a more feasible and effective pre-training procedure that only requires limited training data and fewer computational resources. one popular family of methods proposed different training objectives to align words or sentences from parallel corpora. xlm @xcite extended mlm objective to a parallel corpus version -translation language modeling (tlm). unicoder @xcite improved the transferability by presenting five pre-training tasks. alm @xcite predicted words in the context of code-switching sentences. hictl @xcite introduced sentence-level and word-level alignment with contrastive learning. in-foxlm @xcite proposed cross-lingual contrast (xlco) to maximize mutual information of sentence pairs. ernie-m @xcite presented cross-attention masked language modeling (camlm) and back-translation masked modeling (btmlm) to leverage both parallel and monolingual corpora. more recently, researchers considered computational resources and time and presented works based on existing multilingual language models. @xcite minimized the similarity between word pairs in parallel sentences in a post-hoc manner. @xcite argued that creating word alignments using fastalign @xcite would suffer from the noise of the toolkit and neglected the contextual information. they proposed post-pretraining alignment (ppa) that combined a different tlm objective and a contrastive learning objective. amber @xcite presented objectives that encouraged prediction of the corresponding sentence and consistency between attention matrices, and they pre-trained the model with an extremely large batch size of 8,192 for the first 1m steps. @xcite first introduced knowledge distillation to transfer knowledge to a small model, and it has been widely used for transferring dark knowledge (which refers to information that can tell us how the model tends to generalize) and model compression in natural language processing and computer vision. a series of follow-up works achieved gains on multilingual tasks. @xcite enhanced the generalization ability of unsupervised neural machine translation by adding self-knowledge distillation and language branch knowledge distillation. @xcite reduced the distance between monolingual teachers and the multilingual student to predict multilingual label sequences. to the best of our knowledge, @xcite is the only multilingual language model related work that applied a student model to mimic sentence representations generated from the teacher model. they fed both source and target sentences into the student model to calculate mean square error (mse) loss with the teacher model's source sentences. and aligning multi-level information in parallel corpora with limited resources. the general network architecture is illustrated in figure 1 . the student network consists of three components: an encoder, a projector, and a predictor, while the teacher network contains an encoder and a projector. ## experiments this section explains our training details and shows the experimental results on xnli, paws-x and xquad. we compare our proposed mmkd with existing works following the setting in @xcite and conduct ablation studies to prove the effectiveness of each proposed objective. ## conclusion in this work, we propose a multi-level multilingual knowledge distillation method to pre-train the multilingual language model -mbert. we propose four training objectives to align token-, word-, and sentence-level information from parallel corpora, and we also learn knowledge correlation between teacher and student models. compared to existing studies, we require fewer computational resources and less training time. in the zero-shot cross-lingual setting, our model outperforms models of similar size and reduces the performance gap to larger models across xnli, paws-x, and xquad benchmarks. experimental results show that mmkd obtains significant performance gains on low-resource languages and does well on latin-based languages. visualization result shows that semantic relationships among sentences have been successfully transferred from english to other languages. future work could extend our approach to other larger multilingual language models.