Structure-Level Knowledge Distillation For Multilingual Sequence Labeling (2020)

Abstract: Multilingual sequence labeling is a task of predicting label sequences using a single unified model for multiple languages. Compared with relying on multiple monolingual models, using a multilingual model has the benefit of a smaller model size, easier in online serving, and generalizability to low-resource languages. However, current multilingual models still underperform individual monolingual models significantly due to model capacity limitations. In this paper, we propose to reduce the gap between monolingual models and the unified multilingual model by distilling the structural knowledge of several monolingual models (teachers) to the unified multilingual model (student). We propose two novel KD methods based on structure-level information: (1) approximately minimizes the distance between the student’s and the teachers’ structure-level probability distributions, (2) aggregates the structure-level knowledge to local distributions and minimizes the distance between two local probability distributions. Our experiments on 4 multilingual tasks with 25 datasets show that our approaches outperform several strong baselines and have stronger zero-shot generalizability than both the baseline model and teacher models.

## introduction sequence labeling is an important task in natural language processing. many tasks such as named entity recognition (ner) and part-of-speech (pos) tagging can be formulated as sequence labeling problems and these tasks can provide extra information to many downstream tasks and products such as searching engine, chat-bot and syntax parsing @xcite . most of the previ-ous work on sequence labeling focused on monolingual models, and the work on multilingual sequence labeling mainly focused on cross-lingual transfer learning to improve the performance of low-resource or zero-resource languages @xcite @xcite @xcite , but their work still trains monolingual models. however, it would be very resource consuming considering if we train monolingual models for all the 7,000+ languages in the world. besides, there are languages with limited labeled data that are required for training. therefore it is beneficial to have a single unified multilingual sequence labeling model to handle multiple languages, while less attention is paid to the unified multilingual models due to the significant difference between different languages. recently, multilingual bert (m-bert) @xcite is surprisingly good at zero-shot cross-lingual model transfer on tasks such as ner and pos tagging @xcite . m-bert bridges multiple languages and makes training a multilingual sequence labeling model with high performance possible @xcite . however, accuracy of the multilingual model is still inferior to monolingual models that utilize different kinds of strong pretrained word representations such as contextual string embeddings (flair) proposed by @xcite . to diminish the performance gap between monolingual and multilingual models, we propose to utilize knowledge distillation to transfer the knowledge from several monolingual models with strong word representations into a single multilingual model. knowledge distillation @xcite is a technique that first trains a strong teacher model and then trains a weak student model through mimicking the output probabilities @xcite @xcite or hidden states @xcite of the teacher model. the student model can achieve an accuracy comparable to that of the teacher model and usually has a smaller model size through kd. inspired by kd applied in neural machine translation (nmt) @xcite and multilingual nmt @xcite , our approach contains a set of monolingual teacher models, one for each language, and a single multilingual student model. both groups of models are based on bilstm-crf @xcite , one of the state-of-the-art models in sequence labeling. in bilstm-crf, the crf layer models the relation between neighbouring labels which leads to better results than simply predicting each label separately based on the bilstm outputs. however, the crf structure models the label sequence globally with the correlations between neighboring labels, which increases the difficulty in distilling the knowledge from the teacher models. in this paper, we propose two novel kd approaches that take structure-level knowledge into consideration for multilingual sequence labeling. to share the structure-level knowledge, we either minimize the difference between the student's and the teachers' distribution of global sequence structure directly through an approximation approach or aggregate the global sequence structure into local posterior distributions and minimize the difference of aggregated local knowledge. experimental results show that our proposed approach boosts the performance of the multilingual model in 4 tasks with 25 datasets. furthermore, our approach has better performance in zero-shot transfer compared with the baseline multilingual model and several monolingual teacher models. ## approach in this section, we propose two approaches to learning a single multilingual sequence labeling model (student) by distilling structure-level knowledge from multiple mono-lingual models. the first approach approximately minimizes the difference between structure-level probability distributions predicted by the student and teachers. the second aggregates structure-level knowledge into local posterior distributions and then minimizes the difference between local distributions produced by the student and teachers. our approaches are illustrated in figure 1 . both the student and the teachers are bilstm-crf models @xcite , one of the state-of-the-art models in sequence labeling. a bilstm-crf predicts the distribution of the whole label sequence structure, so token-level distillation is no longer possible and structure-level distillation is required. ## discussion on flair/m-bert fine-tuning previous work has discussed and empirically investigated two ways of adapting monolingual pretrained embedding models to monolingual downstream tasks @xcite : either fixing the models and using them for feature extraction, or fine-tuning them in downstream tasks. they found that both settings have comparable performance in most cases. @xcite found that fine-tuning m-bert with the bottom layers fixed provides further performance gains in multilingual setting. in this paper, we mainly focus on the first approach and utilize the pretrained embedding as fixed feature extractor because flair/m-bert finetuning is too slow for our large-scale experimental design of multilingual kd. designing a cheap and fast fine-tuning approach for pretrained embedding models might be an interesting direction for future work. ## conclusion in this paper our major contributions are the two structure-level methods to distill the knowledge of monolingual models to a single multilingual model in sequence labeling: top-k knowledge distillation and posterior distillation. the experimental results show that our approach improves the performance of multilingual models over 4 tasks on 25 datasets. the analysis also shows that our model has stronger zero-shot transfer ability on unseen languages on the ner and pos tagging task. our code is publicly available at https://github .