An Unsupervised Multiple-Task and Multiple-Teacher Model for Cross-lingual Named Entity Recognition (2022)

Abstract: Cross-lingual named entity recognition task is one of the critical problems for evaluating the potential transfer learning techniques on low resource languages. Knowledge distillation using pre-trained multilingual language models between source and target languages have shown their superiority in transfer. However, existing cross-lingual distillation models merely consider the potential transferability between two identical single tasks across both domains. Other possible auxiliary tasks to improve the learning performance have not been fully investigated. In this study, based on the knowledge distillation framework and multi-task learning, we introduce the similarity metric model as an auxiliary task to improve the cross-lingual NER performance on the target domain. Specifically, an entity recognizer and a similarity evaluator are first trained in parallel as two teachers from the source domain. Then, two tasks in the student model are supervised by these teachers simultaneously. Empirical studies on the three datasets across 7 different languages confirm the effectiveness of the proposed model.

## introduction named entity recognition, ner in short, refers to identifying entity types, i.e. location, person, organization, etc., in a given sentence. the exploiting of deep neural networks, such as bi-lstm-crf @xcite , bi-lstm-cnn (chiu and nichols, 2016) makes this task achieve significant performances. however, since deep neural networks highly rely on a large amount of labelled training data, the annotation acquiring process is expensive and time consuming. this situation is more severe for zero-resource languages. with the help of transfer learning @xcite and multilingual bert (short as mbert) @xcite 2019), it is possible to transfer the annotated training samples or trained models from a rich-resource domain to a zero-resource domain. many studies have been done to solve this crosslingual ner problem. existing models can be separated into three categories, shared feature space based, translation based and knowledge distillation based. shared feature space based models exploit language-independent features, which lacks the domain-specific features for the target language @xcite @xcite . translation based models generate pseudo labeled target language data to train the cross-lingual ner model, but the noise from translation process restrains its performance. @xcite @xcite . knowledge distillation based models train a student model using soft labels of the target language @xcite @xcite . although the above-mentioned models solve the cross-lingual ner problem to some extent, the auxiliary tasks, as in multi-task learning, have not been studied in this problem. due to the distributed representation of natural languages, the relatedness among the embedding of target languages, which is measured by the similarity, can be utilized to further boost the learned encoder and improve the final ner performance on target languages. here we give a concrete example to illustrate the importance of similarity between every two tokens under the situation when only the english data is labeled. given a spanish sentence "arévalo (avila), 23 may (efe).", the token "arévalo" is recognized as org type using the learned model from the english domain. in the meantime, the token "arévalo" has high similarity scores with the spanish tokens "viena" from sentence "viena, 23 may (efe).", and "madrid" from sentence "madrid, 23 may (efe).". also, the tokens "viena" and "madrid" are recognized correctly as loc type using the same english model mentioned above. then "arévalo" can be recognized correctly as loc type under the supervisory signal using the similarity between "viena" and "madrid". to leverage the similarity between the tokens of the source languages, we design an multiple-task and multiple-teacher model (short as mtmt, as shown in figure 1 ), which helps the ner learning process on the target languages. specifically, we first introduce the knowledge distillation to build entity recognizer and similarity evaluator teachers in the source language and transfer the learned patterns to the student in the target language. in the student model, we then borrow the idea of multitask learning to incorporate a similarity evaluation task as an auxiliary task into the entity recognition classifier. during the student learning process, we input unlabelled samples from the target languages into the entity recognizer and evaluator, and take output pesudo labels as supervisory signals for these two tasks in the student model. note that a weighting strategy is also provide therein to take into consideration of the reliability of the teachers. we validate the model performance on the three commonly-used datasets across 7 languages and the experimental results show the superiority of our presented mtmt model. our main contributions are as follows: our approach is closely related to the existing works on cross-lingual ner, knowledge distillation, and siamese network. cross-lingual ner aims to extract entities from a target language but assumes only source language is annotated. the existing models can be categorized to a) shared feature space based models, b) translation based models, c) knowledge distillation based models. shared feature space based models generally train a language-independent encoder using source and target language data @xcite . recently, the pre-trained multilingual language model is effective to address the challenge @xcite . moreover, some research introduces new components on top of the mbert by directly transferring the model learned from the labeled source language to that of target languages @xcite . the performance is still weak due to the lack of annotations of target languages. translation based models generally generate pseudo-labeled target data to alleviate target data scarcity. for example, @xcite gain an improvement by translating the labeled source language to the target language word-by-word. our model achieves considerable improvement by learning entity similarity in target language data without translation. knowledge distillation based models include a teacher model and a student model @xcite . the teacher model is trained on the labeled source language. the student model learns from the soft label predicted by the teacher model on unlabeled target language data. therefore, the student model can capture the extra knowledge about target languages. in our work, the student model not only learns the recognizer teacher knowledge, but also learns the entity similarity knowledge inspired by multi-task learning. siamese network is originally introduced by @xcite to treat signature verification as a matching problem. it has been successfully applied to transfer learning such as one-shot image recognition @xcite ), text similarity @xcite . however, there is a dilemma to adapt the siamese network to tokenlevel recognition tasks such as ner. siamese network assumes the input is a pair, and the output is a similarity score. to handle this issue, we reconstruct the data to pair format. to the best of our knowledge, we are the first to learn the entity similarity by siamese network. ## framework in this section, we introduce our framework and its detailed implementation. our framework is consist of two models: teacher training model learned from the source language and teacher-student distillation learning model learned from the target language. in the teacher training model, there are two sub-models, i.e. an entity recognizer teacher and a similarity evaluator teacher. these two models are two parallel tasks, wherein the entity recognition teacher focuses on identifying the named entities and the similarity evaluator teacher is to decide if two tokens are in the same type. we then present a teacher-student distillation learning model to learn from the two learned teacher models simultaneously. we note that, in this learning process, such a knowledge distillation makes the student model combine the advantages of both source language patterns of entity recognition and entity similarity evaluation. during the learning process, the samples from the target language are fed into the teacher model and the outputs are taken as the supervisory signal for two tasks in the student model. to guarantee the student learning performance, we assign weights for each supervisory signal correspond to the output confidence of teacher sub-models. we argue that the student entity recognition task and the student entity similarity evaluation task improve the representation learning of the student encoder in the siamese structure. ## conclusion in this paper, we propose an unsupervised multipletask and multiple-teacher model for cross-lingual ner. the student model learns two source language patterns of entity recognition and entity similarity evaluation. moreover, to guarantee the student learning performance, we also propose a weighting strategy to take into consideration the reliability of the teachers. our experimental results show that the proposed model yields significant improvements on six target language datasets and outperforms the existing state-of-the-art approaches.