Single-/Multi-Source Cross-Lingual NER via Teacher-Student Learning on Unlabeled Data in Target Language (2020)

Abstract: To better tackle the named entity recognition (NER) problem on languages with little/no labeled data, cross-lingual NER must effectively leverage knowledge learned from source languages with rich labeled data. Previous works on cross-lingual NER are mostly based on label projection with pairwise texts or direct model transfer. However, such methods either are not applicable if the labeled data in the source languages is unavailable, or do not leverage information contained in unlabeled data in the target language. In this paper, we propose a teacher-student learning method to address such limitations, where NER models in the source languages are used as teachers to train a student model on unlabeled data in the target language. The proposed method works for both single-source and multi-source cross-lingual NER. For the latter, we further propose a similarity measuring method to better weight the supervision from different teacher models. Extensive experiments for 3 target languages on benchmark datasets well demonstrate that our method outperforms existing state-of-the-art methods for both single-source and multi-source cross-lingual NER.

## introduction named entity recognition (ner) is the task of identifying text spans that belong to pre-defined categories, like locations, person names, etc. it's a fundamental component in many downstream tasks, and has been greatly advanced by deep neural networks @xcite @xcite . however, these approaches generally require massive manually labeled data, which prohibits their adaptation to lowresource languages due to high annotation costs. one solution to tackle that is to transfer knowledge from a source language with rich labeled data to a target language with little or even no labeled data, which is referred to as cross-lingual ner @xcite . in this paper, following @xcite and @xcite , we focus on the extreme scenario of crosslingual ner where no labeled data is available in the target language, which is challenging in itself and has attracted considerable attention from the research community in recent years. previous works on cross-lingual ner are mostly based on label projection with pairwise texts or direct model transfer. label-projection based methods focus on using labeled data in a source language to generate pseudo-labelled data in the target language for training an ner model. for example, @xcite creates automatically labeled ner data for the target language via label projection on comparable corpora and develops a heuristic scheme to select good-quality projection-labeled data. @xcite and @xcite translate the source language labeled data at the phrase/word level to generate pairwise labeled data for the target language. differently, model-transfer based methods @xcite focus on training a shared ner model on the labeled data in the source language with languageindependent features, such as cross-lingual word representations @xcite , and then directly testing the model on the target language. however, there are limitations in both labelprojection based methods and model-transfer based methods. the former relies on labeled data in the source language for label projection, and thus is not applicable in cases where the required labeled data is inaccessible (e.g., due to privacy/sensitivity issues). meanwhile, the later does not leverage unlabeled data in the target language, which can be much cheaper to obtain and probably contains very useful language information. in this paper, we propose a teacher-student learning method for cross-lingual ner to address the mentioned limitations. specifically, we leverage multilingual bert @xcite as the base model to produce language-independent features. a previously trained ner model for the source language is then used as a teacher model to predict the probability distribution of entity labels (i.e., soft labels) for each token in the non-pairwise unlabeled data in the target language. finally, we train a student ner model for the target language using the pseudo-labeled data with such soft labels. the proposed method does not rely on labelled data in the source language, and it also leverages the available information from unlabeled data in the target language, thus avoiding the mentioned limitations of previous works. note that we use the teacher model to predict soft labels rather than hard labels (i.e., one-hot labelling vector), as soft labels can provide much more information @xcite for the student model. figure 1 shows the differences between the proposed teacher-student learning method and the typical label-projection or model-transfer based methods. we further extend our teacher-student learning method to multi-source cross-lingual ner, considering that there are usually multiple source languages available in practice and we would prefer transferring knowledge from all source languages rather than a single one. in this case, our method still enjoys the same advantages in terms of data availability and inference efficiency, compared with existing works @xcite @xcite . moreover, we propose a method to measure the similarity between each source language and the target language, and use this similarity to better weight the supervision from the corresponding teacher model. we evaluate our proposed method for 3 target languages on benchmark datasets, using different source language settings. experimental results show that our method outperforms existing state-of-the-art methods for both single-source and multi-source cross-lingual ner. we also conduct case studies and statistical analyses to discuss why teacher-student learning reaches better results. the main contributions of this work are: single-source cross-lingual ner: such approaches consider one single source language for knowledge transfer. previous works can be divided into two categories: label-projection and modeltransfer based methods. label-projection based methods aim to build pseudo-labeled data for the target language to train an ner model. some early works proposed to use bilingual parallel corpora and project model expectations @xcite or labels @xcite from the source language to the target language with external word alignment information. but obtaining parallel corpora is expensive or even infeasible. to tackle that, recent methods proposed to firstly translate source-language labeled data at the phrase level @xcite or word level @xcite , and then directly copy labels across languages. but translation introduces extra noise due to sense ambiguity and word order differences between languages, thus hurting the trained model. model-transfer based methods generally rely on language-independent features (e.g., crosslingual word embeddings @xcite @xcite , word clusters @xcite , gazetteers @xcite , and wikifier features @xcite ), so that a model trained with such features can be directly applied to the target language. for further improvement, @xcite proposed constructing a pseudotraining set for each test case and fine-tuning the model before inference. however, these methods do not leverage any unlabeled data in the target language, though such data can be easy to obtain and benefit the language/domain adaptation. multi-source cross-lingual ner: multisource cross-lingual ner considers multiple source languages for knowledge transfer. @xcite and @xcite concatenated the labeled data of all source languages to train a unified model, and performed cross-lingual ner in a direct model transfer manner. @xcite leveraged adversarial networks to learn language-independent features, and learns a mixture-of-experts model @xcite to weight source models at the token level. however, both methods straightly rely on the availability of labeled data in the source languages. differently, enghoff et al. (2018) implemented multi-source label projection and studied how source data quality influence performance. @xcite applied truth inference to model the transfer annotation bias from multiple sourcelanguage models. however, both methods make predictions via an ensemble of source-language models, which is cumbersome and computationally expensive, especially when a source-language model has massive parameter space. teacher-student learning: early applications of teacher-student learning targeted model compression @xcite , where a small student model is trained to mimic a pre-trained, larger teacher model or ensemble of models. it was soon applied to various tasks like image classification @xcite , dialogue generation @xcite , and neural machine translation @xcite , which demonstrated the usefulness of the knowledge transfer approach. encoder layer linear classification layer student loss function gradient back-propagation encoder layer linear classification layer teacher inference training unlabeled target-language data in this paper, we investigate teacher-student learning for the task of cross-lingual ner, in both single-source and multi-source scenarios. different from previous works, our proposed method does not rely on the availability of labelled data in source languages or any pairwise texts, while it can also leverage extra information in unlabeled data in the target language to enhance the cross-lingual transfer. moreover, compared with using an ensemble of source-language models, our method uses a single student model for inference, which can enjoy higher efficiency. ## methodology named entity recognition can be formulated as a sequence labeling problem, i.e., given a sentence @xmath0, an ner model is supposed to infer the entity label y i for each token x i and output a label sequence @xmath1. under the paradigm of cross-lingual ner, we assume there are k source-language models previously trained with language-independent features. our proposed teacher-student learning method then uses those k source-language models as teachers to train an effective student ner model for the target language on its unlabeled data d tgt . ## experiments we conduct extensive experiments for 3 target languages (i.e., spanish, dutch, and german) on standard benchmark datasets, to validate the effectiveness and reasonableness of our proposed method for single-and multi-source cross lingual ner. ## conclusion in this paper, we propose a teacher-student learning method for single-/multi-source cross-lingual ner, via using source-language models as teachers to train a student model on unlabeled data in the target language. the proposed method does not rely on labelled data in the source languages and is capable of leveraging extra information in the unlabelled target-language data, which addresses the limitations of previous label-projection based and model-transfer based methods. we also propose a language similarity measuring method based on language identification, to better weight different teacher models. extensive experiments on benchmark datasets show that our method outperforms the existing state-of-the-art approaches.