Wider & Closer: Mixture of Short-channel Distillers for Zero-shot Cross-lingual Named Entity Recognition (2022)

Abstract: Zero-shot cross-lingual named entity recognition (NER) aims at transferring knowledge from annotated and rich-resource data in source languages to unlabeled and lean-resource data in target languages. Existing mainstream methods based on the teacher-student distillation framework ignore the rich and complementary information lying in the intermediate layers of pre-trained language models, and domain-invariant information is easily lost during transfer. In this study, a mixture of short-channel distillers (MSD) method is proposed to fully interact the rich hierarchical information in the teacher model and to transfer knowledge to the student model sufficiently and efficiently. Concretely, a multi-channel distillation framework is designed for sufficient information transfer by aggregating multiple distillers as a mixture. Besides, an unsupervised method adopting parallel domain adaptation is proposed to shorten the channels between the teacher and student models to preserve domain-invariant features. Experiments on four datasets across nine languages demonstrate that the proposed method achieves new state-of-the-art performance on zero-shot cross-lingual NER and shows great generalization and compatibility across languages and fields.

## introduction named entity recognition (ner) is a fundamental and important task to locate and classify named entities in a text sequence. recently, deep neural networks have achieved great performance on monolingual ner in rich-resource languages with abundant labeled data @xcite @xcite . however, it is too expensive to annotate a large amount of data in low-resource languages for supervised ner * equal contribution. † corresponding author. training. this issue drives research on cross-lingual ner, which utilizes the rich-resource annotated data in source languages to alleviate the scarcity of unlabeled lean-resource data in target languages. in this paper, following @xcite , we focus on the extremely low-resource setting, i.e., zeroshot cross-lingual ner, where labeled data is not available in target languages. the most popular approaches for zero-shot crosslingual ner are based on distillation @xcite @xcite . they employed a supervisedly trained teacher model to predict the soft labels of target languages, and then utilized the soft labels to distill a student model, which was first exploited in @xcite . besides, domain-invariant features have been proven effective for distillation @xcite . @xcite proposed to alleviate the representation discrepancy between languages in the teacher model to exploit language-independent features, which were further distilled to the student model. it is worth noting that distillation-based methods assisted with auxiliary tasks have become the mainstream paradigm due to their robustness and scalability, achieving good performance on zeroshot cross-lingual ner @xcite . however, these methods always ignored the rich and complementary information lying in the intermediate layers of multilingual bert (mbert) @xcite . @xcite and @xcite have verified that the upper layers of mbert are more task-specific and not as important as the lower ones in terms of crosslanguage transfer. but recent studies just adopted the last layer of mbert for distillation @xcite , while neglected the explicit knowledge transfer of the lower layers. besides, domain-invariant features in the teacher model were first exploited, which were then transferred to the student model via distillation @xcite . however, due to the limitation of transfer learning, it is difficult to fully retain the domaininvariant features to the student model. furthermore, auxiliary tasks to assist distillation usually require the operations of translation or data sifting @xcite , resulting in huge pre-processing costs. the ensemble strategies to generate highquality soft labels or augmented data also require oceans of model parameters and a large number of computational resources. on account of the above issues, a mixture of short-channel distillers (msd) method is proposed in this paper to transfer cross-lingual ner knowledge sufficiently and efficiently. on the one hand, a multi-channel distillation framework is designed to let the hierarchical information in the teacher model fully interact with each other, and transfer more complementary information to the student model. specifically, the teacher model is first trained on the annotated source data with each layer being directly supervised by the labels. then, each layer of the teacher model is tasked to predict the soft labels of the unlabeled target data. correspondingly, the layers of the student model are distilled by leveraging the mixture sets of soft labels from the teacher model, constructing multiple information transmission channels for a "wider" bridge between the teacher and student models. on the other hand, an unsupervised auxiliary task of parallel domain adaptation is proposed to explicitly transfer domain information. during every batch of distillation, as figure 1 depicts, unlabeled target data is fed into the student model, while unlabeled source data is fed into both the teacher and student models. the representation discrepancy between the outputs of the teacher and student source language, together with that between the outputs of the teacher source and student target languages is minimized to preserve the cross-model and crosslanguage domain information respectively. in this way, the domain information can be preserved across models and languages, so that the domains of the teacher and student models can be effectively pulled "closer". experiments on four datasets across nine languages are conducted to evaluate the effectiveness of the proposed msd. the results show that our method achieves new state-of-the-art performance on all datasets. in summary, our contributions are as follows: (1) a multi-channel framework is proposed to leverage the rich, hierarchical and complementary information contained in the teacher model, and to interactively transfer cross-lingual ner knowledge to the student model. (2) an unsupervised auxiliary method is designed to explicitly constrain the discrepancy of teacher/student domains without utilizing any external resources. (3) experiments on four datasets across nine languages verify the effectiveness and generalization ability of msd. ## methodology in this section, we present the detailed framework of the proposed mixture of short-channel distillers (msd). on the one hand, the mixture of distillers module is introduced. specifically, multiple channels are built between corresponding layers of the teacher and student's encoders. then a mixture of weights is employed to control the broadened information transmission route. on the other hand, parallel domain adaptation is conducted to explicitly transfer domain information between the teacher and student models during distillation. ## experiments in this section, the proposed msd was evaluated on four zero-shot cross-lingual ner datasets and compared with several state-of-the-art models. some ablation studies were also conducted to validate the effectiveness of the proposed modules. ## conclusion in this paper, we propose a mixture of shortchannel distillers framework for zero-shot crosslingual ner, including a multi-channel distillation framework to fully leverage the complementary and hierarchical information in the teacher model, and an unsupervised parallel domain adaptation method to effectively pull the domains between teacher and student models closer. experimental results show that the proposed method outperforms previous methods on four datasets across nine languages. in the future, we will extend this method to languages where data resources are scarcer.