C on NER : Consistency Training for Cross-lingual Named Entity Recognition (2022)

Abstract: Cross-lingual named entity recognition (NER) suffers from data scarcity in the target languages, especially under zero-shot settings. Existing translate-train or knowledge distillation methods attempt to bridge the language gap, but often introduce a high level of noise. To solve this problem, consistency training methods regularize the model to be robust towards perturbations on data or hidden states.However, such methods are likely to violate the consistency hypothesis, or mainly focus on coarse-grain consistency.We propose ConNER as a novel consistency training framework for cross-lingual NER, which comprises of: (1) translation-based consistency training on unlabeled target-language data, and (2) dropout-based consistency training on labeled source-language data. ConNER effectively leverages unlabeled target-language data and alleviates overfitting on the source language to enhance the cross-lingual adaptability. Experimental results show our ConNER achieves consistent improvement over various baseline methods.

## introduction with the emergence of large scale multilingual pretrained language models @xcite , cross-lingual named entity recognition (ner) performance has seen substantial improvement, especially under zero-shot setting where no labeled target-language training data is available @xcite @xcite . nevertheless, due to linguistic gaps between languages, ner models trained solely on the source-language data are likely to overfit the source language's characteristics, and still suffer from sub-optimal performance when tested on target languages directly. to further enhance cross-lingual adaptability, various methods have been proposed to explicitly incorporate target-language information during training. these approaches can be roughly categorized into: (1) translate-train and (2) knowledge distillation. translate-train produces parallel target-language training data by translating the source-language training data and mapping the labels @xcite @xcite . however, the translated sentences are less natural and the label mapping is also error-prone. moreover, without utilizing the abundant unlabeled target-language data, their performance is usually sub-optimal @xcite . on the other hand, knowledge distillation trains the student model on target-language data with soft labels from a teacher model @xcite @xcite . given the relatively low performance of the teacher model on target languages, the soft labels are noisy and limit the efficacy of unlabeled target-language data when using these labels as supervision signals. in this paper, we primarily explore how to better utilize unlabeled target-language data. instead of adopting knowledge distillation, which is vulnerable to the noise in soft labels, we adopt consistency training as a more fault-tolerant strategy. in general, consistency training works by enhancing the smoothness of output distributions and improving the model's robustness and generalization @xcite . its training process is less sensitive towards noisy labels by regulating multiple predictions on different views of the same data point. several works already attempted to apply consistency training on ner, including token-level and sequence-level consistency methods. token-level consistency works on the same granularity as ner by regularizing the model to be invariant towards gaussian noise @xcite or word replacement @xcite . however, such naïve noise or augmentation might violate the assumption that noised tokens share the same labels as the original ones. on the other hand, a sequence-level consistency method @xcite attempted to adapt back-translation-based @xcite consistency to ner. due to difficulties in word alignment, they only use a coarse-grain consistency on the appearance of a certain type of entity in the original and back-translated sentences, which is plausibly a sub-optimal design for token-level tasks like ner. to this end, other works used constituentbased tagging schemes @xcite . we propose conner, a cross-lingual ner framework that primarily leverages consistency training on unlabeled target-language data using translation. concretely, conner encourages the model to output consistent predictions between a span of tokens in the original sentence and their projection in the translated sentence. we tackle the problem of word alignment with alignment-free translation, and propose span-level consistency constraints to overcome the problem of token number changes during translation. another advantage of translating unlabeled target-language data into the source language is that we can obtain more reliable training signals from the predictions on these translated sentences, because the model is trained with supervision from labeled source-language data and has better performance on the source language. such reliable signals can be propagated back to the unlabeled target-language sentences through regularizing the prediction consistency between aligned spans. furthermore, consistency training on the parallel unlabeled data helps to align different languages in the same representation space for better cross-lingual adaptability. to mitigate overfitting on the source language, we also introduce a dropout-based consistency training on labeled source-language data, where we train our model to be robust towards noise induced by different dropout processes. in summary, conner enhances the model's robustness towards both translation-based and dropout-based perturbations, which alleviates overfitting and achieves better cross-lingual generalization. to illustrate the effectiveness of conner, we conduct experiments on three transfer pairs and benchmark against various baseline methods. experimental results show that conner achieves substantial improvement on cross-lingual ner over the baselines. our major contributions are as follows: (1) we propose a novel consistency training framework for cross-lingual ner and achieve consistent improvement over multiple baselines across various transfer pairs. (2) we present translation-based consistency training to effectively utilize unlabeled targetlanguage data. it obviates using word-alignment tools and handles token number changes. (3) we introduce dropout-based consistency training to ner and reinforce the model's robustness towards input noise and alleviate overfitting on labeled sourcelanguage data. ## methodology in this section, we introduce conner, our consistency training framework for cross-lingual ner. as shown in fig. 1 , the overall framework consists of three components, namely (1) training with supervised cross-entropy loss, (2) dropoutbased consistency training and (3) translationbased consistency training. specifically, supervised cross-entropy loss is calculated on labeled sourcelanguage data, similar to a vanilla ner model. secondly, dropout-out based consistency training (section 2.4) feeds the same labeled source-language sample twice through the model, and enforces the model to output consistent probability distributions on the same token from two different dropout operations. lastly, translation-based consistency training (section 2.3) translates unlabeled target-language data into the source language via alignment-free translation method, and encourages the model to make consistent predictions on conjugate spans between the original and translated sentences. ## experiments in this section, we evaluate our consistency training framework conner on cross-lingual ner, and compare with various state-of-the-art models. ## analysis 4.1 ablation study to analyze the contribution of each component and justify the framework design of conner, we conduct the following ablation studies: (1) trans- as shown in moreover, if we apply the dropout-based consistency on unlabeled target-language data (dropoutunlabel), we observe an unreasonably low performance (e.g., german). we attribute this to catastrophic forgetting of multilinguality. without the guidance from the labels on target-language data, the model is likely to enter a failure mode that trivially minimizes the divergence for target-language tokens. as a result, target-language features are diverted away from the source-language in the representation space, causing the model to fail on the target language. in contrast, the translation-based consistency of conner makes use of parallel unlabeled data such that the source and target language are always aligned through the conjugate spans. ## conclusions in this work, we proposed a novel consistency framework for cross-lingual ner. our method conner enhances the model's robustness towards perturbations on labeled and unlabeled data, via translation-based and dropout-based consistency. with translation-based consistency, we tackle the challenges of word alignment and token number changes via alignment-free projection and tokenlevel to span-level conversion. compared with multiple baseline methods, conner achieves consistent performance improvements.