C o L a D a: A Collaborative Label Denoising Framework for Cross-lingual Named Entity Recognition (2023)

Abstract: Cross-lingual named entity recognition (NER) aims to train an NER system that generalizes well to a target language by leveraging labeled data in a given source language. Previous work alleviates the data scarcity problem by translating source-language labeled data or performing knowledge distillation on target-language unlabeled data. However, these methods may suffer from label noise due to the automatic labeling process. In this paper, we propose CoLaDa, a Collaborative Label Denoising Framework, to address this problem. Specifically, we first explore a model-collaboration-based denoising scheme that enables models trained on different data sources to collaboratively denoise pseudo labels used by each other. We then present an instance-collaboration-based strategy that considers the label consistency of each token’s neighborhood in the representation space for denoising. Experiments on different benchmark datasets show that the proposed CoLaDa achieves superior results compared to previous methods, especially when generalizing to distant languages.

## introduction the named entity recognition (ner) task aims to locate and classify entity spans in a given text into predefined entity types. it is widely used for many downstream applications, such as relation extraction and question answering. deep neural networks have made significant progress on this task leveraging large-scale human-annotated data for training. however, fine-grained token-level annotation makes it costly to collect enough high-quality labeled data, especially for low-resource languages. such scenarios motivate the research on zero-shot cross-lingual ner, which attempts to leverage labeled data in a rich-resource source language to solve the ner task in a target language without annotated data. recent attempts at cross-lingual ner can be roughly categorized from two aspects: learning language-independent features via feature alignment @xcite and learning language-specific features from automatically labeled target-language data @xcite . despite bringing great success to crosslingual ner, the former line of research misses exploiting language-specific features and thus shows substandard performance, especially when transferring to distant languages, e.g., from english to arabic @xcite . hence, a series of studies focuses on the latter category, which typically creates pseudo-labeled target-language data and uses it to perform conventional supervised learning or teacher-student learning. for example, as shown in fig 1 @xcite , earlier studies @xcite @xcite , such as tmp @xcite , first translate labeled data in the source language and then perform label projection. recently, several approaches have utilized a weak model, which could be an ner model either trained on the source language's labeled data as in tsl @xcite , or further finetuned on the generated translation data as in unitrans @xcite , to annotate the unlabeled target-language data for improvement, as shown in fig 1 @xcite and fig 1(c ). unfortunately, these methods inevitably suffer from the label noise induced by inaccurate translation and label projection, or the weak model's limited capability. although some methods are proposed to mitigate the label noise problem by additionally training an instance selector @xcite or designing heuristic rules for data selection @xcite , they independently manipulate either the translation data (d trans ) @xcite or the target-language data (d tgt ) pseudo-labeled by ner models trained in the source language @xcite . hence, all these methods ignore the complementary characteristics between both for denoising. particularly, from the text view, d tgt is collected from a natural text distribution of the target-language data, while d trans can be regarded as a way of data augmentation to provide more lexicon variants. from the labeling function view, labels of d trans are obtained via the label projection algorithm, which have little association with those of d tgt generated by ner models. with such consideration, we propose a modelcollaboration-based denoising scheme, which incorporates models trained on both data sources to mutually denoise the pseudo-labels of both data sources in an iterative way. as shown in fig 1(d ), we first leverage m tgt trained on the pseudolabeled target-language data d tgt to denoise the translation data annotated by label projection. in this way, the learned model m trans will be less affected by noise in the translation data. we then employ the improved m trans to re-label the targetlanguage unlabeled data d tgt . it is expected that there is less noise in the relabeled data, and thus we can produce a more powerful m tgt . we perform this procedure for several iterations, so that all the involved data sources and models can be improved in an upward spiral. moreover, borrowing the idea from anomaly detection @xcite that a given data point's neighborhood information can be used to measure its anomalism, here we find that the similar tokens in the feature space can also collaborate for denoising. previous studies @xcite have shown that instances with the same label are more likely to locate close to each other in the representation space. our intuition is that, if a token's label conflicts a lot with labels of other tokens in its neighborhood, then this label is probably noisy. therefore, we further propose an instancecollaboration-based denoising strategy to explore the neighborhood structure of each token for denoising, as shown in figure 2 . specifically, we utilize the label consistency of each token's neighborhood in the representation space to re-weight the soft-labeled examples in knowledge distillation. we integrate the instance-collaboration-based denoising strategy into the model-collaboration-based denoising scheme and propose a collaborative label denoising framework, i.e., colada, for cross-lingual ner. we conduct extensive experiments on two popular benchmarks covering six languages for evaluation. experimental results show that our method outperforms existing state-of-theart methods. qualitative and quantitative analyses further demonstrate the effectiveness of our framework in reducing the data noise. ## problem formulation here we take the typical sequence labeling formulation for the named entity recognition task. given a sequence with l tokens @xmath0, . . . , x l ) as the input text, an ner system is excepted to assign each token x i with a label y i . in this paper, we assume to have the labeled training data d @xmath1, y s )} in the source language, the unlabeled data d @xmath2, and translation data d @xmath3, y t )} obtained by data projection from d src . our goal is to train an ner model m that can generalize well to the target language utilizing these resources. instance collaboration {𝒙 𝑡 , 𝒚 𝑡 } ℳ 𝑡𝑔𝑡 (𝑖-1) {𝒙 𝑡 , 𝒑 𝑢 } pseudo-labeled translation data instance collaboration 𝑥 𝑖 𝑡 𝒉 ො 𝑦 𝑖 𝑝 𝑖 𝑢 ℳ 𝑡𝑟𝑎𝑛𝑠 (𝑖) 𝑤 𝑝 𝑖 𝑢 , 𝑥 𝑖 𝑡 to step 2 translation data 𝓓 𝒕𝒓𝒂𝒏𝒔 𝛽 1 step 1 𝒊-th iteration {𝒙 𝑢 } ℳ 𝑡𝑟𝑎𝑛𝑠 (𝑖) {𝒙 𝑢 , 𝒑 𝑡 } pseudo-labeled target-language data 𝑥 𝑖 𝑢 𝒉 ො 𝑦 𝑖 𝑝 𝑖 𝑡 ℳ 𝑡𝑔𝑡 (𝑖) 𝑤 𝑝 𝑖 𝑡 , 𝑥 𝑖 𝑢 to 𝒊+1 iteration unlabeled target-language data 𝓓 𝒕𝒈𝒕 step 2 𝛽 2 weighted sum of loss training signal ## colada framework figure 3 depicts an overview of the colada framework. it is an iterative model-collaboration-based denoising framework which consists of two steps: noise-robust learning on translation data and noiserobust learning on unlabeled target-language data. an instance-collaboration-based denoising strategy (sec 3.1) is then integrated into the modelcollaboration-based denoising procedure (sec 3.2). ## analysis 5.1 ablation study to further validate the effectiveness of each mechanism in the proposed framework, we introduce the following variants of colada in an ablation study: 1) colada w/o instance collaboration, where we directly set the reliability score in eq. ( 3 ) to 1 for all tokens. 2) colada w/o translation data denoise, where we set β 1 in eq. ( 4 ) to 0. 3) colada w/o iteratively denoise, where we remove the iterative enhancement and only conduct the denoising process for one iteration. 4) colada w/o model collaboration, where we set β 1 in eq. ( 4 ) to 0, remove the iteration mechanism, and directly take the model finetuned on d trans as the teacher model to train a student model with instance-collaborationbased denoising on d tgt . 5) colada w/o instance & model collaboration, which further drops the instance-collaboration-based denoising from 4). ## conclusion to address the problem of label noise in crosslingual ner, this paper presents colada, a collaborative label denoising framework. we propose a model-collaboration-based denoising scheme to make two models trained on different data sources to denoise the labels of each other and hence promote each other's learning. we further propose an instance-collaboration-based strategy that collaboratively considers the label consistency among similar tokens in the feature space to re-weight the noisy labels assigned by a teacher model in knowledge distillation. by integrating the instancecollaboration strategy into the model-collaboration denoising scheme, our final framework colada achieves superior performance over prior start-ofthe-art methods by benefiting from better handling the data noise.