PRAM : An End-to-end Prototype-based Representation Alignment Model for Zero-resource Cross-lingual Named Entity Recognition (2023)

Abstract: Zero-resource cross-lingual named entity recognition (ZRCL-NER) aims to leverage rich labeled source language data to address the NER problem in the zero-resource target language. Existing methods are built either based on data transfer or representation transfer. However, the former usually leads to additional computation costs, and the latter lacks explicit optimization specific to the NER task. To overcome the above limitations, we propose a novel prototype-based representation alignment model (PRAM) for the challenging ZRCL-NER task. PRAM models the cross-lingual (CL) NER task and transfers knowledge from source languages to target languages in a unified neural network, and performs end-to-end training, avoiding additional computation costs. Moreover, PRAM borrows the CL inference ability of multilingual language models and enhances it with a novel training objective—attribution-prediction consistency (APC)—for explicitly enforcing the entity-level alignment between entity representations and predictions, as well as that across languages using prototypes as bridges. The experimental results show that PRAM significantly outperforms existing state-of-the-art methods, especially in some challenging scenarios.

## introduction named entity recognition (ner) aims to identify the boundaries and categories of entities in a chunk of text @xcite . automatic ner is useful for various downstream applications, such as search engines @xcite , dialogue systems @xcite , and knowledge graphs @xcite . most of the recent advances in ner are achieved by deep neural networks that are trained on a large amount of labeled data @xcite chiu and nichols, * work done during an internship at tencent. † equal contribution ‡ corresponding author 2016; @xcite . however, it is not universal that every kind of language has sufficient labeled data for training a deep ner model. this motivates research on a new challenging task named zero-resource cross-lingual ner (zrcl-ner), which aims to leverage a rich labeled source language to address the ner problem in an unlabeled target language. most advanced methods for the challenging task have concentrated on transferring knowledge from the rich-resource language (rrl) to the zeroresource language (zrl). according to the type of transferred knowledge, these methods can be divided into two categories, i.e., data transfer based methods and representation transfer based methods. the data transfer based methods transfer the knowledge from rrl to the zrl by generating pseudo-labeled data @xcite @xcite or soft labels for unlabeled data @xcite @xcite in the zrl via translation or teacher models, respectively. however, they follow a two-separated training setup that needs extra translation or teacher models, introducing additional computation costs. in contrast, the representation transfer based methods directly model the cross-lingual ner in a unified model, borrowing the cross-lingual inference ability of the multilingual pre-trained language models (mplms) @xcite @xcite . to further enhance the crosslingual inference ability, they enforce the mplms to learn language-independent features via representation alignment w.r.t. token @xcite or cross-lingual antagonistic training w.r.t. language @xcite . these optimizations are implicit for the ner task and thus show limited improvements on the zrcl-ner. to overcome the above limitations, we propose a novel prototype-based representation alignment model (pram) for zrcl-ner. our pram builds upon the mplm that falls into the representation transfer based framework, and performs end-to-end training that avoids tedious training stages. this prevents excess computational costs that is not necessary for improving a model's performance on the target language. we explicitly enforce entity-level alignment between representations and predictions in both source and target languages by a new training objective called attribution-prediction consistency (apc). by treating prototypes as category anchors, we can enforce the alignment of similar entity representations across languages by guiding similarity distributions of entity representations relative to prototypes through predictions. specifically, we treat the centroid of entity representations belonging to the same class in the source language as a prototype. the apc minimizes js divergence between predicted probability distribution (model output for each entity mention) and similarity distribution (similarities between each mention representation and all prototypes). in this way, with the predicted probability distribution becoming more discriminative in the training, the apc can impose the alignment between the entity representation and its corresponding prototype. the alignment property in the representation space, in turn, can regularize the deviation of model predictions. probability distribution changes in source languages lead to similar changes in target languages due to mplm's cross-lingual inference ability. thus, the similar entity representations of the source and target languages are clustered with prototypes and this alignment can enhance cross-lingual inference. to explore the performance of pram, we conduct experiments on three ner datasets under both single-and multi-source settings. the experimental results show that pram outperforms all state-of-the-arts (sotas), especially in some more challenging scenarios. specifically, in the setting of single source transfer on the wikiann dataset, pram achieves 3.92% absolute improvements on f1-score compared with sota. on the other hand, pram achieves an average improvement of 3.88% on the f1 score compared with sota in the multisource setting. moreover, we demonstrate the apc is able to impose entity-level alignment across languages by visualizing entity representations of the same class from different languages. the contribution of this work is three-fold: ## methodology this section introduces our prototype-based representation alignment model (pram), as illustrated in figure 1 . first, the zrcl-ner task is described formally. then, we introduce the basic cross-lingual ner model, which we need to remold. subsequently, our apc module is elaborated. finally, we describe how to train pram in singleand multi-source settings. ## experiment we evaluate pram in both single-source and multisource transfer settings, and compare it with stateof-the-art models. moreover, an ablation study and several analytical experiments are conducted to demonstrate the effectiveness of our model. ## conclusion this paper proposes a novel and effective prototypebased representation alignment model (pram) for zrcl-ner. a novel training objective named apc is proposed to cooperate with the crosslingual inference ability of mplm, which can enhance the alignment of entity representations and predictions, as well as the representations of homogeneous entities across languages. the experimental results show that pram achieves excellent performance in both single-source and multi-source transfer settings. last but not least, the training of pram is performed end-to-end and only additionally utilizes unlabeled target data.