A dv P icker: E ffectively L everaging U nlabeled D ata via A dversarial D iscriminator for C ross- L ingual NER (2021)

Abstract: Neural methods have been shown to achieve high performance in Named Entity Recognition (NER), but rely on costly high-quality labeled data for training, which is not always available across languages. While previous works have shown that unlabeled data in a target language can be used to improve cross-lingual model performance, we propose a novel adversarial approach (AdvPicker) to better leverage such data and further improve results. We design an adversarial learning framework in which an encoder learns entity domain knowledge from labeled source-language data and better shared features are captured via adversarial training - where a discriminator selects less language-dependent target-language data via similarity to the source language. Experimental results on standard benchmark datasets well demonstrate that the proposed method benefits strongly from this data selection process and outperforms existing state-of-the-art methods; without requiring any additional external resources (e.g., gazetteers or via machine translation).

## introduction named entity recognition (ner) is a fundamental information extraction task, which seeks to identify named entities in text and classify them into predefined entity types (such as person, organization, location, etc.) and it is key in various downstream tasks, e.g., question answering @xcite . neural ner models are highly successful for languages with a large amount of quality annotated data. however, most languages don't have enough labeled data to train a fully supervised model. this motivates research on cross-lingual transfer, which leverages labeled data from a source language (e.g., 1 code is publicly available at https://aka.ms/ advpicker english) to address the lack of training data problem in a target language. in this paper, following @xcite and @xcite , we focus on zero-shot cross-lingual ner, where labeled data is not available in the target language. the state-of-the-art methods for zero-shot cross-lingual ner are mainly divided into three categories: i) feature-based methods @xcite @xcite , which train a ner model to capture language-independent features of the labeled source-language data and then apply it to the target language; ii) translation-based methods @xcite , which build pseudo target-language dataset via translating from labeled source-language data and mapping entity labels; and iii) pseudo-labeling methods, which generate pseudo-labeled data for training a target-language ner model via a source-language model @xcite or annotation projection @xcite . however, each method has its own disadvantages. feature-based methods only learn the knowledge in the source language, but cannot leverage any target-language information. translation-based methods require high-quality translation resources, which are expensive to obtain. and pseudo-labeled methods assume that all pseudo-labeled data is beneficial for cross-lingual transfer learning, which is not always the case. therefore, here we propose a novel approach -advpicker -which combines feature-based and pseudo-labeling methods, while not requiring any extra costly resources (e.g., translation models or parallel data). furthermore, to address the described problems, we enhance the source-language ner model with unlabeled target language data via adversarial training. unlike other pseudolabeling methods, we only leverage the languageindependent pseudo-labeled data selected by an adversarial discriminator, to alleviate overfitting the model in language-specific features of the sourcelanguage. specifically, we first train an encoder and a ner classifier on labeled source-language data to learn entity domain knowledge. meanwhile, a language discriminator and the encoder are trained on a token-level adversarial task which enhances the ability of the encoder to capture shared features. we then apply the encoder and the ner classifier on unlabeled target-language data to generate pseudolabels and use an adversarial discriminator to select less language-specific data samples. finally, we utilize knowledge distillation to train a targetlanguage ner model on this selected dataset. we evaluate our proposed advpicker over 3 target languages on standard benchmark datasets. our experimental results show that the proposed method benefits strongly from this data selection process and outperforms existing sota methods; without requiring any additional external resources (e.g., gazetteers or machine translation). our major contributions are as follows: to the best of our knowledge, this is the first successful attempt in selecting data by adversarial discriminator for xl-ner; • experiments on standard multi-lingual datasets showcase advpicker achieves new state-of-the-art results in cross-lingual ner. ## advpicker in this section, we introduce our approach (ad-vpicker) which utilizes the adversarial learning approach to select language-independent pseudolabeled data for training an effective targetlanguage ner model. figure 1 illustrates the framework of the proposed advpicker. specifically, as shown in figure 1 (a), we train an encoder and a ner classifier on the labeled source-language data. meanwhile, a language discriminator and the encoder are trained on the token-level adversarial task. we then apply encoder and classifier over unlabeled target-language data to generate pseudolabels and use the adversarial discriminator to select the most language-independent pseudo-labeled data samples. finally, we utilize knowledge distillation to train a target-language ner model on this selected dataset. in the following section, we describe the language-independent data selection process, including the token-level adversarial training, data selection by the discriminator, and knowledge distillation on select language-independent data. ## conclusion in this paper, we propose a novel approach to combine the feature-based method and pseudo labeling via language adversarial learning for cross-lingual ner. advpicker is the first successful attempt in selecting language-independent data by adversarial discriminator to cross-lingual ner. our experimental results show that the proposed system benefits strongly from this new data selection process and outperforms existing state-of-the-art methods, even without requiring additional extra resources.