Zero-Resource Cross-Domain Named Entity Recognition (2020)

Abstract: Existing models for cross-domain named entity recognition (NER) rely on numerous unlabeled corpus or labeled NER training data in target domains. However, collecting data for low-resource target domains is not only expensive but also time-consuming. Hence, we propose a cross-domain NER model that does not use any external resources. We first introduce a Multi-Task Learning (MTL) by adding a new objective function to detect whether tokens are named entities or not. We then introduce a framework called Mixture of Entity Experts (MoEE) to improve the robustness for zero-resource domain adaptation. Finally, experimental results show that our model outperforms strong unsupervised cross-domain sequence labeling models, and the performance of our model is close to that of the state-of-the-art model which leverages extensive resources.

## introduction named entity recognition (ner) is a fundamental task in text understanding and information extraction. recently, supervised learning approaches have shown their effectiveness in detecting named entities @xcite @xcite . however, there is a vast performance drop for low-resource target domains when massive training data are absent. to solve this data scarcity issue, a straightforward idea is to utilize the ner knowledge learned from highresource domains and then adapt it to low-resource domains, which is called cross-domain ner. due to the large variances in entity names across different domains, cross-domain ner has thus far been a challenging task. most existing methods consider a supervised setting, leveraging labeled ner data for both the source and target domains @xcite . however, labeled data in target domains is not always available. unsupervised domain adaptation naturally arises as a possible way to circumvent the usage of labeled ner data in target domains. however, the only existing method, proposed by @xcite , requires an external unlabeled data corpus in both the source and target domains to conduct the unsupervised cross-domain ner task, and such resources are difficult to obtain, especially for low-resource target domains. therefore, we consider unsupervised zero-resource cross-domain adaptation for ner which only utilizes the ner training samples in a single source domain. to meet the challenge of zero-resource crossdomain adaptation, we first propose to conduct multi-task learning (mtl) by adding an objective function to detect whether tokens are named entities or not. this objective function helps the model to learn general representations of named entities and to distinguish named entities from sequences in target domains. in addition, we observe that in many cases, different entity categories could have a similar or the same context. for example, in the sentence "arafat subsequently cancelled a meeting between israeli and plo officials," the person entity "arafat", can be replaced with an organization entity within the same context, which illustrates the confusion among different entity categories and makes zero-resource adaptation much more difficult. intuitively, when the entity type of a token is hard to be predicted based on the token itself and the token's context, we want to borrow the opinions (i.e., representations) from different experts. hence, we propose a mixture of entity experts (moee) framework to tackle the confusion of entity categories, and the predictions are based on the tokens and the context, as well as all entity experts. experimental results show that our model is able to outperform current strong unsupervised crossdomain sequence tagging approaches, and reach comparable results to the state-of-the-art unsupervised method that utilizes extensive resources. text sequence ...... w 1 w 2 w n bilstm conditional random field ...... sequence tagging task1 mixture of entity experts conditional random field ...... sequence tagging task2 text sequence ... ... per expert loc expert non-entity expert ... expert gate ... gate label meta-expert feature 2 related work most of the existing work on cross-domain ner has been to investigate the supervised setting, where both source and target domains have labeled data @xcite @xcite . yang et al. (2017) jointly trained models on the source and target domain with shared parameters. lin and lu (2018) added adaptation layers on top of existing models, and wang et al. (2018) introduced label-aware feature representations for ner adaptation. lee et al. (2018) utilized the idea of transfer learning by first initializing a target model with parameters learned from source-domain ner, and then using labeled target domain data to fine-tune the model. however, no prior work has focused on the unsupervised setting of cross-domain ner, except for jia @xcite . in @xcite , however, external unlabeled data corpora resources in both the source and target domains are required to train language models for domain adaptations. this limitation has motivated us to develop a model that doesn't need any external resources. tackling the low-resource scenario where there are zero or minimal existing resources has always been an interesting yet challenging task @xcite @xcite @xcite . instead of utilizing large amounts of bilingual resources, @xcite only utilized a few word pairs for zero-shot cross-lingual dialogue systems. unsupervised machine translation approaches @xcite have also been introduced to circumvents the need of parallel data. winata et al. (2020) introduced the cross-accent speech recognition task and utilized meta-learning to cope with the data scarcity issue in target accents. @xcite and @xcite proposed to do cross-domain slot filling with minimal resources. to the best of our knowledge, we are the first to propose methods on cross-domain adaptation for ner with zero external resources. ## methodology as illustrated in fig. 1 , our model combines a bidirectional lstm and conditional random field (crf) into a bilstm-crf structure @xcite with mtl and moee modules. the parameters of bilstm are shared in the multi-task learning. ## conclusion in this paper, we propose a zero-resource crossdomain framework for the named entity recognition task, which consists of multi-task learning and mixture of entity experts modules. the former learns the general representations of named entities to cope with the model's inability to recognize named entities, while the latter learns to combine the representations of different entity experts, which are based on the bilstm hidden states. experimental results show that our model outperforms strong cross-domain sequence tagging models, and the performance is close to that of the state-of-the-art model that utilizes extensive resources.