Cross-domain Named Entity Recognition via Graph Matching (2022)

Abstract: Cross-domain NER is a practical yet challenging problem since the data scarcity in the real-world scenario. A common practice is first to learn a NER model in a rich-resource general domain and then adapt the model to specific domains. Due to the mismatch problem between entity types across domains, the wide knowledge in the general domain can not effectively transfer to the target domain NER model. To this end, we model the label relationship as a probability distribution and construct label graphs in both source and target label spaces. To enhance the contextual representation with label structures, we fuse the label graph into the word embedding output by BERT. By representing label relationships as graphs, we formulate cross-domain NER as a graph matching problem. Furthermore, the proposed method has good applicability with pre-training methods and is potentially capable of other cross-domain prediction tasks. Empirical results on four datasets show that our method outperforms a series of transfer learning, multi-task learning, and few-shot learning methods.

## introduction named entity recognition (ner) is a crucial component in many language understanding tasks @xcite and is often applied in various domains. due to the data scarcity in the real-world scenario, obtaining adequate domain-specific data is usually expensive and time-consuming. hence, cross-domain ner, which is capable of adapting ner models to specific domains with limited data, has been drawing increasing attention in recent years. however, one of the primary challenges of crossdomain ner is the mismatch between source and target domain labels @xcite . for example, the label sets between atis @xcite and conll 2003 are non-overlapping. figure 1 : a demonstration of graph matching. in both two cases, our model learns graph structures from the source label space and makes correct predictions. in two label spaces, each node is a target label and the matching nodes and edges are opaque. to address this issue, some approaches utilize multi-task learning @xcite for transferring knowledge across domains. however, these methods require full training on both source and target domain data when adapting to each new domain. since the source domain dataset is usually much larger than the target domain dataset, the multi-task learning methods are inefficient when adapting to low-resource domains. recently, as pre-trained language models (plms) such as bert @xcite have shown remarkable success in ner, transferlearning-based methods also show effectiveness for cross-domain ner. a typical approach is to first train a ner model initialized with plm on rich-resource domain (e.g., @xcite ), and then fine-tune the entire model with a new task-specific linear classifier (pretrain fine-tune) @xcite . despite its simplicity, this approach provides strong results on several benchmarks @xcite , and we serve it as the baseline in our research. inspired by the idea in @xcite , where labels across domains are connected by probability distributions, we propose a novel approach, label structure transfer for cross-domain ner (lst-ner) to address the label mismatch problem. by modeling the label relationships as label graphs, we transfer the label structure from the source model (i.e., the ner model trained on source domain) to the target model (i.e., fine-tuned model). we are the first to capture label graph structures for cross-domain ner to our best knowledge. in this study, we focus on enhancing cross-domain ability based on pretrain-finetune training paradigm, with only target domain labeled data for domain adaptation. therefore, pre-training @xcite and self-training @xcite based methods, which leverages massive unlabeled data, are not considered. to explicitly capture the connections between two domains labels, we construct a label graph by probability distributions of target labels estimated by the source ner model. in the label graph, graph nodes refer to target labels, and edges refer to the relationships between labels. we represent each node as the probability distribution and add an edge between two nodes if the labels have similar distributions. by representing label relationships as label graphs in both source and target label spaces, the label knowledge can be transferred via graph matching. we introduce gromov-wasserstein distance (gwd) for aligning two label graphs because of its capability of capturing edge similarity. we show an example in fig 1 to demonstrate how graph matching works. in the example, "acl" is a "conference" named entity in the target domain. when label sets between source and target domains match perfectly, the source ner model naturally predicts "conference" with the highest probability. then, the target model straightforward learns this property from the source domain. when two label sets are mismatching, the source ner model may predict "acl" as an "organization" since the label "organization" is seen in the source domain. by score distributions of "iccl" and "david" in the source domain, we can model their relationships with "acl" as graph structures. then, the target model learns label structures via graph matching and predicts "acl" as "conference" correctly. in this way, the label relationships can be learned even when two domain label sets are different. furthermore, we enhance the contextual representation by fusing the constructed label graph into the word embedding by graph convolutional net-work (gcn), where an auxiliary task is introduced for better extracting label-specific components for each entity type. we performed extensive experiments on eight different domains in both rich-and low-resource settings. empirical results show that our method outperforms a series of competitive baselines. another line of research is based on transfer learning. except from the pretrain-finetune paradigm, some approaches proposed adding adaption layers lin and lu (2018) or adapter modules @xcite to the backbone network. compared with them, our method constructs label graphs dynamically and performs label semantic fusion via attention mechanism, and thus has fewer parameters for training. besides, our method is built on word contextual embedding by plm. therefore, our model can combine with various backbone networks and thus has better applicability. few-shot ner. few-shot ner aims at recognizing new categories in a highly low-resource scenario @xcite , which also shows good cross-domain ability. @xcite induced different undefined classes from the "others" class to alleviate the over-fitting problem. yang and katiyar (2020) proposed nnshot and structshot based on the nearest neighbor classifier, and structshot further applies the viterbi algorithm when decoding. the few-shot learning methods focus on building models that can generalize from very few examples. unlike these methods, our approach aims to enhance domain adaptation ability in both lowresource and rich-resource scenarios. ## conclusion this paper proposes a novel and lightweight transfer learning approach for cross-domain ner. our proposed method learns graph structure via matching label graphs from source to target domain. through extensive experiments, we demonstrated the effectiveness of our approach, reporting better results over a series of transfer learning, multi-task learning, few-shot learning methods. in conclusion, our approach is general, which can be combined with domain-adaptive pre-training and potentially applied to other cross-domain prediction tasks. besides, there are some limitations of our approach. for example, when the target domain entity types are fine-grained and largely different from the source domain entity types (e.g., in atis dataset), our approach shows limited improvement on the pretrain-finetune paradigm. to this end, future directions include investigations on employing multi-task learning for modeling the semantic discrepancy of labels across domains and fusing hierarchical label relationships into the label graphs.