A Label-Aware Autoregressive Framework for Cross-Domain NER (2022)

Abstract: Cross-domain named entity recognition (NER) aims to borrow the entity information from the source domain to help the entity recognition in the target domain with limited labeled data. Despite the promising performance of existing approaches, most of them focus on reducing the discrepancy of token representation between source and target domains, while the transfer of the valuable label information is often not explicitly considered or even ignored. Therefore, we propose a novel autoregressive framework to advance cross-domain NER by first enhancing the relationship between labels and tokens and then further improving the transferability of label information. Specifically, we associate each label with an embedding vector, and for each token, we utilize a bidirectional LSTM (Bi-LSTM) to encode the labels of its previous tokens for modeling internal context information and label dependence. Afterward, we propose a Bi-Attention module that merges the token representation from a pre-trained model and the label features from the Bi-LSTM as the label-aware information, which is concatenated to the token representation to facilitate cross-domain NER. In doing so, label information contained in the embedding vectors can be effectively transferred to the target domain, and Bi-LSTM can further model the label relationship among different domains by pre-train and then fine-tune setting. Experimental results on several datasets confirm the effectiveness of our model, where our model achieves significant improvements over the state of the arts.

## introduction named entity recognition (ner) is a fundamental task in natural language processing (nlp), aiming to identify salient information from raw texts, such as persons, locations, and so on. ner can be viewed as a specific sequence labeling problem, where models built upon pre-trained language models have recently achieved significant improvements. however, most conventional approaches trained on specific domains (source domains) are hard to generalize to new domains (target domains) due to the differences in text genre and limitation of labeled data. thus, cross-domain ner has been proposed for alleviating this problem, which aims to learn information from the source domain to enhance ner in the target domain. for example, @xcite utilized a parameter generation network to combine cross-domain language modeling and ner, thereby enhancing the model to extract knowledge of domain differences from raw texts. furthermore, @xcite ; @xcite proposed to continue pre-training the language models on the target domain-related corpus. despite the outstanding performance, existing approaches mainly focus on handling the text discrepancy between different domains and apply conditional random fields (crf) @xcite to capture label-label dependence in neighbor tags. several issues cannot be appropriately solved. first, most of them rely heavily on the powerful encoder to implicitly extract token-label relationships due to the limitation of the sequence labeling framework, which is insufficient, especially for the limited data in a new domain, where the encoder is hard to be fully trained. in cross-domain ner, token-label relationships are more critical since better token-label interaction can help the model distinguish the differences and similarities between the two domains. for example, in the general domain, "bayes" usually is a "person" entity, while in the artificial intelligence (ai) domain, for "supervised learning are naive bayes classifier", "bayes" is an "algorithm" entity. clearly, if a model is aware that the ne label of the previous phrase "supervised learning" is an "ai field" entity and thus pays more attention to this phrase, predicting "bayes" as an "algorithm" entity shall be an easier task. such previous label information (e.g., labels of "supervised learning") can be explicitly used to help the model enhance the relationship between tokens and labels instead of purely depending on the encoder itself. second, remote label-label relationships and label semantic information are also important for cross-domain ner instead of only modeling the interdependency among adjacent labels as crf does. for example, for "like a girl which is from the scenery and fish", "like a girl" is an "album" entity and "scenery and fish" is a "song" entity. although they are not adjacent, since entity type 'album" is semantically close to "song", they tend to exist in the same sentence. when the model has predicted an "album" entity, it will pay more attention to the "song" entity during predicting other words in the sentence, which is a type of helpful supplementary information. third, when two domains share the same ne types, these shared labels usually represent similar meanings such that they are easily adapted to the target domain, which is paid less attention in previous studies. therefore, fully using shared ne labels and further appropriately modeling the correlations between the shared labels and target domain-related ones are also beneficial to advance cross-domain ner. in this paper, we propose a novel autoregressive cross-domain ner framework to help the model facilitate domain adaptation by improving the relationship between the source text and its named entity (ne) labels and enhancing label information transfer. in detail, we associate each label with an embedding vector (randomly initialized and learned later), and for each token in the original text, we input the embeddings of the label sequence generated from previous steps into a bidirectional lstm (bi-lstm), whose hidden states model label sequence information. next, we propose a bi-attention module to perform two attention between token representations from a pre-trained model and label features from the bi-lstm to calculate label background and context information and then concatenate them as the label-aware information. we then fuse label-related knowledge into current token representation for promoting cross-domain ner. in doing so, our model can learn label embeddings and the potential relationship between tokens and labels by pre-training on the source domain, especially for shared entity labels, and then adapt them to the target domain by fine-tuning. experimental results on several datasets show that our approach outperforms existing studies. ## related work in nlp, ner aims to identify entities from unstructured text, which has been studied widely over the past decades. recently, neural networks have significantly improved the performance of ner, owing to their strong ability in feature extraction @xcite @xcite @xcite @xcite @xcite . for example, @xcite ; @xcite combined bi-lstm with a crf layer to enhance ner. @xcite ; @xcite ; @xcite further introduced the transformer-based @xcite encoders to extract more effective information from the sequence, which is then used to facilitate ner. however, although these models achieved great performance, they required largescale labeled training data to adapt to different domains. therefore, cross-domain ner has drawn substantial attention in recent years and gradually become one of the hot research topics in nlp. many approaches have been proposed to enhance cross-domain ner @xcite @xcite @xcite . for example, @xcite utilized a parameter generation network to perform cross-domain and cross-task knowledge transfer and employed multi-task learning to combine ner and language modeling tasks. furthermore, jia and zhang (2020) presented a multi-cell compositional lstm structure that incorporated the entity type by a separate cell state to enhance the cross-domain ner. compared with these studies, our model provides a simple but effective solution for addressing cross-domain ner by improving label information transfer and predicting current labels through corresponding tokens and previous labels together. ## conclusion in this paper, we have proposed a novel framework for cross-domain ner to enhance the relationship between the source text and labels and improve label information transfer, where each ne label is jointly predicted by corresponding token and previous ne labels. we not only adopt a commonlyused pre-trained model to extract token representation, but also introduce a random initialized embedding matrix and bi-lstm-based label encoder to model the label sequence generated from previous steps. after that, we construct two different attention between hidden states of bi-lstm and token representations to produce label background and context information, which are then concatenated as label-aware information and applied to predict labels. thanks to this design, the label information can be effectively transferred from the source to the target domain. comprehensive experimental results on several benchmark datasets illustrate the effectiveness of our model, which achieves significant improvements over existing methods.