Generalizing Few-Shot Named Entity Recognizers to Unseen Domains with Type-Related Features (2023)

Abstract: Few-shot named entity recognition (NER) has shown remarkable progress in identifying entities in low-resource domains. However, few-shot NER methods still struggle with out-of-domain (OOD) examples due to their reliance on manual labeling for the target domain. To address this limitation, recent studies enable generalization to an unseen target domain with only a few labeled examples using data augmentation techniques. Two important challenges remain: First, augmentation is limited to the training data, resulting in minimal overlap between the generated data and OOD examples. Second, knowledge transfer is implicit and insufficient, severely hindering model generalizability and the integration of knowledge from the source domain. In this paper, we propose a framework, prompt learning with type-related features (PLTR), to address these challenges. To identify useful knowledge in the source domain and enhance knowledge transfer, PLTR automatically extracts entity type-related features (TRFs) based on mutual information criteria. To bridge the gap between training and OOD data, PLTR generates a unique prompt for each unseen example by selecting relevant TRFs. We show that PLTR achieves significant performance improvements on in-domain and cross-domain datasets. The use of PLTR facilitates model adaptation and increases representation similarities between the source and unseen domains.

## introduction named entity recognition (ner) aims to detect named entities in natural languages, such as locations, organizations, and persons, in input text @xcite @xcite . this task has gained significant attention from both academia and industry due to its wide range of uses, such as question answering and document parsing, serving as a crucial component in natural language understanding @xcite @xcite . the availability of labeled data for ner is limited to specific domains, leading to challenges for generalizing models to new domains @xcite @xcite . to overcome this issue, recent research focuses on enabling models to effectively learn from a few labeled examples in new target domains @xcite @xcite @xcite or on exploring data augmentation techniques, leveraging automatically generated labeled examples to enrich the training data @xcite . however, these methods still require manual labeling for target domains, limiting their applicability in zero-shot scenarios with diverse domains. recently, @xcite have explored a new task, few-shot cross-domain ner, aiming to generalize an entity recognizer to unseen target domains using a small number of labeled in-domain examples. to accomplish this task, a data augmentation technique, named factmix, has been devised. factmix generates semi-fact examples by replacing the original entity or non-entity words in training instances, capturing the dependencies between entities and their surrounding context. despite its success, factmix faces two challenges: augmentation is limited to the training data. since the target domain is not accessible during training, factmix exclusively augments the training data from the source domain. as a result, there is minimal overlap between the generated examples and the test instances at both the entity and context levels. for instance, only 11.11% of the entity words appear simultaneously in both the generated data (by factmix) and the ai dataset (target domain). at the context level, as demonstrated (b) type-related features in the source domain. figure 1 : (a) average sbert similarities @xcite between pairs of sentences that contain the same type of entities. the source domain dataset is @xcite ; the target domain datasets include ai, music, and science @xcite . in the "cross-domain" setting, one sentence is from the source domain and the other is from the target domain. in the "factmix" setting, one sentence is from augmented data by factmix @xcite , and the other is from the target domain. in the "in-domain" setting, both sentences are from the target domain. (b) examples of type-related features in the source domain. in fig. 1(a) , the average sentence similarity between the augmented instances and the test examples is remarkably low. these gaps pose severe challenges in extrapolating the model to ood data. to address this problem, we incorporate natural language prompts to guide the model during both training and inference processes, mitigating the gap between the source and unseen domains. knowledge transfer is implicit and insufficient. intuitively, better generalization to unseen domains can be accomplished by incorporating knowledge from the source domain @xcite . however, in factmix, the transfer of knowledge from the source domain occurs implicitly at the representation level of pre-trained language models. factmix is unable to explicitly identify the typerelated features (trfs), i.e., tokens strongly associated with entity types, which play a crucial role in generalization. e.g., as illustrated in fig. 1(b ), the words "established" and "along with" exhibit a close relationship with organization and person entities, respectively, in both domains. this knowledge can greatly assist in recognizing organizations and persons in the target domain. to tackle this limitation, we introduce mutual information criteria to extract informative trfs from the source domain. furthermore, we construct a unique prompt for each unseen instance by selecting relevant trfs. intuitively, these generated prompts serve as distinctive signatures, linking unfamiliar examples to the knowledge within the source domain. contributions. in this paper, we present a framework, named prompt learning with type-related features (pltr) for few-shot cross-domain ner, to effectively leverage knowledge from the source domain and bridge the gap between training and unseen data. as fig. 2 shows, pltr is composed of two main phases: (i) type-related feature extraction, and (ii) prompt generation and incorporation. to identify valuable knowledge in the source domain, pltr uses mutual information criteria to extract entity type-related features (trfs). pltr implements a two-stage framework to mitigate the gap between training and ood data. firstly, given a new example, pltr constructs a unique sequence by selecting relevant trfs from the source domain. then, the constructed sequences serve as prompts for performing entity recognition on the unseen data. finally, a multi-task training strategy is employed to enable parameter sharing between the prompt generation and entity recognition. similar to factmix, pltr is a fully automatic method that does not rely on external data or human interventions. pltr is able to seamlessly integrate with different few-shot ner methods, including standard fine-tuning and prompt-tuning approaches. in summary, our contributions are: (i) to the best of our knowledge, ours is the first work to study prompt learning for few-shot cross-domain ner; (ii) we develop a mutual information-based approach to identify important entity type-related features from the source domain; (iii) we design a two-stage scheme that generates and incorporates a prompt that is highly relevant to the source domain for each new example, effectively mitigating the gap between source and unseen domains; and (iv) experimental results show that our proposed pltr achieves state-of-the-art performance on both in-domain and cross-domain datasets. cross-domain ner. the task of cross-domain ner aims to transfer ner models across diverse text styles @xcite @xcite @xcite @xcite @xcite @xcite . yang et al. (2017) train ner models jointly in the source and target domains, while jia et al. (2019) and jia and zhang (2020) leverage language models for cross-domain knowledge transfer. zhang et al. (2022) introduce a modular learning approach that decomposes ner into entity span detection and type classification subtasks. however, these methods still rely on ner annotations or raw data in the target domain. few-shot ner and prompt-based learning. fewshot named entity recognition (ner) is the task of identifying predefined named entities using only a small number of labeled examples (wiseman and stratos, 2019; yang and katiyar, 2020; das et al., 2022; zeng et al., 2020; ma et al., 2023). various approaches have been proposed to address this task. for instance, huang et al. (2021) investigate the effectiveness of self-training methods on external data using distance-based approaches, where the label of the nearest neighbors is copied. zeng et al. (2020) involves generating counterfactual examples through interventions to augment the original dataset. additionally, prompt-based learning, which has gained prominence in natural the field of few-shot cross-domain learning is inspired by the rapid learning capability of humans to recognize object categories with limited examples, known as rationale-based learning @xcite @xcite @xcite . in the context of ner, @xcite introduce the fewshot cross-domain setting and propose a two-step rationale-centric data augmentation method, named factmix, to enhance the model's generalization ability. in this paper, we focus on few-shot cross-domain ner. the most closely related work is fact-mix @xcite . factmix faces two challenging problems: (i) augmentation is limited to the training data, and (ii) the transfer of knowledge from the source domain is implicit and insufficient. in our proposed pltr, to identify useful knowledge in the source domain, mutual information criteria are designed for automatic type-related feature (trf) extraction. in addition, pltr generates a unique prompt for each unseen example based on relevant trfs, aiming to reduce the gap between the source and unseen domains. ## method in this section, we present the two primary phases of the proposed pltr method, as depicted in ## experiments we aim to answer the following research questions: (rq1) does pltr outperform state-of-the-art finetuning methods on the few-shot cross-domain ner task? (sec. 6.1) (rq2) can pltr be applied to prompt-tuning ner methods? (sec. 6.2) micro f1 is adopted as the evaluation metric for all settings. ## experimental results to answer rq1 and rq2, we assess the performance of pltr on both in-domain and crossdomain few-shot ner tasks. this evaluation is conducted in two settings: a fine-tuning setting with 100 training instances per type, and a prompttuning setting with 5 training instances per type. ## analysis now that we have answered our research questions, we take a closer look at pltr to analyze its performance. we examine whether the prompts are designed appropriately. besides, we study how the number of training samples and selected type-related features influence the performance (sec. a.2), how pltr affects representation similarities between the source and target domains, and whether pltr is extensible to different source domains and entity types (sec. a.3). furthermore, we provide insights into the possible factors that limit further improvements. ablation studies. are removed during test-time inference. in this case, the f1 scores across all datasets and settings suffer a significant drop compared to our proposed pltr. this demonstrates the crucial role of incorporating prompts during both the training and inference processes. in the "rdw" and "rew" variants, prompts are constructed using randomly selected words from the source domain and the given example, respectively. the performance of both the "rdw" and "rew" model variants consistently falls short of pltr, indicating that pltr effectively identifies important knowledge from the source domain and establishes connections between unseen examples and the knowledge within the source domain. additionally, to explore the efficacy of typerelated feature selection (refer to sec. 4.2), we conducted an evaluation of pltr (bert-base) using various frequency ratios ρ (in eq. 4). the results are presented in the influence of training samples. to examine the impact of the number of training samples, we compare the performance of pltr and factmix on few-shot cross-domain ner using 100, 300, and 500 training samples per entity type. fig. 3 displays the results based on the bert-base-cased model. pltr exhibits the largest improvements over factmix when the dataset comprises only 100 training instances per entity type, as opposed to the 300 and 500 training instances scenarios. furthermore, pltr consistently outperforms the prior state-of-the-art approach, factmix, across all experimental settings with varying numbers of training examples, demonstrating its superiority. analysis of sentence similarities. in our analysis of sentence similarities, we investigate the impact of pltr on the representation similarities between the source and target domains. we compute the average sbert similarities for sentence representations in pltr (bert-base) between the source and target domains; the results are presented in fig. 4 . with the prompts generated by pltr, the representation similarities between the source and unseen domains noticeably increase. this is, pltr facilitates a more aligned and connected representation space, mitigating the gap between the source and target domains. ## conclusions in this paper, we establish a new state-of-the-art framework, pltr, for few-shot cross-domain ner. to capture useful knowledge from the source domain, pltr employs mutual information criteria to extract type-related features. pltr automatically selects pertinent features and generates a unique prompt for each unseen example, bridging the gap between domains. experimental results show that pltr not only effectively generalizes standard fine-tuning methods to unseen domains, but also demonstrates promising performance when incorporated with prompt-tuning-based approaches. additionally, pltr substantially narrows the disparity between in-domain examples and ood instances, enhancing the similarities of their sentence representations.