Type-Aware Decomposed Framework for Few-Shot Named Entity Recognition (2023)

Abstract: Despite the recent success achieved by several two-stage prototypical networks in few-shot named entity recognition (NER) task, the over-detected false spans at span detection stage and the inaccurate and unstable prototypes at type classification stage remain to be challenging problems. In this paper, we propose a novel Type-Aware Decomposed framework, namely TadNER, to solve these problems. We first present a type-aware span filtering strategy to filter out false spans by removing those semantically far away from type names. We then present a type-aware contrastive learning strategy to construct more accurate and stable prototypes by jointly exploiting support samples and type names as references. Extensive experiments on various benchmarks prove that our proposed TadNER framework yields a new state-of-the-art performance.

## introduction named entity recognition (ner) aims to detect entity spans and classify them into pre-defined categories (entity types). when there are sufficient labeled data, deep learning-based methods @xcite @xcite can get impressive performance. in real applications, it is desirable to recognize new categories which are unseen in training/source domain. however, collecting extra labeled data for these new types will be surely timeconsuming and labour-expensive. consequently, few-shot ner @xcite , which involves identifying unseen entity types based on a few labeled samples for each class (i.e., support samples) in test domain, has attracted great research interests in recent years. end-to-end metric learning based methods @xcite are the mainstream in few-shot ner. these methods need to simultaneously learn the complex structure consisting of entity boundary and entity type. when the domain gap is large, their performance will drop dramatically because it is extremely hard to capture such complicated structure information with only a few support examples for domain adaptation. this leads to the insufficient learning of boundary information, resulting that these methods often misclassify entity boundaries and cannot obtain very satisfying performance. recently, there is an emerging trend in adopting two-stage prototypical networks @xcite for few-shot ner, which decompose ner into two separate span extraction and type classification tasks and perform one task at each stage. since decomposed methods only need to handle one single boundary detection task at the first stage, they can find more accurate boundaries and obtain better performance than end-to-end approaches. while making good progress, these two-stage prototypical networks still face two challenging problems, i.e., the over-detected false spans and the inaccurate and unstable prototypes in corresponding stages. (1) at the span extraction stage in test phase, the decomposed approaches usually recall many over-detected false spans whose types only exist in the source domain. for example, "1976" in figure 1 (a) belongs to a date type in the source domain since there are many samples like "obama was born in 1961" in training, and thus it is easily recognized as a span by the span detector. however, there is no such label in the test domain and "1976" is thus assigned a false loc type. (2) the prototypical networks in decomposed methods target at learning a type-agnostic metric similarity function to classify entities in test samples (i.e., query samples) via their distance to prototypes. since the prototypes are constructed using very few support samples in the type-agnostic feature space, they might be inaccurate and unstable. for example, in figure 1 (c), a prototype is just the support sample in one-shot ner and thus deviates far away from the real class center. based on the above observations, we propose a type-aware decomposed framework, namely tadner, for few-shot ner. our method follows the span detection and type classification learning scheme in the decomposed framework but moves two steps further to overcome the aforementioned issues. firstly, we present a type-aware span filtering strategy to filter out false spans by removing those semantically far away from type names foot_0 . by this means, the over-detected spans like "1976" whose types do not exist in test domain can be removed due to the long semantic distance to type names, as shown in figure 1 (b) . secondly, we present a type-aware contrastive learning strategy to construct more accurate and stable prototypes by jointly leveraging type names and support samples as references. through this way, the type names can serve as the guidance for prototypes and make them not deviate too far away from the class centers even in some extreme outlier cases, as shown in figure 1 (d ). extensive experimental results on 5 benchmark datasets demonstrate the superiority of our tadner over the state-of-the-art decomposed methods. in particular, in the hard intra few-nerd and 1-shot domain transfer settings, tadner achieves a 8% and 9% absolute f1 increase, respectively. ## method in this section, we formally present our proposed tadner. the overall structure of our tadner is shown in figure 2 . note that the type-aware contrastive learning and type-aware span filtering strategies take effect at the type classification stage in the training and test domain, respectively. task formulation given a sequence @xmath0, x 2 , ..., x n } with n tokens, ner aims to assign each token x i a corresponding label y @xmath1, where t is the entity type set and o denotes the non-entity label. for few-shot ner, a model is trained in a source domain dataset d source with the entity type set the model is then fine-tuned in a test/target domain dataset d target with the entity type set t @xmath2, t 2 , ...t n } using a given support set s target . the entity token set and corresponding label set in s target are denoted as e @xmath3, e s 2 , ..., e s m } and y @xmath4, y s 2 , ..., y s m }, where y s @xmath5. the model is supposed to recognize entities in the query set q target of the target domain. besides, t source and t target have no or very little overlap, making few-shot ner very challenging. more specifically, in the n-way k-shot setting, there are n labels in t target and k examples associated with each label in the support set s target . ## related work few-shot ner few-shot ner methods can be categorized into two types: prompt-based and metric-based. prompt-based methods focus on leveraging pre-trained language model knowledge for ner through prompt learning @xcite @xcite . they rely on templates, prompts, or good examples to utilize the pre-trained knowledge effectively. metric-based methods aim to learn a feature space with good generalizability and classify test samples using nearest class prototypes @xcite @xcite or neighbor samples @xcite . there are also some efforts to improve few-shot ner by incorporating type name (label) semantics @xcite . these methods usually treat labels as class representatives and align tokens with them, yet neglecting the joint training of entity words and label representations. hence they can only use either support sets or labels as class references. instead, our method exploits support samples and type names simultaneously, which helps construct more accurate and stable prototypes in the target domain. task decomposition and contrastive learning recently, decomposed-based methods have emerged as effective solutions for the ner problem @xcite @xcite @xcite . these methods can learn entity boundary information well in data-limited scenarios and often get better results. however, the widely used prototypical networks in these methods may encounter inaccurate and unstable prototypes given limited support samples at the type classification stage. besides, they may face the problem of over-detected false spans produced at the span detection stage. our method can address these two issues via the proposed type-aware contrastive learning and typeaware span filtering strategies. our method is also inspired by contrastive learning @xcite . due to its good generalization performance, two recent methods @xcite borrow this idea for few-shot ner, which construct contrastive loss between tokens or between the token and the prompt. however, they are both the end-to-end approach and thus have the inherent drawback that cannot learn good entity boundary information. in contrast, our method is a decomposed one and our contrastive loss is constructed between tokens with additional type name information, which can find accurate boundary and learn a type-aware feature space. ## conclusion in this paper, we propose a novel tadner framework for few-shot ner, which handles the span detection and type classification sub-tasks at two stages. for type classification, we present a typeaware contrastive learning strategy to learn a typeaware and generalized feature space, enabling the model to construct more accurate and stable prototypes with the help of type names. based on it, we introduce a type-aware span filtering strategy for removing over-detected false spans produced at the span detection stage. extensive experiments demonstrate that our method achieves superior performance over previous sota methods, especially in the challenging scenarios. in the future, we will try to extend tadner to other nlp tasks.