Few-shot Named Entity Recognition with Entity-level Prototypical Network Enhanced by Dispersedly Distributed Prototypes (2022)

Abstract: Few-shot named entity recognition (NER) enables us to build a NER system for a new domain using very few labeled examples. However, existing prototypical networks for this task suffer from roughly estimated label dependency and closely distributed prototypes, thus often causing misclassifications. To address the above issues, we propose EP-Net, an Entity-level Prototypical Network enhanced by dispersedly distributed prototypes. EP-Net builds entity-level prototypes and considers text spans to be candidate entities, so it no longer requires the label dependency. In addition, EP-Net trains the prototypes from scratch to distribute them dispersedly and aligns spans to prototypes in the embedding space using a space projection. Experimental results on two evaluation tasks and the Few-NERD settings demonstrate that EP-Net consistently outperforms the previous strong models in terms of overall performance. Extensive analyses further validate the effectiveness of EP-Net.

## introduction as a core language understanding task, named entity recognition (ner) faces rapid domain shifting. when transferring ner systems to new domains, one of the primary challenges is dealing with the mismatch of entity types @xcite . for example, only 2 types are overlapped between i2b2 @xcite and ontonotes @xcite , which have 23 and 18 entity types, respectively. unfortunately, annotating a new domain takes considerable time and efforts, let alone the domain knowledge required @xcite . few-shot ner is targeted in this scenario since it can transfer prior experience from resource-rich (source) domains to resource-scarce (target) domains. figure 1 : a comparison of token-and entity-level prototypical networks for few-shot ner, where the former builds prototypes for token labels and requires label dependency, while the latter builds prototypes for entity types and does not require label dependency. the dotted line denotes that the pair of token-prototype (or span-prototype) is the most similar. for clarity, we only list spans with lengths less than 2 and assume there are only 2 pre-defined entity types. previous few-shot ner models @xcite @xcite generally formulate the task as a sequence labeling task and employ token-level prototypical networks @xcite . these models first obtain token labels according to the most similar token-prototype pair and then obtain entities based on these labels, as figure 1 shows. the sequence labeling benefits from label dependency @xcite . however, when it comes to fewshot ner models, the label dependency is off the table, because a few labeled data is way insufficient to learn the reliable dependency, and the label sets could vary from domain to domain. to tackle this, some methods try to transfer roughly estimated dependency. @xcite first learn the abstract label transition probabilities in source domains and then copy them to target domains. as figure 2a shows, the abstract @xmath0. however, this makes the target probability sum of @xmath1. to avoid the possible probability overflows, @xcite propose an even distribution method. as figure 2b shows, the abstract @xmath2. however, this could lead to severe contradictions between the target probabilities and reality. for example, there are 4,983 date entities and only one email entity in the i2b2 test set, so the target probabilities of @xmath3. consequently, the current dependency transferring may lead to misclassifications due to the roughly estimated target transition probabilities, even though it sheds some light on few-shot ner. in addition, the majority of prototypical models for few-shot ner @xcite obtain prototypes by averaging the embeddings of each class's support examples, while @xcite and @xcite demonstrate that such prototypes distribute closely in the embedding space, thus often causing misclassifications. in this paper, we aim to tackle the above issues inherent in token-level prototypical models. to this end, we propose ep-net, an entity-level prototypical network enhanced by dispersedly distributed prototypes, as figure 1 shows. ep-net builds entity-level prototypes and considers text spans as candidate entities. thus it can determine whether a span is an entity directly according to the most similar prototype to the span. this also eliminates the need for the label dependency. for example, ep-net determines the "rain" and "tonight" are two entities, and their types are the weather and time, respectively (figure 1 ). 1 in addition, to distribute these prototypes dispersedly, ep-net trains them using a distance-based loss from scratch. and ep-net aligns spans and prototypes in the same embedding space by utilizing a deep neural network to map span representations to the embedding spaces of prototypes. in essence, ep-net is a span-based model. several span-based models @xcite @xcite have been proposed for the supervised ner task. our ep-net differs from these models in two ways: (1) the ep-net obtains entities based on the span-prototype similarity, while these models do so by classifying span representations. (2) the ep-net works effectively with 1 we also add a none type and assign it to spans that are not entities. few labeled examples, whereas these models need a large number of labeled examples to guarantee good performance. we evaluate our ep-net on the tag set extension and domain transfer tasks, as well as the few-nerd settings. experimental results demonstrate that ep-net consistently achieves new state-of-theart overall performance. qualitative analyses ( §5.5-5.6) and ablation studies ( §5.7) further validate the effectiveness of ep-net. in summary, we conclude the contributions as follows: (1) as far as we know, we are among the first to propose an entity-level prototypical network for few-shot ner. (2) we propose a prototype training strategy to augment the prototypical network with dispersedly distributed prototypes. (3) our model achieves the current best overall performance on two evaluation tasks and the few-nerd. few-shot ner. previous few-shot ner models @xcite generally formulate the task as a sequence labeling task and propose to use the token-level prototypical network. thus these models call for label dependency to guarantee good performance. however, it is hard to obtain exact dependency since the label sets vary greatly across domains. as an alternative, @xcite propose to transfer estimated dependency. they copy the learned abstract dependency from source to target domains, but the target dependency contradicts the probability definition. @xcite propose structshot, which improves the above dependency transferring by equally distributing the abstract dependency to target domains, whereas the target dependency contradicts the reality. @xcite introduce contrastive learning to the structshot, which inherits the estimated dependency transferring. we demonstrate that the roughly estimated dependency may harm model performance. in addition, promptbased models @xcite @xcite @xcite have been widely researched for this task recently, but the model performance heavily relies on the chosen prompts. current with our work, @xcite also propose a span-level prototypical network to bypass label dependency, but their work is still hampered by closely distributed prototypes. in contrast, our model constructs dispersedly distributed entity-level prototypes, thus avoiding the roughly estimated label dependency and closely distributed prototypes. ## task formulation and setup in this section, we formally define the task and then introduce the standard evaluation setup. ## model in this section, we first provide an overview of ep-net in §4.1, and then illustrate the model initializations in §4.2 and discuss the model in §4.3. ## conclusion in this paper, we propose an entity-level prototypical network for few-shot ner (ep-net). and we augment ep-net with dispersedly distributed prototypes. the entity-level prototypes enable ep-net to avoid suffering from the roughly estimated label dependency brought by abstract dependency transferring. moreover, ep-net distributes the prototypes dispersedly via supervised prototype training and maps spans to the embedding space of the prototypes to eliminate the alignment biases. experimental results on two evaluation tasks and the few-nerd settings demonstrate that ep-net beats the previously published models, creating new stateof-the-art overall performance. extensive analyses further validate the model's effectiveness. appendix a the greedy sampling algorithm algorithm 1: greedy sampling algorithm require: shot k, dev set x of a domain d and its entity type set t 1: sort types in t based on their frequencies in x 2: s ← ∅ // initialize the support set 3: {counti ← 0} // initialize the count of each type in s 4: while @xmath4: while @xmath5: sample (x , e) ∈ x s.t. @xmath6.type 1 // sample a sentence containing entities of ti // type, w/o replacement 7: s ← s ∪ {(x , e)} 8: update {countj} ∀ @xmath7.type 9: end while 10: end while 11: return s 1 e.type denotes the types of entities annotated in e b details of the evaluation task b.1 tag set extension the group a, b, and c split from the ontonotes dataset are as follows.