Large-Scale Label Interpretation Learning for Few-Shot Named Entity Recognition (2024)

Abstract: Few-shot named entity recognition (NER) detects named entities within text using only a few annotated examples. One promising line of research is to leverage natural language descriptions of each entity type: the common label PER might, for example, be verbalized as ”person entity.” In an initial label interpretation learning phase, the model learns to interpret such verbalized descriptions of entity types. In a subsequent few-shot tagset extension phase, this model is then given a description of a previously unseen entity type (such as ”music album”) and optionally a few training examples to perform few-shot NER for this type. In this paper, we systematically explore the impact of a strong semantic prior to interpret verbalizations of new entity types by massively scaling up the number and granularity of entity types used for label interpretation learning. To this end, we leverage an entity linking benchmark to create a dataset with orders of magnitude of more distinct entity types and descriptions as currently used datasets. We find that this increased signal yields strong results in zero- and few-shot NER in in-domain, cross-domain, and even cross-lingual settings. Our findings indicate significant potential for improving few-shot NER through heuristical data-based optimization.

## introduction few-shot named entity recognition (ner) refers to identifying and classifying named entities within text by learning from a few annotated examples. a widely adopted strategy in few-shot ner employs transfer learning with pre-trained language models (plms) to interpret labels based on their semantic meaning @xcite @xcite @xcite . the main idea is that such models learn to interpret a natural language description of an entity type for use in a word-level decoder. they learn in two phases: 1. a label interpretation learning phase on a ner-annotated dataset with a set of entity types and their verbalizations. for instance, the common label per might be verbalized as "person entity." in this phase, the model learns to associate entity type verbalizations with matching ner annotations. limitations. however, as figure 1 indicates, prior studies used only very limited numbers of distinct entity types for label interpretation learning. this is an artifact of relying on common ner datasets such as conll-03 @xcite , ontonotes @xcite , wnut-17 @xcite , or fewnerd @xcite , which only contain a small number of distinct entity types (between 4 and 66 types). furthermore, the majority of their entity types have a simple semantic definition, such as "person," "location," or "organization," and occur across several datasets. we hypothesize that these limitations overly constrain the semantic signal that is observed during label interpretation learning, thus constituting a main limiting factor to few-shot ner. contributions. with this paper, we introduce a novel approach named litset (label interpretation learning by scaling entity types) and systematically investigate the intuition that increasing the number of distinct entity types and their semantic exactness in label interpretation learning introduces a strong semantic prior to understand unseen entities in fewshot settings. to this end, we heuristically create a dataset with orders of magnitude more distinct entity types than commonly employed (cf. figure 1 ) and use it for extensive experimentation. in more detail, our contributions are: our approach leverages the recently released entity linking benchmark zelda @xcite and enriches it with type descriptions from wikidata @xcite (cf. section 3). we find that label interpretation learning on our heuristically derived corpus matches and, in many cases, significantly outperforms strong baselines. our findings indicate significant potential for improving few-shot ner through heuristical data-based optimization. we release the generated dataset and source code under the apache 2 license on github foot_0 . ## validation experiment for impact of entity types and label descriptions we first conduct an experiment to validate the intuition that a richer training signal for label interpretation learning positively impacts few-shot ner. to this end, we create a set of training datasets for label interpretation learning that each contain the same number of entities but vary in the number of distinct entity types and their label verbalization. we then compare the few-shot ner ability of models trained on each of these datasets. ## large-scale label interpretation learning as our validation experiment shows a positive impact of increasing the number and expressivity of entity types, we now aim to scale the signal for label interpretation learning to orders of magnitude more entity types. to this end, we heuristically derive a ner-annotated dataset using the recently released entity linking benchmark zelda and annotate it with wikidata information (section 3.1). we also introduce a modified training procedure for the bi-encoder to handle a very large space of entity types that applies to all architectures of its kind (section 3.2). we call this approach litset (label interpretation learning by scaling entity types). ## experiments we evaluate the impact of label interpretation training in various tagset extension settings. throughout all experiments, we compare label interpretation learning on litset with training on different baseline datasets. we present all hyperparameters used for our experiments in appendix d. specifically, we conduct the following experiments: further, we support our experiments by analyzing the impact of different label semantics used between label interpretation learning and few-shot fine-tuning (cf. section 4.1). at last, we refer to our ablation experiments using (1) different transformers as label encoders and (2) negative sampling (cf. appendices e and f). ## related work despite advancements achieved through pretrained word embeddings @xcite @xcite @xcite , few-shot ner focuses explicitly on generalizing to previously unseen label categories by leveraging a small number of labeled examples. metric learning @xcite is a common approach for few-shot ner @xcite @xcite and employs a distance metric to learn a shared representation space and assign labels based on class prototypes @xcite @xcite . additional components like contrastive loss @xcite or metalearning @xcite @xcite often further improve the performance. our approach aligns with this research by employing the bi-encoder architecture proposed in @xcite with an adapted loss calculation. however, prior work did not investigate the impact of the dataset used for label interpretation learning. we do so by increasing the training signal with expressive label verbalizations. thus, our approach may be applied to all prior work that relies on label verbalizations but may require architectural adaptations to accommodate arbitrary labels. template-filling and prompting methods with (large) language models @xcite @xcite @xcite have been widely used for few-shot ner @xcite @xcite @xcite . however, these approaches, relying on masked language model (mlm) objectives, may not be directly comparable to our method due to the scale of our labels. in its basic form, the template-based approach requires one forward pass per label or is limited by the model's maximum sequence length. additionally, our approach does not depend on large language models, which are often unavailable or impractical for few-shot ner. while specific efforts have been made to adapt to tags in few-shot domains @xcite , these studies evaluated only a limited number of labels. our approach shares similarities with @xcite and @xcite , where models were pre-trained using event mentions and entity links, respectively. however, our approach differs significantly. in @xcite , the pre-training objective targets the latent typing of entities, whereas our approach focuses on explicitly scaling up entity typing of few-shot ner models. our distinction from @xcite lies in exploring the effectiveness of distantly supervised training in a genuine few-shot context, wherein classes are not observed during label interpretation training. ## conclusion this paper introduces litset, a novel approach for label interpretation training with a large-scale set of entity types. we utilize an entity linking dataset annotated with wikidata information, resulting in a dataset with significantly more distinct labels. we conducted a thorough heuristical, databased optimization of few-shot ner models using litset. our experiments demonstrate that lit-set consistently outperforms various in-domain, cross-domain, and cross-lingual baselines and is transferable to other architectures and transformer models. for example, we surpass fewnerd by +14.7 f1 on average in pp. and chinese ontonotes by +9.0 f1 on average in pp. in low-resource settings. our method and experiments provide valuable insights into the factors influencing the performance of few-shot ner models utilizing label semantics.