Prompt-Based Metric Learning for Few-Shot NER (2023)

Abstract: Few-shot named entity recognition (NER) targets generalizing to unseen labels and/or domains with few labeled examples. Existing metric learning methods compute token-level similarities between query and support sets, but are not able to fully incorporate label semantics into modeling. To address this issue, we propose a simple method to largely improve metric learning for NER: 1) multiple prompt schemas are designed to enhance label semantics; 2) we propose a novel architecture to effectively combine multiple prompt-based representations. Empirically, our method achieves new state-of-the-art (SOTA) results under 16 of the 18 considered settings, substantially outperforming the previous SOTA by an average of 9.12% and a maximum of 34.51% in relative gains of micro F1.

## introduction named entity recognition (ner) is a key natural language understanding task that extracts and classifies named entities mentioned in unstructured texts into predefined categories. few-shot ner targets generalizing to unseen categories by learning from few labeled examples. recent advances for few-shot ner use metric learning methods which compute the token-level similarities between the query and the given support cases. @xcite proposed to use prototypical networks that learn prototypical representations for target classes. later, this method was introduced to few-shot ner tasks @xcite . @xcite proposed structshot, which uses a pretrained language model as a feature extractor and performs viterbi decoding at inference. @xcite proposed container based on contrastive learning. this approach optimizes an objective that * corresponding author. despite the recent efforts, there remain a few critical challenges for few-shot ner. first of all, as mentioned above, metric learning computes tokenlevel similarities between the query and support sets. however, the architectures used for computing similarities in previous work are agnostic to the labels in the support set. this prevents the model from fully leveraging the label semantics of the support set to make correct predictions. second, while prompts have been demonstrated to be able to reduce overfitting in few-shot learning (schick and schütze, 2020), due to a more complex sequence labeling nature of ner, the optimal design of prompts remains unclear for few-shot ner. in light of the above challenges, we explore a better architecture that allows using prompts to fully leverage the label semantics. we propose a simple method of prompt-based metric learning (proml) for few-shot ner, as shown in figure 1 . specifically, we introduce mask-reducible prompts, which is a special class of prompts that can be easily reverted to the original input by using a mask. by performing a masked weighted average over the representations obtained from multiple prompts, our method accepts multiple choices of prompts as long as they are mask-reducible. these prompts improve label efficiency by inserting semantic annotations into the text inputs. as instantiations of this framework, we design an option prefix prompt to provide the model with the candidate label options, and a label-aware prompt to associate each entity with its entity type in the input. as shown in figure 2 , a single prompt provides useful information but has some shortcoming. however, with a weighted average, multiple prompts are combined, which fully leverages label information. in our experiments, we find that using multiple prompts with the masked weighted average is effective for few-shot ner. empirically, our method achieves new state-of-the-art (sota) results under 16 of the 18 considered settings, substantially outperforming the previous sota by an average of 9.12% and a maximum of 34.51% in relative gains of micro f1. ## related work few-shot ner. few-shot ner targets generalizing to unseen categories by learning from few labeled examples. noisy supervised methods @xcite perform supervised pretraining over large-scale noisy web data such as winer @xcite . self training methods @xcite perform semisupervised training over a large amount of unlabelled data. alternative to these data-enhancement approaches, metric learning based methods have been widely used for few-shot ner @xcite @xcite . recently, prompt-based methods @xcite @xcite are proposed for few-shot ner as well. to introduce more finegrained entity types in few-shot ner, a large-scale human-annotated dataset few-nerd @xcite was proposed. @xcite ; @xcite formulate ner task as a span matching problem and decompose it to several procedures. @xcite decomposed the ner task into span detection and entity typing, and they separately train two models and finetune them on the test support set, achieving sota results on few-nerd @xcite . different from the above related works, our approach is a general framework of using prompts for token-level metric learning problems. meta learning. the idea of meta learning was first introduced in few-shot classification tasks for computer vision, attempting to learn from a few examples of unseen classes. since then metric-based methods have been proposed, such as matching networks @xcite and prototypical networks @xcite , which basically compute similarities according to the given support set, learn prototypical representations for target classes, respectively. it has been shown that these methods also enable few-shot learning for nlp tasks such as text classification @xcite , relation classification @xcite , named entity recognition @xcite @xcite , and machine translation @xcite . our ap-proach also falls into the category of metric-based meta learning and outperforms previous work on ner with an improved architecture. label semantics for ner. there have been some approaches that make use of label semantics @xcite . @xcite propose a crf framework with labelenhanced representations based on the architecture of @xcite . however, they mainly focus on slot tagging tasks while their performance on ner tasks is poor. @xcite introduce label semantics by aligning token representations with label representations. both of them only use label semantics for learning better label representations. in contrast, our approach incorporates label semantics into the inputs so that the model is able to jointly model the label information and the original text samples. this makes the similarity scores dependent on the support set labels and is particularly crucial for metric learning. our experiments also verify the advantages of our approach compared to previous work using labels semantics. prompt-based approaches for ner. with the emergence of prompt-based methods in nlp research, very recently, some prompt-based approaches for few-shot ner have been proposed @xcite @xcite . however, they use prompts to help with the label predictions based on classification heads instead of metric learning. moreover, some of these methods require searching for templates @xcite , good examples @xcite , or labelaware pivot words @xcite , which makes the results highly dependent on the search quality. different from these methods, our approach does not rely on a search process. more importantly, another key difference is that we employ prompting in the setting of metric learning. education, location, other: the university of chicago is a private research university in chicago, illinois. [the university of chicago|education] is a private research university in [chicago, illinois|location]. the university of chicago is a private research university in chicago, illinois. prompt pattern plain option prefix labelaware plain plain option prefix plain option prefix labelaware option prefix option prefix labelaware prompts for support set prompts for query set the university of washington is a public research university in seattle , washington. the university of washington is a public research university in seattle, washington. the university of washington is a public research university in seattle, washington. the university of washington is a public research university in seattle, washington. tagging results for query set education education education education education education location location location location location (only for support) lack of label information. reduce label space, but no direct demonstration. provide full label information to support. fully leverages label information. dataset d. after training on the training dataset d s , the model is required to predict labels for sentences from the test dataset d t . different from the standard ner task, the fewshot ner setting consists of a meta training phase and a test phase. at the meta training phase, the model trains on a training dataset d s . at the test phase, for various test datasets {d t (j) }, with only few labeled samples, the model is required to perform quick adaptions. in this paper, we mainly focus on two evaluation protocols and two task formulations which will be explained as follows. episode evaluation an episode, or a task, is defined as a pair of one support set and one query set (s, q) each consisting of sentences downsampled from the test set. for an n -way k-shot downsampling scheme, there are n labels among the support set s where each label is associated with k examples. the query set q shares the same label set with the support set. based on the support set, the model is required to predict labels for the query set. to perform an episode evaluation, a collection of t episodes {(s t , q t )} t @xmath0. the evaluation results are computed within each episode and are averaged over all t episodes. ## conclusions we propose a novel prompt-based metric learning framework proml for few-shot ner that leverages multiple prompts to guide the model with label semantics. proml is a general framework consistent with any token-level metric learning method and can be easily plugged into previous methods. we test proml under 18 settings and find it substantially outperforms previous sota results by an average of 9.12% and a maximum of 34.51% in relative gains of micro f1. we perform ablation studies to show that multiple prompt schemas benefit the generalization ability for our model. we demonstrate the visualization results for embedding space to unseen entities, showing that comparing with previous sota, proml learns better representations. we also present case studies and perform some analysis.