COPNER : Contrastive Learning with Prompt Guiding for Few-shot Named Entity Recognition (2022)

Abstract: Distance metric learning has become a popular solution for few-shot Named Entity Recognition (NER). The typical setup aims to learn a similarity metric for measuring the semantic similarity between test samples and referents, where each referent represents an entity class. The effect of this setup may, however, be compromised for two reasons. First, there is typically a limited optimization exerted on the representations of entity tokens after initing by pre-trained language models. Second, the referents may be far from representing corresponding entity classes due to the label scarcity in the few-shot setting. To address these challenges, we propose a novel approach named COntrastive learning with Prompt guiding for few-shot NER (COPNER). We introduce a novel prompt composed of class-specific words to COPNER to serve as 1) supervision signals for conducting contrastive learning to optimize token representations; 2) metric referents for distance-metric inference on test samples. Experimental results demonstrate that COPNER outperforms state-of-the-art models with a significant margin in most cases. Moreover, COPNER shows great potential in the zero-shot setting.

## introduction as a fundamental task in nature language process (nlp), named entity recognition (ner) aims to identify the spans of text according to a pre-defined set of entity classes, such as person, organization and location @xcite . many down-stream tasks heavily rely on these extracted entities, such as aspect-level sentiment @xcite , intention recognition @xcite , and knowledge graph construction @xcite . an enormous number of neural methods have shown promising ability on ner tasks (chiu * corresponding author @xcite @xcite , whereas insufficient labeled data in different domains are still a significant challenging for the community. considering obtaining full annotated data is labor-intensive and time-consuming, few-shot ner studies @xcite @xcite @xcite are raising more attention, which can alleviate annotation dependence and help neural methods transfer to other tasks easier. recently, prompt learning (pl) becomes a popular technology in nlp and shows great potential for dealing with few-shot issues @xcite @xcite . typical prompt learning is designed for understanding sentence-level tasks by decoding a special marker of an input. however, it is challenging to adapt pl to token-level tasks, which needs to identify the class of each token. inspired by pl, templatener @xcite applies manual templates for few-shot ner, which needs to enumerate all potential spans and forward propagate many times for each input, which is timeconsuming. compared with templatener, distance metircbased approaches @xcite @xcite are more popular and efficient in few-shot ner tasks. the key idea of this paradigm is to learn a similarity metric for measuring the semantic similarity between test samples and referents (e.g., prototypes or the nearest neighbors). the referents are usually derived from a few labeled samples through a pre-trained language model, e.g., bert @xcite . while being less costly, these methods are still limited in two aspects: firstly, their ability to capture entity-class-related semantics is limited, because their main goal is to learn a suitable similarity function rather than optimizing the parameters of a sentence encoder for better entity representations. secondly, as an anchored metric referent is derived from only few labeled data, it is insufficient to properly represent the semantics of the corresponding entity class. this issue may severely impact the performance of a few-shot ner model in the inference phase. to address these issues, we propose a novel approach named contrastive learning with prompt guiding for few-shot ner (copner). the core idea of copner is to leverage class-specific words (cws) from natural language to serve as the agents of corresponding entity types. specifically, the cws are included by appending a prompt to the original input sentence. as shown in figure 1 (a), an original input sentence " @xcite obama was born in 1961 @xcite " is concatenated with a prompt "person date none", where "person", "date" and "none" are the cws for the entity classes of person, date, and non-entity, respectively. in the training phase, the representations of cws are served as token-level supervision signals that guide the sentence encoder to pull the representations of tokens belonging to the same class, and also the representation of the anchored cw, to be closer. in this way, the sentence encoder can learn to capture the dependence between tokens for aligning the semantics of a mentioned entity with the semantics of the corresponding cw in the unified semantic space. in the inference phase, the representations of cws are treated as metric referents for predicting entity classes. as the representations of cws contain general and discriminates semantics at the scratch and are further trained to align with the corresponding entity set, they are more appropriate and stable than the referents derived from previous works. further, we explore different methods of prompt construction, aiming to understand the effects of different forms of prompts. we summarize existing few-shot ner research into three settings, including cross-label-space, domain transfer, and in-label-space. copner is evaluated under all these settings. we conduct experiments on six ner datasets and cop-ner largely outperforms state-of-the-art(sota) approaches in most cases, especially in complex scenarios. specifically, compared with sota results in few-nerd (intra) and few-nerd (in-ter), copner raises the f1 scores of 8.28% and 8.03% in these two fine-grained few-shot ner tasks. in domain transfer settings, copner also improves by 9.0% averaged f1 scores under 1-shot settings. additionally, considering cws can inherently carry the relevant category information, we explore the zero-shot ability of copner with satisfied results. the main contributions of this paper are summarized as follows: ## methodology the key idea of copner is to construct prompts with cws for both the model training and inference. for each sentence, the copner concatenates a prompt to it and feeds the supplemented one into a pre-trained language model (plm). then, the plm is trained in a contrastive learning fashion. in this process, the prompts play the role of the token-level anchors for constructing the positive pairs and negative pairs of contrastive learning. finally, in the inference stage, the representations of cws (in the prompt) are served as metric referents for predicting. the whole process of copner is shown in figure 1. we first train the used plm in source label spaces. next, the plm is fine-tuned by a few support sets for adapting to new label spaces in evaluation tasks, e.g., cross-label-space and domain transfer. finally, a token-level distance-based metric classifier is employed to obtain the final results in the inference phase. ## results and discussion in this section, we discuss the results of different few-shot ner settings and conduct experiments under the zero-shot setting. we also explore the effectiveness of different components of copner. ## conclusion in this paper, we propose copner, a novel fewshot ner approach taking the advantage of contrastive learning and prompt guiding. copner achieves sota performance in few-shot ner settings by constructing prompts with cws and exploiting the ability of contrastive learning to obtain enhanced representations and stable metric referents. copner can also handle zero-shot ner tasks. in the future, we will extend copner to more token-level few-shot classification tasks and further exploit its ability to handle zero-shot tasks.