Prompt-based Text Entailment for Low-Resource Named Entity Recognition (2022)

Abstract: Pre-trained Language Models (PLMs) have been applied in NLP tasks and achieve promising results. Nevertheless, the fine-tuning procedure needs labeled data of the target domain, making it difficult to learn in low-resource and non-trivial labeled scenarios. To address these challenges, we propose Prompt-based Text Entailment (PTE) for low-resource named entity recognition, which better leverages knowledge in the PLMs. We first reformulate named entity recognition as the text entailment task. The original sentence with entity type-specific prompts is fed into PLMs to get entailment scores for each candidate. The entity type with the top score is then selected as final label. Then, we inject tagging labels into prompts and treat words as basic units instead of n-gram spans to reduce time complexity in generating candidates by n-grams enumeration. Experimental results demonstrate that the proposed method PTE achieves competitive performance on the CoNLL03 dataset, and better than fine-tuned counterparts on the MIT Movie and Few-NERD dataset in low-resource settings.

## introduction recently, pre-trained language models (plms) have achieved promising improvement on several nlp tasks @xcite @xcite . nevertheless, fine-tuning language models still needs a moderate number of labeled data for downstream tasks. when difficulties result in limited labeled data available, the trained model shows large variance in downstream performance under full fine-tuning @xcite . for example, labeling technical and professional terms can be time-consuming and labor-intensive in medical scenarios. moreover, crowd-sourced annotation is also limited by the reality of existing samples (e.g., * corresponding authors when online health assistants are applied to rare diseases). to address learning challenges in these lowresource scenarios, researchers find that plms can learn well by prompt-based learning @xcite . prompt-based learning models the probability of text directly; it does not need an extra fully-connected layer usually used by fine-tuning. the main idea is to reformulate nlp tasks as cloze-style question answering for better using the knowledge in plms. the model predicts the word probability of masked positions and then derives the final output via mapping relations between words and labels. previous works have shown the ability of prompt-based learning under low-resource settings @xcite @xcite . for example, some prompt-based works have explored in classification and generation tasks where it is relatively easy to reformulate into cloze-style tasks (cf. section 4). nevertheless, the application to named entity recognition (ner) still poses challenges for current methods. unlike text classification and text generation, ner is the task of identifying named entities (e.g., person name, location) in a given sentence, and each unit of the input needs to be predicted. if we directly use masked language modeling (mlm) head to predict each unit label, the lexical and semantic coherence are ignored as there exists latent relationships between the tokens @xcite @xcite . in this work, we propose prompt-based text entailment (pte) for low-resource ner. firstly, we reformulate ner as a text entailment task. textual entailment (te) is the task of studying the relation of two sentences, premise (p) and hypothesis (h): whether h is true given p @xcite . specifically, we treat the original sentence as premise and entity type-specific prompt as a hypothesis. given an entity type, the p and h are fed into plms to get entailment scores for each candidate. then, the entailment score is the probability of a specific token at the mask position of the prompt. after that, the entity type with the top entailment score is selected as the final label. during inference, we enumerate all possible text spans or words in the input sentence as named entity candidates @xcite . the reformulation provides a unified entailment framework for ner tasks where annotations are insufficient, as the model shares the same inference pattern across different domains. as such, we can also leverage generic text entailment datasets such snli @xcite and mnli @xcite to pre-train models, which transfer knowledge from the general domain and get better performance in new domains. our method can be a step forward towards the development of a solution for the lowresource ner because any new domain does not typically have extensive annotated data in the real world, whereas it is feasible to obtain a couple of examples (e.g., online assistant). moreover, considering the existence of noisy annotations, te only needs to specify the labels of certain entities for training rather than the complete annotations of the entire sequence. experimental results demonstrate that the proposed method pte achieves competitive f1 score on the conll03 dataset @xcite , and better than fine-tuned counterparts by a large margin on the mit movie @xcite and few-nerd datasets @xcite in low-resource settings. ## experiments we compare our methods with several baselines on both rich-resource settings and low-resource settings. we use the conll2003 @xcite as the rich-resource dataset, and mit movie @xcite , few-nerd @xcite as the cross-domain low-resource datasets. and we conduct experiments on the conll03 dataset in both full and low-resource settings. the dataset statistics and experimental settings are included in appendix b and c. the standard precision, recall, and f1 score are used for model evaluation. ## related work previous works have shown the ability of promptbased learning under low-resource settings @xcite @xcite . @xcite address low-resource text classification by manually designing templates as prompt-based learning in a iterative training manner. @xcite improve low-resource performance with well-designed templates with demonstrations. @xcite apply continuous prompts for low-resource learning. recently, some works @xcite @xcite also focus on low-resource ner. in contrast, we propose to use prompt-tuning to treat ner as the te task. unlike traditional ner methods, we use prompt-based learning without an additional linear layer for fine-tuning. by defining different prompts, the model is able to perform well in low-resource settings, which adapts to new domains with few labeled data. in contrast to recent work which also adopts prompt-based fine-tuning for ner @xcite , we show that the effectiveness of the text entailment reformulation for named entity recognition using plms. ## conclusion in this paper, we apply prompt-based learning to low-resource named entity recognition. for token classification of ner, we reformulate it into a text entailment task. our method transfers knowledge in different nlp tasks and domains, and performs better in low-resource scenarios. future work includes how to apply pte to other nlp tasks.