Template-free Prompt Tuning for Few-shot NER (2022)

Abstract: Prompt-based methods have been successfully applied in sentence-level few-shot learning tasks, mostly owing to the sophisticated design of templates and label words. However, when applied to token-level labeling tasks such as NER, it would be time-consuming to enumerate the template queries over all potential entity spans. In this work, we propose a more elegant method to reformulate NER tasks as LM problems without any templates. Specifically, we discard the template construction process while maintaining the word prediction paradigm of pre-training models to predict a class-related pivot word (or label word) at the entity position. Meanwhile, we also explore principled ways to automatically search for appropriate label words that the pre-trained models can easily adapt to. While avoiding the complicated template-based process, the proposed LM objective also reduces the gap between different objectives used in pre-training and fine-tuning, thus it can better benefit the few-shot performance. Experimental results demonstrate the effectiveness of the proposed method over bert-tagger and template-based method under few-shot settings. Moreover, the decoding speed of the proposed method is up to 1930.12 times faster than the template-based method.

## introduction pre-trained language models (lms) have led to large improvements in nlp tasks @xcite @xcite . popular practice to perform downstream classification tasks is to replace the pretrained model's output layer with a classifier head and fine-tune it using a task-specific objective function. recently, a new paradigm, prompt-based learning, has achieved great success on few-shot classification tasks by reformulating classification tasks as cloze * equal contribution. † corresponding authors. ## problem setup in this work, we focus on few-shot ner task. different from previous works that assume a richresource source domain and available support sets during testing, we follow the few-shot setting of @xcite , which supposes that only a small number of examples are used for fine-tuning. such setting makes minimal assumptions about available resources and is more practical. specifically, when training on a new dataset d with the label space y, we assume only k training examples for each class in the training set, such that the total number of examples is k @xmath0. then, the model is tested with an unseen test set (x test , y test ) ∼ d test . here, for ner task, a training sample refers to a continual entity span @xmath1, . . . , x m } that is labeled with a positive class (e.g.,"person"). ## approach in this work, we propose a template-free prompt tuning method, entity-oriented lm (entlm) fine-tuning, for few-shot ner. we first give a description of the template-based prompt tuning. then we introduce the entlm method along with the label word engineering process. ## experiments in this section, we conduct few-shot experiments to verify the effectiveness of the proposed method. we also conducts intensive analytical experiments for label words selection.