Few-Shot Named Entity Recognition: An Empirical Baseline Study (2021)

Abstract: This paper presents an empirical study to efficiently build named entity recognition (NER) systems when a small amount of in-domain labeled data is available. Based upon recent Transformer-based self-supervised pre-trained language models (PLMs), we investigate three orthogonal schemes to improve model generalization ability in few-shot settings: (1) meta-learning to construct prototypes for different entity types, (2) task-specific supervised pre-training on noisy web data to extract entity-related representations and (3) self-training to leverage unlabeled in-domain data. On 10 public NER datasets, we perform extensive empirical comparisons over the proposed schemes and their combinations with various proportions of labeled data, our experiments show that (i)in the few-shot learning setting, the proposed NER schemes significantly improve or outperform the commonly used baseline, a PLM-based linear classifier fine-tuned using domain labels. (ii) We create new state-of-the-art results on both few-shot and training-free settings compared with existing methods.

## introduction named entity recognition (ner) involves processing unstructured text, locating and classifying named entities (certain occurrences of words or expressions) into particular categories of pre-defined entity types, such as persons, organizations, locations, medical codes, dates and quantities. ner serves as an important first component for tasks such as information extraction @xcite , information retrieval @xcite , question answering @xcite , task-oriented dialogues @xcite and other language understanding applications @xcite . deep learning we study three orthogonal strategies to improve ner models in the limited labeled data settings. to deal with the challenge of few-shot learning, we focus on improving the generalization ability of plms for ner from three complementary directions, shown in figure 1 . instead of limiting ourselves in making use of limited in-domain labeled tokens with the classic approach, (i) we create prototypes as the representations for different entity types, and assign labels via the nearest neighbor criterion; (ii) we continuously pre-train plms using web data with noisy labels that is available in much larger quantities to improve ner accuracy and robustness; (iii) we tag the in-domain unlabeled data with soft labels via self-training @xcite , and perform semi-supervised learning in conjunction with the limited labeled data. our contributions include: (i) we present the first systematic study for few-shot ner, a problem that is little explored in the literature. three distinctive schemes and their combinations are investigated. (ii) we perform comprehensive comparisons of these schemes on 10 public ner datasets from different domains. (iii) compared with existing methods on few-shot and training-free ner settings, the proposed schemes achieve sota performance despite their simplicity. to shed light on future research on few-shot ner, our study suggests that: (i) noisy supervised pre-training can significantly improve ner accuracy, and we will release our pre-trained checkpoints. (ii) selftraining consistently improves few-shot learning when the ratio of data amounts between unlabeled and labeled data is high. (iii) the performance of prototype learning varies on different datasets. it is useful when the number of labeled examples is small, or when new entity types are given in the training-free settings. linear classifier fine-tuning. following the recent self-supervised plms @xcite , a typical method for ner is to utilize a transformer-based backbone network to extract the contextualized representation of each token @xmath0. a linear classifier (i.e., a linear layer with parameter θ 1 = {w, b} followed by a softmax layer) is applied to project the representation z into the label space f θ 1 (z) = softmax(wz +b). in another word, the end-to-end learning objective for linear classifier based ner can be obtained via a function composition @xmath1, with trainable parameters θ = {θ 0 , θ 1 }. the pipeline is shown in figure 2 (a). the model is optimized by minimizing the cross-entropy: in practice, θ 1 = {w, b} is always updated, while θ 0 can be either frozen @xcite or updated @xcite . ## methods when only a small number of labeled tokens are available, it renders difficulties for the classical supervised fine-tuning approach: the model tends to over-fit the training examples and shows poor generalization performance on the testing set @xcite . in this paper, we provide a comprehensive study specifically for limited ner data settings, and explore three orthogonal directions shown in figure 1 : (i) how to adapt metalearning such as prototype-based methods for fewshot ner? (ii) how to leverage freely-available web data as noisy supervised pre-training data? (iii) how to leverage unlabeled in-domain sentences in a semi-supervised manner? note that these three directions are complementary to each other and can be used jointly to further extrapolate the methodology space in figure 1 . person is an average of three tokens: mr., bush and jobs). for a token in the query set, its distances from different prototypes are computed, and the model is trained to maximize the likelihood to assign the query token to its target prototype. (c) the wikipedia dataset is employed for supervised pre-training, whose entity types are related but different (e.g., musician and artist are more fine-grained types of person in the downstream task). the associated types on each token can be noisy. (d) self-training: an ner system (teacher model) trained on a small labeled dataset is used to predict soft labels for sentences in a large unlabeled dataset. the joint of the predicted dataset and original dataset is used to train a student model. ## conclusion and future work we have presented an empirical study on several directions in few-shot ner. three foundational methods and their combinations are systematically investigated: prototype-based methods, noisy supervised pre-training and self-training. they are intensively compared on 10 public datasets under various settings. all of them improve the plm's generalization ability when learning from a few labeled examples, among which supervised pretraining and self-training turn out to be particularly effective. our proposed schemes achieve sota on both few-shot and training-free settings compared with recent studies. we will release our benchmarks and code, in hope of inspiring future fewshot ner research with more advanced methods to tackle this challenging and practical problem. for future work, we believe our studies can be combined with other interesting explorations in distant supervised learning, such as augmentationbased methods @xcite and methods dealing with noisy labels @xcite . it would also be promising for researchers to consider larger pre-trained language models to learn better entity representations.