Simple and Effective Few-Shot Named Entity Recognition with Structured Nearest Neighbor Learning (2020)

Abstract: We present a simple few-shot named entity recognition (NER) system based on nearest neighbor learning and structured inference. Our system uses a supervised NER model trained on the source domain, as a feature extractor. Across several test domains, we show that a nearest neighbor classifier in this feature-space is far more effective than the standard meta-learning approaches. We further propose a cheap but effective method to capture the label dependencies between entity tags without expensive CRF training. We show that our method of combining structured decoding with nearest neighbor learning achieves state-of-the-art performance on standard few-shot NER evaluation tasks, improving F1 scores by 6% to 16% absolute points over prior meta-learning based systems.

## introduction named entity recognition (ner) aims at identifying and categorizing spans of text into a closed set of classes, such as people, organizations, and locations. as a core language understanding task, ner is widely adopted in several domains, such as news @xcite , medical @xcite , and social media @xcite . however, one of the primary challenges in adapting ner to new domains is the mismatch between the different domain-specific entity types. for example, only two out of the twenty-three entity types annotated in the i2b2 2014 @xcite data can be found in the ontonotes 5 @xcite annotations. unfortunately, obtaining ner annotations for a novel domain can be quite expensive, often requiring domain knowledge. few-shot classification @xcite ) models aim at recognizing new classes based on only few labeled examples (support set) figure 1 : an few-shot ner example. professions (e.g., 'minister' and 'president') and dates (e.g., 'today' and 'tomorrow') are part of the o class. nearest neighbor classifier is better at predicting the o class using an instance-based metric compared to methods using class-based metric. from each class. in the context of ner, these fewshot classification methods can enable rapid building of ner systems for a new domain by labeling only a few examples per entity class. several previous studies @xcite propose using prototypical networks @xcite , a popular few-shot classification algorithm, to address the few-shot ner problem. however, these approaches only achieve 10 ∼ 30% f1 scores on average, when transferring knowledge between different ner datasets with one or five shot examples, warranting more effective methods for the problem. the direct adaption of existing few-shot classification methods to few-shot ner is challenging for two reasons. first, ner is essentially a structured learning problem. it is crucial to model label dependencies as shown in @xcite instead of directly classifying each token independently using the existing few-shot classification approaches. second, few-shot classification models @xcite typically learn to represent each semantic class by a prototype based on the labeled examples in its support set. however, for ner, unlike the entity classes, the outside (o) class does not represent any unified semantic meaning. in fact, tokens labeled with o in a dataset actually correspond to different semantic spaces that should be separately represented in a metric-based learning framework. consider, for example, in fig. 1 , semantic classes such as professions (e.g., 'minister') and dates (e.g., 'today') may also belong to the o class for some ner datasets. thus, previous approaches end up learning a noisy prototype for representing o class in this low-resource setting. in this paper, we propose a simple, yet effective method structshot for few-shot ner. instead of learning a prototype for each entity class, we represent each token in the labeled examples of the support set by its contextual representation in the sentence. we learn these contextual representations from training a standard supervised ner model @xcite , on the source domain. whereas meta-learning approaches @xcite simulate few-shot evaluation setup during training, our approach does not need to do so. this makes it possible to deploy a unified ner system supporting both classical and emerging types of entities, without the overhead of maintaining a separate few-shot system. during evaluation, structshot uses a nearest neighbor (nn) classifier and a viterbi decoder for prediction. as shown in fig. 1 , for each token ("president") in the target example, the nn classifier finds its nearest token ("minister") from the support examples, instead of relying on an erroneous class-level (outside) prototypical representation. we also improve our nearest neighbor predictions by using viterbi decoder @xcite to capture label dependencies. we perform extensive in-domain and out-ofdomain experiments for this problem. we test our systems on both identifying new types of entities in the source domain as well as identifying new types of entities in various target domains in one-shot and five-shot settings. in addition to the previous evaluation setup followed by @xcite , we propose a more standard and reproducible evaluation setup for few-shot ner by using standard test sets and development sets from benchmark datasets of several domains. in particular, we sample support sets from the standard development set and evaluate our models on the standard test set. for all our experiments, we find that our proposed systems outperform previous meta-learning systems by 6% to 16% absolute f1 score. ## problem statement and setup in this section, we formalize the task of few-shot ner and propose a standard evaluation setup to fa-cilitate meaningful comparison of results for future research. ## model in this section, we present our few-shot ner algorithm based on structured nearest neighbor learning (structshot). our method uses a ner model @xcite trained on the source domain, as a token embedder to generate contextual representations for all tokens. at inference, these static representations are simply used for nearest neighbor token classification. we also use a viterbi decoder to capture label dependencies by leveraging tag transitions estimated from the source domain. ## experiments in this section, we compare structshot against existing methods on two few-shot ner scenarios: tag set extension and domain transfer. we adopt several benchmark ner corpora in different domains for the few-shot experiments. foot_4 ## discussion in this section, we investigate two questions: 1) why structshot is so effective? and 2) why few-shot ner is so difficult? ## conclusion we introduce structshot, a simple few-shot ner system that achieves sota performance without any few-shot specific training. we identify two weaknesses of previous systems related to their handling of o class and modeling label dependencies. our systems overcomes these challenges with nearest neighbor learning and structured decoding. we further propose a standard evaluation setup for few-shot ner and show that structshot significantly outperforms prior sota systems on popular benchmarks across multiple domains. in the future, we want to extend our system to other few-shot sequence tagging problems such as part-of-speech tagging and slot filling.