CONT ai NER : Few-Shot Named Entity Recognition via Contrastive Learning (2022)

Abstract: Named Entity Recognition (NER) in Few-Shot setting is imperative for entity tagging in low resource domains. Existing approaches only learn class-specific semantic features and intermediate representations from source domains. This affects generalizability to unseen target domains, resulting in suboptimal performances. To this end, we present CONTaiNER, a novel contrastive learning technique that optimizes the inter-token distribution distance for Few-Shot NER. Instead of optimizing class-specific attributes, CONTaiNER optimizes a generalized objective of differentiating between token categories based on their Gaussian-distributed embeddings. This effectively alleviates overfitting issues originating from training domains. Our experiments in several traditional test domains (OntoNotes, CoNLL’03, WNUT ‘17, GUM) and a new large scale Few-Shot NER dataset (Few-NERD) demonstrate that on average, CONTaiNER outperforms previous methods by 3%-13% absolute F1 points while showing consistent performance trends, even in challenging scenarios where previous approaches could not achieve appreciable performance.

## introduction named entity recognition (ner) is a fundamental nlu task that recognizes mention spans in unstructured text and categorizes them into a predefined set of entity classes. in spite of its challenging nature, recent deep-learning based approaches @xcite @xcite @xcite have achieved impressive performance. as these supervised ner models require large-scale human-annotated datasets, few-shot techniques that can effectively perform ner in resource constraint settings have recently garnered a lot of attention. the islands lie 860 km east of christchurch o place o qty. qty. o o place ## task formulation given a sequence of n tokens {x 1 , x 2 , . . . x n }, ner aims to assign each token x i to its corresponding tag label y i . few-shot setting for few-shot ner, a model is trained in a source domain with a tag-set {c s (i) } and tested in a data-scarce target domain with a tag-set {c d (j) } where i, j are index of different tags. since {c s (i) } ∩ {c d (j) } = ∅, it is very challenging for models to generalize to unseen test tags. in an n-way k-shot setting, there are n tags in the target domain |{c d (j) }| = n , and each tag is associated with a support set with k examples. tagging scheme for fair comparison of con-tainer against previous sota models, we follow an io tagging scheme where i-type represents that all of the tokens are inside an entity, and o-type denotes all the other tokens @xcite . evaluation scheme to compare with sota models in few-nerd leaderboard @xcite , we adpot episode evaluation as done by the authors. here, a model is assessed by calculating the micro-f1 score over multiple number of test episodes. each episode consists of a k-shot support set and a k-shot unlabeled query (test) set to make predictions . while few-nerd is explicitly designed for episode evaluation, traditional ner datasets (e.g., ontonotes, conll'03, wnut '17, gum) have their distinctive tag-set distributions. thus, sampling test episodes from the actual test data perturbs the true distribution that may not represent the actual performance. consequently, yang and katiyar (2020) proposed to sample multiple support sets from the original development set and use them for prediction in the original test set. we also use this evaluation strategy for these traditional ner datasets. as demonstrated in figure 2 , we first train our model in source domains. next, we finetune model representations using few-sample support sets to adapt it to target domains. the training and finetuning of container is illustrated in algorithm 1. finally, we use an instance level nearest neighbor classifier for inference in test sets. ## results and analysis we prudently analyze different components of our model and justify the design choices made in the scheming of container. we also examine the results discussed in section 4 that gives some intuitions about few-shot ner in general. ## conclusion we propose a contrastive learning based framework container that models gaussian embedding and optimizes inter token distribution distance. this generalized objective helps us model a class agnostic feature extractor that avoids the pitfalls of prior few-shot ner methods. container can also take advantage of few-sample support data to adapt to new target domains. extensive evaluations in multiple traditional and recent few-shot ner datasets reveal that, container consistently outperforms prior sotas, even in challenging scenarios. while we investigate the efficacy of distribution optimization based contrastive learning in few-shot ner, it will be of particular interest to investigate its potency in other domains as well.