Label Semantics for Few Shot Named Entity Recognition (2022)

Abstract: We study the problem of few shot learning for named entity recognition. Specifically, we leverage the semantic information in the names of the labels as a way of giving the model additional signal and enriched priors. We propose a neural architecture that consists of two BERT encoders, one to encode the document and its tokens and another one to encode each of the labels in natural language format. Our model learns to match the representations of named entities computed by the first encoder with label representations computed by the second encoder. The label semantics signal is shown to support improved state-of-the-art results in multiple few shot NER benchmarks and on-par performance in standard benchmarks. Our model is especially effective in low resource settings.

## introduction named entity recognition (ner) seeks to locate named entity spans in unstructured text and classify them into pre-defined categories such as @xcite . as a fundamental natural language understanding task, ner often serves as an upstream component for more complex tasks such as question answering @xcite , relation extraction @xcite and coreference resolution @xcite . however, building an accurate ner system has traditionally required large amounts of high quality annotated in-domain data @xcite . this usually involves well defined annotation guidelines and training of annotators, which requires rich domain knowledge and can be prohibitively expensive @xcite . ## model we present our ner model. as shown in figure 1 , it consists of two bert-based encoders where one encoder is used to encode the document and its tokens and the other to encode labels. we formalize the differences between datasets used in our experimentation ( §2.1), then present how two bert-based encoders (and the modification with glove-based encoder for labels) are used to leverage semantics in labels for ner ( §2.2). finally we discuss the training procedure ( §2.3) and how labels are represented ( §2.4). ## experiments we evaluate our model and we compare it against existing few shot methods in two scenarios: high resource and low resource (few shot). in both cases, we assume there is a source dataset (which may be a set) with abundant data, and our goal is to maximize model performance on unseen target datasets which follow different taxonomies from the source dataset. ## analysis here, we show how semantics in label names help in low resource scenarios and how our model benefits from pre-finetuning stage. ## conclusion we propose a neural architecture that leverages semantics of label names for named entity recognition. our model significantly outperforms the state-of-the-art few shot ner baselines on low resource settings, and performs on-par in the high resource setting. we perform extensive experiments to show that the label encoder incorporates strong prior knowledge from bert and a dataset (source dataset) used in a pre-finetuning stage. we demonstrate that the semantics of label names in target datasets are critical to retrieve the prior knowledge. we also show that our model is robust to variation of label names and that it is able to differentiate between semantically closed labels.