Learning from Language Description: Low-shot Named Entity Recognition via Decomposed Framework (2021)

Abstract: In this work, we study the problem of named entity recognition (NER) in a low resource scenario, focusing on few-shot and zero-shot settings. Built upon large-scale pre-trained language models, we propose a novel NER framework, namely SpanNER, which learns from natural language supervision and enables the identification of never-seen entity classes without using in-domain labeled data. We perform extensive experiments on 5 benchmark datasets and evaluate the proposed method in the few-shot learning, domain transfer and zero-shot learning settings. The experimental results show that the proposed method can bring 10%, 23% and 26% improvements in average over the best baselines in few-shot learning, domain transfer and zero-shot learning settings respectively.

## introduction named entity recognition (ner) aims at identifying and categorizing spans of text into a pre-defined set of classes, such as people, organizations, and locations. as a fundamental language understanding task, ner is widely adopted in question answering @xcite , information retrieval @xcite and other language understanding applications @xcite @xcite . recent advances with pre-trained language models like bert @xcite , gpt-2 @xcite and roberta @xcite have shown remarkable success in ner . however, the success of these large-scale models still relies on fine-tuning them on large amounts of in-domain labeled data. unfortunately, obtaining ner annotations not only is expensive and time consuming, but also may not be feasible in many sensitive user applications due to data access and privacy constraints. this motivates us to study the problem of low-shot ner. low-shot ner focuses on identifying custom entities in a new domain with only a few in-domain examples or even without any in-domain labeled data, which are refereed to as few-shot ner and zero-shot ner respectively. the success of lowshot ner requires the model to be capable of transferring learned knowledge to recognize new entity classes. conventional ner models usually treat each class as a one-hot vector (represented by a class label) for training, and thus the trained model cannot capture the semantic meanings of those labels. in fact, the trained model could be highly associated with known classes and it is difficult to transfer learned knowledge to novel entity classes. to tackle this problem, several recent works @xcite @xcite @xcite employ prototypebased method to represent each class by a prototype based on the labeled examples and use nearest neighbor method for ner. however, each entity class in ner task may include several fine-grained entity classes and has diverse semantic meanings. correspondingly, the tokens or entities belonging to the same entity class are not necessarily close to each other @xcite , making it challenging to represent each entity class by a prototype based on a few examples. for example, misc, one of the four entity classes in the benchmark dataset @xcite , is a collection of fine-grained entity classes including events, nationalities, products and works of art. fac is an entity class in the ontonotes5 @xcite collection, including buildings, airports, highways, bridges and others. thus, prototype-based methods may end up learning noisy representations of prototypes and cannot achieve satisfactory performance. moreover, prototype-based methods have unavoidable reliance on labeled examples, thereby making them unable to extend to the zero-shot learning setting, which is also an important and practical scenario in the low-shot ner. costa has not played since being struck by the ac milan forward after a bad-tempered european champions' league game on november 27 class description loc in geography, location and place are used to identify a pointor an area… per a person is a being that has certain capacities or attributes such as reason, morality… org an organization or organisation is an entity comprising multiple people… misc examples of miscellaneous entities include events, nationalities, products and works of art. entity class inference ac milan costa european champions' league costa has not played since being struck by the ac milan forward after a bad-tempered european champions' league game on november 27 ## methodology in this paper, we study how to develop an effective model which can identify custom entities in a novel domain with a small set of labeled data or even without using any labeled data. to this end, we decompose ner task into two sub-tasks: span detection and entity class inference. the span detection module is class-agnostic and can transfer knowledge across different entity classes. on top of span detection, the entity class inference module takes extracted spans as input to infer the semantic relationship between the spans with natural language description of entity classes. learning from natural language has an important advantage over existing categorical label learning methods, which is its ability to capture semantic meanings of labels and enable flexible zero-shot transfer. ## experiments in this section, we empirically study and compare the proposed method with state-of-the-art methods in few-shot learning, domain transfer and zero-shot learning settings. backbone we use the the pre-trained bert base uncased model (∼110m parameters) as the backbone network. the inputs during training and inference 2 https://github.com/juand-r/entity-recognition-datasets are lowercased to make them case-insensitive. the implementation details and hyper-parameter configurations are presented in appendix. ## conclusions in this work, we study how to conduct ner in a low resource setting (when there are few or zero labeled data). to this end, we develop a novel ner framework spanner that decomposes ner task into span detection and entity class inference. this framework enables zero-shot ner and improves the performance of few-shot ner by capturing semantic meanings of entity classes. extensive experiments on 5 benchmark datasets and various settings demonstrate the effectiveness of the proposed model, particularly in the low-resource settings.