Few-shot Named Entity Recognition with Self-describing Networks (2022)

Abstract: Few-shot NER needs to effectively capture information from limited instances and transfer useful knowledge from external resources. In this paper, we propose a self-describing mechanism for few-shot NER, which can effectively leverage illustrative instances and precisely transfer knowledge from external resources by describing both entity types and mentions using a universal concept set. Specifically, we design Self-describing Networks (SDNet), a Seq2Seq generation model which can universally describe mentions using concepts, automatically map novel entity types to concepts, and adaptively recognize entities on-demand. We pre-train SDNet with large-scale corpus, and conduct experiments on 8 benchmarks from different domains. Experiments show that SDNet achieves competitive performances on all benchmarks and achieves the new state-of-the-art on 6 benchmarks, which demonstrates its effectiveness and robustness.

## introduction few-shot named entity recognition (fs-ner) aims to identify entity mentions corresponding to new entity types (i.e., novel types) with only a few illustrative examples. fs-ner is a promising technique for open-domain ner which contains various unforeseen types and very limited examples and therefore has attached great attention in recent years @xcite . the main challenge of fs-ner is how to accurately model the semantics of unforeseen entity types using only a few illustrative examples. to achieve this, fs-ner needs to effectively capture information in few-shot examples, meanwhile exploiting and transferring useful knowledge from external resources. unfortunately, information entailed in illustrative examples is very limited, i.e., the limited information challenge. and external knowledge usually doesn't directly match with the new task because it may contain irrelevant, heterogeneous, or even conflicting knowledge @xcite ) -which we refer as knowledge mismatch challenge. for example, the schemas in wikipedia, ontonotes @xcite and wnut17 @xcite are conflicting, where "america" is geographic entity in wikipedia, gpe in ontonotes, and location in wnut17. such a knowledge mismatch problem makes it unsuitable to directly transfer external knowledge to downstream tasks. consequently, how to sufficiently leverage limited few-shot examples and precisely transfer external knowledge are the critical challenges for fs-ner. to this end, this paper proposes a self-describing mechanism for fs-ner. the main idea behind selfdescribing mechanism is that all entity types can be described using the same set of concepts, and the mapping between types and concepts can be universally modeled and learned. in this way, the knowledge mismatch challenge can be resolved by uniformly describing different entity types using the same concept set. for example, in figure 1 the types in different schemas are mapped to the same concept set {park, garden, country, ...}, therefore the knowledge in different sources can be universally described and transferred. furthermore, because the concept mapping is universal, the few examples are only used to construct the mapping between novel types and concepts, the limited information problem can be effectively addressed. based on the above idea, we propose selfdescribing networks -sdnet, a seq2seq generation network which can universally describe mentions using concepts, automatically map novel entity types to concepts, and adaptively recognize entities on-demand. specifically, to capture the semantics of a mention, sdnet generates a set of universal concepts as its description. for example, generate {capital, city} for "dr. kohl came to [beijing], ...". to map entity types to concepts, sd-net generates and fuses the concept description of the mentions with the same entity type. for example, map gpe to {country, capital, city} using its mentions "beijing" and "america". to recognize entity, sdnet directly generates all entities in a sentence via a concept-enriched prefix prompt, which contains the target entity types and their concept descriptions. for example, recognizing entity in "france is beautiful." by generating"france is gpe" using prefix prompt "[eg] gpe: {country, capital, city}". because the concept set is universal, we pretrain sdnet on large-scale, easily accessible web resources. concretely, we collect a pre-training dataset which contains 56m sentences with more than 31k concepts by leveraging the links from wikipedia anchor words to the wikidata items. by projecting both mentions and entity types to a universal concept space, sdnet can effectively enrich entity types to resolve the limited information problem, universally represent different schemas to resolve the knowledge mismatch problem, and can be effectively pre-trained in a unified way. moreover, all the above tasks are modeled in a single generation model by using prefix prompt mechanism to distinguish different tasks, which makes the model controllable, universal and can be continuously trained. we conduct experiments on 8 few-shot ner benchmarks with different domains. experiments show that sdnet leads to very competitive performance and achieves the new state-of-the-art on 6 of these benchmarks. 1 generally speaking, the contributions of this paper are: ## related work to deal with the limited information challenge, current fs-ner studies mostly focus on leveraging external knowledge, many knowledge resources are used: 1) plms. early fs-ner studies @xcite mainly use plms for better encoding. and prompt-based ner formulation is proposed to exploit the plms' knowledge more effectively @xcite @xcite @xcite @xcite @xcite . 2) existing annotation datasets. these studies @xcite @xcite @xcite focus on reusing annotations in existing datasets, and the annotations can be used to pre-train ner models. 3) distantly annotated datasets. some works @xcite @xcite try to automatically construct ner datasets via distant supervision, but which often suffer from the partiallylabeled @xcite @xcite and noise label @xcite @xcite problem. to deal with the knowledge mismatch problem, @xcite al. (2015 et al. (2019) ; yu and yang (2020) merge the labels in different schemas into the same taxonomy for knowledge sharing. @xcite relabels the external noisy datasets using current labels. compared with these methods, we resolve the knowledge mismatch problem by mapping all entity types to a universal concept set, and the concept mapping and target entities are automatically generated using a self-describing networks. ## self-describing networks for fs-ner in this section, we describe how to build few-shot entity recognizers and recognize entities using selfdescribing networks. figure 3 (b) shows the entire procedure. specifically, sdnet is a seq2seq network which performs two generation tasks successively 1) mention describing, which generates the concept descriptions of mentions; 2) entity generation, which adaptively generates entity mentions corresponding to desirable novel type one by one. using sdnet, ner can be directly performed through the entity generation process by putting type descriptions into its prompt. given a novel type, its type description is built through mention describing upon its illustrative instances. in the following, we will first introduce sdnet, then describe how to construct type descriptions and build few-shot entity recognizers. ## learning in this section, we first describe how to pre-train sdnet using large-scale external data, so that the common ner ability can be captured through the mention describing ability and the entity generation ability. then we describe how to quickly adapt and transfer ner knowledge via fine-tuning. figure 3 shows the two processes and we describe them as follows. ## conclusions in this paper, we propose self-describing networks, a seq2seq generation model which can universally describe mentions using concepts, automatically map novel entity types to concepts, and adaptively recognize entities on-demand. a large-scale sdnet model is pre-trained to provide universal knowledge for downstream ner tasks. experiments on 8 datasets show that sdnet is effective and robust. for future work, we will extend self-describing mechanism to other nlp tasks like event extraction @xcite and complex ner tasks like nested @xcite or discontinuous ner @xcite .