Few- NERD : A Few-shot Named Entity Recognition Dataset (2021)

Abstract: Recently, considerable literature has grown up around the theme of few-shot named entity recognition (NER), but little published benchmark data specifically focused on the practical and challenging task. Current approaches collect existing supervised NER datasets and re-organize them to the few-shot setting for empirical study. These strategies conventionally aim to recognize coarse-grained entity types with few examples, while in practice, most unseen entity types are fine-grained. In this paper, we present Few-NERD, a large-scale human-annotated few-shot NER dataset with a hierarchy of 8 coarse-grained and 66 fine-grained entity types. Few-NERD consists of 188,238 sentences from Wikipedia, 4,601,160 words are included and each is annotated as context or a part of the two-level entity type. To the best of our knowledge, this is the first few-shot NER dataset and the largest human-crafted NER dataset. We construct benchmark tasks with different emphases to comprehensively assess the generalization capability of models. Extensive empirical results and analysis show that Few-NERD is challenging and the problem requires further research. The Few-NERD dataset and the baselines will be publicly available to facilitate the research on this problem.

## introduction named entity recognition (ner), as a fundamental task in information extraction, aims to locate and classify named entities from unstructured natural language. a considerable number of approaches equipped with deep neural networks have shown promising performance @xcite on fully supervised ner. notably, pre-trained language models (e.g., bert @xcite ) with an additional classifier achieve significant success on this task and gradually become the base paradigm. such studies demonstrate that deep models could yield remarkable results accompanied by a large amount of annotated corpora. with the emerging of knowledge from various domains, named entities, especially ones that need professional knowledge to understand, are difficult to be manually annotated on a large scale. under this circumstance, studying ner systems that could learn unseen entity types with few examples, i.e., few-shot ner, plays a critical role in this area. there is a growing body of literature that recognizes the importance of few-shot ner and contributes to the task @xcite @xcite @xcite . unfortunately, there is still no dataset specifically designed for few-shot ner. hence, these methods collect previously proposed supervised ner datasets and reorganize them into a few-shot setting. common options of datasets include ontonotes @xcite , conll'03 @xcite , wnut'17 @xcite , etc. these research efforts of few-shot learning for named entities mainly face two challenges: first, most datasets used for few-shot learning have only 4-18 coarse-grained entity types, making it hard to construct an adequate variety of "n-way" metatasks and learn correlation features. and in reality, we observe that most unseen entities are finegrained. second, because of the lack of benchmark datasets, the settings of different works are inconsistent @xcite , leading to unclear comparisons. to sum up, these methods make promising contributions to few-shot ner, nevertheless, a specific dataset is urgently needed to provide a unified benchmark dataset for rigorous comparisons. to alleviate the above challenges, we present a large-scale human-annotated few-shot ner dataset, few-nerd, which consists of 188.2k sentences extracted from the wikipedia articles and 491.7k entities are manually annotated by well-trained annotators (section 4.3). to the best of our knowledge, few-nerd is the first dataset specially constructed for few-shot ner and also one of the largest human-annotated ner dataset (statistics in section 5.1). we carefully design an annotation schema of 8 coarse-grained entity types and 66 fine-grained entity types by conducting several pre-annotation rounds. (section 4.1). in contrast, as the most widely-used ner datasets, conll has 4 entity types, wnut'17 has 6 entity types and ontonotes has 18 entity types (7 of them are value types). the variety of entity types makes few-nerd contain rich contextual features with a finer granularity for better evaluation of fewshot ner. the distribution of the entity types in few-nerd is shown in figure 1 , more details are reported in section 5.1. we conduct an analysis of the mutual similarities among all the entity types of few-nerd to study knowledge transfer (section 5.2). the results show that our dataset can provide sufficient correlation information between different entity types for few-shot learning. for benchmark settings, we design three tasks on the basis of few-nerd, including a standard supervised task (few-nerd (sup)) and two few-shot tasks (few-nerd-intra) and few-nrtd (inter)), for more details see section 6. few-nerd (sup), few-nerd (intra), and few-nerd (inter) assess instance-level generalization, type-level generalization and knowledge transfer of ner methods, respectively. we implement models based on the recent state-of-theart approaches and evaluate them on few-nerd (section 7). and empirical results show that few-nerd is challenging on all these three settings. we also conduct sets of subsidiary experiments to analyze promising directions of few-shot ner. hopefully, the research of few-shot ner could be further facilitated by few-nerd. ## related work as a pivotal task of information extraction, ner is essential for a wide range of technologies @xcite @xcite . and a considerable number of ner datasets have been proposed over the years. for example, conll'03 @xcite is regarded as one of the most popular datasets, which is curated from reuters news and includes 4 coarsegrained entity types. subsequently, a series of ner datasets from various domains are proposed @xcite @xcite @xcite . these datasets formulate a sequence labeling task and most of them contain 4-18 entity types. among them, due to the high quality and size, ontonotes 5.0 @xcite ) is considered as one of the most widely used ner datasets recently. as approaches equipped with deep neural networks have shown satisfactory performance on ner with sufficient supervision @xcite , few-shot ner has received increasing attention @xcite @xcite . few-shot ner is a considerably challenging and practical problem that could facilitate the understanding of textual knowledge for neural model @xcite . due to the lack of specific benchmarks of few-shot ner, current methods collect existing ner datasets and use different few-shot settings. to provide a benchmark that could comprehensively assess the generalization of models under few examples, we annotate few-nerd. to make the dataset practical and close to reality, we adopt a fine-grained schema of entity annotation, which is inspired and modified from previous fine-grained entity recognition studies @xcite @xcite . ## benchmark settings we collect and manually annotate 188,238 sentences with 66 fine-grained entity types in total, which makes few-nerd one of the largest human-annotated ner datasets. to comprehensively exploit such rich information of entities and contexts, as well as evaluate the generalization of models from different perspectives, we construct three tasks based on few-nerd (statistics are reported in ## conclusion and future work we propose few-nerd, a large-scale few-shot ner dataset with fine-grained entity types. this is the first few-shot ner dataset and also one of the largest human-annotated ner dataset. few-nerd provides three unified benchmarks to assess approaches of few-shot ner and could facilitate future research in this area. by implementing state-of-the-art methods, we carry out a series of experiments on few-nerd, demonstrating that few-shot ner remains a challenging problem and worth exploring. in the future, we will extend few-nerd by adding cross-domain annotations, distant annotations, and finer-grained entity types. few-nerd also has the potential to advance the construction of continual knowledge graphs.