Simple Questions Generate Named Entity Recognition Datasets (2022)

Abstract: Recent named entity recognition (NER) models often rely on human-annotated datasets requiring the vast engagement of professional knowledge on the target domain and entities. This work introduces an ask-to-generate approach, which automatically generates NER datasets by asking simple natural language questions to an open-domain question answering system (e.g., “Which disease?”). Despite using fewer training resources, our models solely trained on the generated datasets largely outperform strong low-resource models by 19.5 F1 score across six popular NER benchmarks. Our models also show competitive performance with rich-resource models that additionally leverage in-domain dictionaries provided by domain experts. In few-shot NER, we outperform the previous best model by 5.2 F1 score on three benchmarks and achieve new state-of-the-art performance.

## introduction named entity recognition (ner) is the task of extracting named entities of specific types from text. an ner dataset essentially reflects the need to extract specific entity types. for instance, ncbidisease @xcite was created for extracting disease entities from text. recent ner models have provided robust performance when trained on carefully designed human-annotated datasets @xcite @xcite . however, suppose that we want to build an ner model for extracting bacteria or other specific types for which human-annotated datasets are insufficient. whenever we extract such entity types, should we rely on professional knowledge to create new datasets? * jl currently works at google research. the collaboration started before he joined google. previous weakly supervised ner models @xcite tackled this problem using rich in-domain dictionaries (e.g., the comparative toxicogenomics database) and unlabeled training sentences (i.e., in-domain sentences), where entities in the dictionary are used to annotate the training sentences. however, these approaches easily fail in practice because in-domain dictionaries and sentences are often unavailable or expensive to construct for many entity types. it will be challenging to build ner models for enzyme or astronomical object entities without expert-level knowledge required for building dictionaries and searching for a large number of in-domain sentences for annotation. in this study, we introduce gener, an automated dataset generation framework for ner, which automatically constructs high-quality ner datasets. in particular, concrete needs for ner are described using simple natural language questions such as "which @xcite ?", where [type] is substituted by the required entity type (e.g., "which disease?"). such questions do not require professional knowledge of the target domain and allow even non-experts to easily build domain-specific ner datasets. using a phrase retrieval model designed for open-domain question answering (qa) @xcite , gener first retrieves candidate entities (i.e., phrases) and evidence sentences from a large-scale open-domain corpus (e.g., wikipedia). the retrieved entities form a pseudo-dictionary, which is used to annotate the evidence sentences to create the dataset. we then train standard ner models on our generated dataset using a recent self-training method @xcite . as shown in model resource x train / y train / v wikigold ncbidisease rich-resource models (w/ training label or in-domain dict.) fully supervised ✓ / ✓ / ✗ 86.8 88.6 bond ✓ / ✗ / ✓ 59.8 71.4 low-resource models tallor ✓ / ✗ / △ 30.3 44.3 gener (ours) each method is categorized based on how much it relies on in-domain resources during training. x train : (unlabeled) training sentences. y train : human-annotated training labels. v: in-domain dictionaries by domain experts. in-domain resources are either fully used (✓), partially used (△), or not used (✗). while using the fewest in-domain resources, gener shows strong performance on various domains and entity types. we demonstrate the effectiveness of gener using six popular ner benchmarks across four domains: news @xcite , wikipedia @xcite , twitter @xcite , and biomedicine @xcite @xcite . models solely trained on our generated datasets from gener significantly outperformed tallor on all benchmarks by an average f1 score of 19.4. although our models did not use rich in-domain dictionaries, they sometimes outperformed the previous best weakly supervised model, bond @xcite , on two benchmarks. moreover, gener achieved new state-of-the-art results on three few-shot ner benchmarks, outperforming the previous best model quip @xcite by an f1 score of 5.2. finally, we conducted extensive ablation studies and analyses to highlight important factors for low-resource ner. our contributions are summarized as follows: ## method this section describes gener, which automatically generates ner datasets from scratch and provides ner models trained on these datasets. figure 1 provides an overview of the gener. ## experiments we evaluated gener in two scenarios wherein training resources are scarce: low-resource ner (section 4.1) and few-shot ner (section 4.2). we used entity-level precision (p), recall (r), and f1 score (f1) as the evaluation metrics. query term selection evaluating our models requires investigating the needs inherent in the benchmark datasets. in most ner benchmarks, entity types are coarsely defined and can be classified into many subtypes. these subtypes are often defined differently based on the datasets. for instance, the organization type of conll-2003 mostly includes sports teams and companies, but that of wikigold additionally covers bands. therefore, to understand the needs of each ner dataset, we sampled 100 exbaselines we compared gener with other lowresource models that do not use a full-size indomain dictionary. among the previous lowresource models, tallor @xcite uses the least amount of in-domain resources: unlabeled training sentences x train and the set of seed entities, which is a small dictionary v that contains 20-60 manually selected (i.e., @xmath0. in addition to tallor, we provide baselines that use similar in-domain resources as tallor, i.e., seed entities, neural tagger, and self-training. more details of each baseline are presented in appendix c.1. additionally, we also report the performance of rich-resource models that have access to either human-annotated training labels y train or rich indomain dictionaries v constructed by domain experts. this type of model includes fully supervised roberta @xcite , biobert @xcite , and the previous best weakly supervised model bond @xcite . results as shown in ## discussion and conclusion gener is the first attempt to automatically generate ner datasets using a general-purpose qa system. gener, while using fewer in-domain resources, largely outperformed existing lowresource models on all six benchmarks, sometimes even outperforming the rich-resource model bond. gener achieved a new state-of-the-art performance on three benchmarks upon evaluating the few-shot setting. our code and datasets have been made publicly available to facilitate further research. we discuss some of the important aspects of gener that have not been explored in depth and provide possible future directions.