Automatic Creation of Named Entity Recognition Datasets by Querying Phrase Representations (2023)

Abstract: Most weakly supervised named entity recognition (NER) models rely on domain-specific dictionaries provided by experts. This approach is infeasible in many domains where dictionaries do not exist. While a phrase retrieval model was used to construct pseudo-dictionaries with entities retrieved from Wikipedia automatically in a recent study, these dictionaries often have limited coverage because the retriever is likely to retrieve popular entities rather than rare ones. In this study, we present a novel framework, HighGEN, that generates NER datasets with high-coverage pseudo-dictionaries. Specifically, we create entity-rich dictionaries with a novel search method, called phrase embedding search, which encourages the retriever to search a space densely populated with various entities. In addition, we use a new verification process based on the embedding distance between candidate entity mentions and entity types to reduce the false-positive noise in weak labels generated by high-coverage dictionaries. We demonstrate that HighGEN outperforms the previous best model by an average F1 score of 4.7 across five NER benchmark datasets.

## introduction named entity recognition (ner) models often require a vast number of manual annotations for training, which limits their utility in practice. in several studies, external resources such as domain-specific dictionaries have been employed as weak supervision to reduce annotation costs @xcite @xcite . however, such dictionaries exist only for certain domains and building a dictionary for a new domain requires a high level of expertise and effort. to address this problem, a recent study proposed a framework called gener, which generates ner datasets without hand-crafted dictionaries @xcite . in gener, user questions that reflect the needs for ner are received as inputs figure 1 : the most frequent ten entities in the top 1,000 phrases retrieved from the 2018-12-20 version of wikipedia for the three questions: "which politician?", "which city?", and "which band?". due to a bias in the entity popularity @xcite , a current phrase retrieval model, densephrases @xcite , primarily returns popular entities, limiting the coverage of dictionaries. however, because the entity popularity of wikipedia is biased @xcite , existing open-domain qa models tend to retrieve popular entities rather than rare ones. this limits the coverage of dictionaries generated by gener. figure 1 shows examples of a bias in the entity population retrieved from the open-domain qa model. "david cameron," "beijing," and "the beatles" frequently appear in the top 1,000 retrieved phrases for each type of ques-tion. low-coverage dictionaries created from these biased results can cause incomplete annotations (i.e., false-negative entities), which impedes the training of ner models. unfortunately, increasing the number of retrieved phrases (i.e., larger top-k) is not an appropriate solution because it is computationally inefficient and causes a high falsepositive rate in the dictionary. therefore, a new search method that can efficiently retrieve diverse entities with a reasonable top-k and a new ner dataset generation framework based on this search method are needed. in this study, we present highgen, an advanced framework for generating ner datasets with automatically constructed 'high-coverage' dictionaries. specifically, we first obtain phrases and sentences and constitute an initial dictionary in a similar manner to gener. subsequently, we expand the initial dictionary using a phrase embedding search, in which the embeddings of the retrieved phrases are averaged to re-formulate query vectors. these new queries specify contexts in which different entities of the same type appear, allowing our retriever to search over a vector space in which various entities are densely populated. 1 the expanded dictionary is used to annotate the retrieved sentences. because a larger dictionary can induce more false-positive annotations during rule-based string matching, we introduce a new verification process to ensure that weak labels annotated by the string matching are correct. the verification process is performed by comparing the distance between the embeddings of a candidate entity and the target entity type. we trained recent ner models @xcite @xcite with the datasets generated by highgen and evaluated the models on five datasets. our models outperformed the baseline models trained using the previous best model gener by an average f1 score of 4.7 (section 4). in addition, we show an additional advantage of highgen over gener, which generates datasets using only a few hand-labeled examples without input user questions. highgen outperformed few-shot ner models on two datasets (section 5). finally, we perform an analysis of the factors affecting the retrieval diversity and ner performance (section 6). we make the following contributions: foot_1 ## method highgen comprises three stages of natural language search, phrase embedding search (figure 2 ), and dictionary matching and verification (figure 3 ). we highlight that the natural language search is similarly used in gener, but the last two stages are novel and first proposed in our study. ## experiments in this experiment, it was assumed that humanannotated datasets did not exist; thus, our models were trained only using synthetic data { x, ŷ} by highgen. to avoid excessive hyperparameter search, we used the same sets of input questions and the same number of sentences for each question (i.e., k 1 , . . . , k l ) as those used in the previous study @xcite . a new hyperparameter introduced in highgen, the number of phrases retrieved by phrase embedding search (i.e., k ′ 1 , . . . , k ′ l ), was set to 30k. please refer to in appendix a for the full list of hyperparameters and implementation details. for metrics, the entity-level precision, recall, and f1 scores were used @xcite . ## few-shot ner we show an additional use case for highgen to create ner datasets using only a few hand-labeled examples, without using input questions. this can eliminate a tuning/engineering effort of users that might be required for designing appropriate questions to identify ner needs, which is a distinct advantage of highgen over gener. specifically, highgen takes sentences with annotated phrases as input and retrieves x2 and v2 using the phrase embedding search (defined in equations ( 1 ) and ( 2 )), which are used as the unlabeled sentences and pseudo-dictionary to produce the final dataset. we tested two types of models. (1) the entitylevel model uses every annotated phrase as a separate query; thus, the number of queries equals the number of human annotations. on the other hand, (2) the class-level model first averages phrase vectors of the same entity types and uses them as queries; thus, the number of queries equals the number of entity types. the entity-level model would have an advantage in terms of entity recall and the class-level model can mitigate noise that each phrase vector may contain. setups. we sampled datasets from conll-2003 and bc5cdr so that each dataset consists of five sentences per entity type, which results in 20 and 10 results. ## analysis 6.1 retrieval performance we compared natural language search and phrase embedding search in terms of their accuracy and diversity. with reference to @xcite , we used 11 fine-grained questions within the following four coarse-grained entity types of (i) person (athlete, politician, actor), (ii) location (country, city, state in the usa), (iii) organization (sports team, company, institution), and (iv) biomedicine (disease, drug). we report the average scores for each coarse-grained entity type. metrics. (i) the precision at 100 (p@100) represents the accuracy of the top 100 retrieved phrases. because there are no gold annotations for the retrieved phrases, we manually determined whether the phrases correspond to the correct entity types. ## conclusion in this study, we presented an advanced dataset generation framework, highgen, which combines (1) phrase embedding search to address the problem of efficiently retrieving various entities using an open-domain retriever and (2) verification method to deal with false positives in a large dictionary. in the experiments, we demonstrated the superiority of highgen using five ner benchmarks and performed extensive ablation studies, comparison of retrieval performance, and analysis of potential uses of the phrase embedding search in few-shot ner scenarios. we hope that our study will provide practical help in several data-poor domains and valuable insights into entity retrieval and weakly supervised ner.