DISTANT : Distantly Supervised Entity Span Detection and Classification (2023)

Abstract: We propose a distantly supervised pipeline NER which executes entity span detection and entity classification in sequence named DISTANT (DIstantly Supervised enTity spAN deTection and classification).The former entity span detector extracts possible entity mention spans by the distant supervision. Then the later entity classifier assigns each entity span to one of the positive entity types or none by employing a positive and unlabeled (PU) learning framework. Two models were built based on the pre-trained SciBERT model and fine-tuned with the silver corpus generated by the distant supervision. Experimental results on BC5CDR and NCBI-Disease datasets show that our method outperforms the end-to-end NER baselines without PU learning by a large margin. In particular, it increases the recall score effectively.

## introduction the development of named entity recognition (ner) is often hindered by a lack of annotated datasets. for example, medical and biological tasks often differ in target entity types and their granularity levels. in these scenarios, a domain-specific dictionary is often used for distant supervision to search for possible mention spans for the target entity types. so far, many distantly-supervised ner methods have been proposed. however, the performance of these methods often largely depends on the quality of the domain dictionary. therefore, it usually necessitates tuning the matching threshold values to account for such noisy and low-coverage labels to obtain optimal results. in this setting, existing end-to-end methods are often not flexible enough to change their internal behaviors. hence, we propose a two-step pipeline framework for this task to provide more control knobs to adjust its internal behaviors regarding the quality of the available domain dictionary at hand. our method extends span-based ner methods to unsupervised ones @xcite @xcite ; which divides the unsupervised ner task into two consecutive tasks: entity span detection and entity classification. the former extracts textual spans for the candidates of entities from a sentence disregarding the types of entities, then the latter classifies whether it belongs to any predefined entity type or none of them. distantly supervised ner often faces a low recall score problem because of the noisy and lowcoverage domain dictionary. in addition, entity names in the biological and medical domains have a high cardinality because of synonymous diversity. hence we employ positive and unlabeled (pu) learning @xcite in the pipeline to cope with the low recall score problem. specifically, we define the latter entity classification task as a partially-supervised one by taking the entity spans matched by the dictionary as positive and those not matched as unlabeled samples. then using a small portion of the positive samples, named spy, we probe the behavior of the unlabeled samples within the entity classifier. the method is based on the pu learning for binary classification proposed by @xcite and extends it to multinomial classification tasks so that it can be used for ner tasks. the contribution of this work can be summarized as follows: span-based ner methods (sohrab and miwa, 2018; yongming et al., 2022; yu et al., 2022) first detect entity spans followed by entity classification, which allows extracting overlapping or nested entities that conventional sequential labeling methods cannot easily handle. nguyen et al. (2023) employed the information bottleneck principle to enhance span-based ner. unlike these supervised approaches, our method applied a span-based method to unsupervised ner tasks. for distantly supervised ner, shang et al. 2018 proposed autoner with a "tie or break" tagging scheme to make the model more amenable to extracting false-negative examples that do not match any dictionary items, coupled with span classification. @xcite proposed bond, which leverages a pre-trained language model with a self-training. a similar method was proposed by @xcite . @xcite proposed a pu learning method to solve a partially-supervised binary classification by sending known positive spy samples into unknown samples. this allows reliably inferring the behavior of unknown samples in a classification task. our method extends the method to multinomial classification tasks so that it can be applied to general ner tasks. pu learning for distantly supervised ner was studied by @xcite based on risk minimization loss defined for binary label classification, which is different from our spy-based pu learning. ## proposed method our task is to extract mention spans that belong to one of predefined entity types {e k |@xmath0. . . k}, where k is the number of entity types, from a sentence x by using a domain-specific dictionary {d k |@xmath1. . . k} for each entity type. optionally, we assume the existence of an auxiliary dictionary d aux without specific entity type information, although we assume that each term of d aux belongs to one of the concepts of {e k }. ## results for experiments, we evaluate using bc5cdr @xcite and @xcite datasets. bc5cdr dataset contains 1,500 documents with chemical and disease entity annotations. the dataset is split into training, development, and testing data, each with 500 documents; whereas the ncbi-disease dataset contains 592 training, 100 development, and 100 testing documents with disease annotations. we used only the plain text part of the training and development dataset to train the distant and baseline ners. we utilized the testing datasets with annotation to evaluate performance. for the domain dictionaries, we used the dictionary used in autoner @xcite for the bc5cdr dataset, which can be downloaded from the author's github site foot_2 . the dictionary contains entity items for chemical and disease types and others without entity type information. on the other hand, for the ncbi-disease dataset, we used the ctd-disease dictionary foot_3 concatenated with the disease dictionary used for the bc5cdr dataset. for the baselines, we used naïve dictionary matching, bond @xcite and au-toner @xcite . we also compared our model with the supervised model (scibert + crf) to verify the performance gap between the distantly supervised and the supervised model. all models are implemented by ourselves. we should note that we did not use d aux to train the bond model because the proposed method does not allow us to use a dictionary without entity types. for the bc5cdr dataset, since we used preprocessed domain dictionary, the f1 score of the naïve dictionary match achieved a score of nearly 64%, although autoner and our distant both outperformed the naïve method. compared to the au-toner model, there was no significant difference in precision. however, our recall score outperformed autoner, which resulted in a 3.8 pp increase in the f1 score. although there is still a relatively large gap between distant and scibert+crf, our method outperformed the baselines. the second to the last rows show the performance of span detection of distant. because the number of detectable positive entities is restricted by the result of span detection, the recall score of the span detection indicates the upper bound of the recall score of distant. for the ncbi-disease dataset, we had relatively lower performances compared with the results from the bc5cdr dataset. this is because of the poor coverage of the domain dictionary, as indicated by the results of the naïve dictionary match. even though the preprocessed disease dictionary was confirmed to cover much of the disease entities in the bc5cdr dataset, the same dictionary failed to capture most of the disease entities in the ncbi-diesase dataset, even if it was appended by the ctd-disease dictionary. although the low performances, the results of autoner and distant outperformed the dictionary match result. we could not acknowledge a large performance gain for bond against the dictionary match. compared with autoner, dis-tant improves the recall by more than 10 pp, which resulted in a 3.2 pp increase in the f1 score. these results indicate that our proposed dis-tant outperformed the baselines in two datasets where we have or do not have a high-quality domain dictionary that covers most of the entities. we should note that the performance of distantlysupervised ner severely depends on the quality of the domain dictionary, and even the naïve dictionary match method with a high-quality domain dictionary would work better than any sophisticated methods if such a dictionary is unavailable. ## analysis figure 2 shows the changes in f1 scores of dis-tant evaluated on bc5cdr with respect to three different spy sampling ratios δ (0.05, 0.1, and 0.2) for three different percentile threshold θ (1%, 2.5%, and 5%). as the results illustrate, the f1 scores largely changed from 78.0% to 81.2%, suggesting that the result is sensitive to the threshold values. the optimal combination of δ and θ was obtained as 0.1 and 2.5%. we should note that, in any case, the proposed method is better than autoner. ## conclusion we propose a distantly-supervised pipeline ner named distant, which executes entity span detection and entity classification in sequence. we exploit pu learning to train our model. our model outperformed the end-to-end ner baselines without pu learning by a large margin when the quality of the domain dictionary is relatively high enough. we also confirmed that our method outperformed the dictionary match even if the coverage of the domain dictionary is quite low. overall, our method effectively increased recall scores without severely degrading precision scores. the proposed method does not require laborious human annotations and can be applied to any ner task using a domain-specific dictionary. the results shown in the original paper @xcite . because the performance of distantlysupervised ner is severely dependent on the domain dictionary used, we can not simply compare the performance of the methods if common domain dictionaries are not used.