Creating Ontology-annotated Corpora from W ikipedia for Medical Named-entity Recognition (2024)

Abstract: Acquiring annotated corpora for medical NLP is challenging due to legal and privacy constraints and costly annotation efforts, and using annotated public datasets may do not align well to the desired target application in terms of annotation style or language. We investigate the approach of utilizing Wikipedia and WikiData jointly to acquire an unsupervised annotated corpus for named-entity recognition (NER). By controlling the annotation ruleset through WikiData’s ontology, we extract custom-defined annotations and dynamically impute weak annotations by an adaptive loss scaling. Our validation on German medication detection datasets yields competitive results. The entire pipeline only relies on open models and data resources, enabling reproducibility and open sharing of models and corpora. All relevant assets are shared on GitHub.

## introduction a major reoccurring pain point in natural language processing (nlp) remains the issue of lacking resources regarding text corpora, including adequate annotation information in particular. especially in medical and clinical nlp environments, the situation is notoriously difficult due to privacy and legal restrictions on data use and sharing; rendering efforts to share open datasets from the clinical domain complex and cumbersome. yet there are notable and important attempts to provide text resources close to the clinical domain to the public research domain, e.g. mimic-iii/-iv @xcite . while these resources are extremely valuable, a single annotated dataset is governed by pre-defined parameters: first, the actual language, which may not align with the desired target language, and may pose another challenge regarding cross-lingual transfer. second, the domain-dependent linguistic text properties and styles may vary from corpus to corpus. third, the provided annotation data are usually tied to a certain ontology in entity linking, or label classes in named-entity recognition (ner). forth, even if the label classes appear identical across multiple corpora, the underlying annotation guidelines that were employed as rulesets for annotation decision-making are usually not identical or consistent with annotation guidelines from other datasets. while the language and the text style are inherently fixed, new annotation data for a given corpus could be manually created if a custom annotation guideline is needed. however, a manual annotation is costly, resource-intensive and may remain non-reproducible to a certain degree due to the human-provided input. hence, creating such an alternative annotation layer is not feasible in practice. in this work, we investigate the practicality of applying a fully unsupervised approach for annotated data acquisition in the medical context, yielding a corpus that is subsequently used as training material for an ner model. to achieve this, we compose several steps to obtain our final results. our approach combines two public knowledge sources, wikipedia foot_1 and wikidata @xcite 3 , to extract text data and annotation information, whereas the core annotation ruleset can be defined by leveraging the graph-like ontology structure of wikidata. the crafted dataset is used to train a conventional ner model. to demonstrate the feasibility of our approach, we evaluate our trained ner models on several external public datasets. since our approach is in particular of interest for medium-to low-resource languages, we choose german as our non-english target language, but it is also motivated by the fact that external annotated datasets are available in that language for a final evaluation. due to its simple availability, quality and text size, wikipedia has been subject to nlp research in the past decade, in particular exploited further to obtain ner corpora. therefore, it has been applied in numerous works for generic ner @xcite @xcite @xcite @xcite @xcite @xcite , mostly using wikipedia text or features, while others also include structured knowledge bases like dbpedia @xcite or freebase @xcite . similar to our work, jiang et al. 2021 also cover english biomedical ner for weak annotation data by modifying the loss function to be "noise-aware" and applying annotation imputation. yet they also include a set of small, manually fully-annotated labels, and use pubmed texts with automatic dictionary-based label synthesis instead of wikipedia resources. regarding open domain ner, liang et al. 2020 tackle the challenge by a similar two-stage self-training approach using wikidata, but do not further cover any custom wikipedia parsing. to the best of our knowledge, no work using wikipedia and wiki-data has been reported so far in the german, medical domain. ## results regarding our implementation, significant portions are re-purposed from existing work @xcite . to assess our proposed approach in medical, non-english ner, we mimic a medication detection task in german texts due to the availability of public datasets with annotations including label classes semantically related to drug or medication, namely bronco150 @xcite , cardio:de (richter-pechanski et al., 2023), gptnermed @xcite , gernermed++ @xcite , and ggponc 2 @xcite (with short, fine annotation layer). to address the medication detection task, our simple entity selection strategy hereby filters all wikidata entries that have an atc code assigned through the wikidata property p267 to eventually obtain a weakly-annotated corpus. based on this corpus, we fine-tune an ner model with the huggingface transformers (wolf et al., 2020) library for different ω unk scalars while ω pos /ω neg fine-tuning using token weighting input data (for inference) (without annotations) ## discussion and limitations while a conclusive verdict is not feasible based on the pure evaluation scores due to the diverse and incoherence issues of the external datasets, the fact ner model is reported to achieve .91 f1-score on its test set on the clinical_drug label class, likewise cardio:de achieves .85/0.81 f1-scores for the activeing/drug label classes. however, major disagreements are reported in cross-corpus model transfer. for instance, (richter-pechanski et al., 2023) report a .21 f1-score for the drug class when applying the ggponc 2 ner model to the held-back part of the cardio:de corpus, highlighting certain innate limitations when comparing f1-scores across different datasets and annotation guidelines, as well as the need for the use of multiple datasets during evaluation. as for another, less severe instance, @xcite and @xcite report .73/.72. f1-scores respectively on the bronco150 corpus for the medication label class, hinting towards more consistent mutual annotation agreements. other factors in ner are not further addressed in this work, such as efforts towards support for nested entities or discontinuous annotations or the support for label classes beyond medication detection. the latter aspect may be achieved by the use of an updated sparql query definition for certain label classes that align well to the annotation schema from wikipedia articles but may fail for other label classes like strengthor frequencyrelated entities which may not be covered well by wikipedia or wikidata. in this regard, potential limitations within the wikidata knowledge base are not investigated that may influence the quality of our results in other domains. other limiting factors such as the quality of the pre-trained language models are not quantified in isolation. however, in general, the effective use of more sophisticated sparql queries for entity selection may unlock further potential gains, as well as its application in other languages and domains since our method is not inherently bound to the medical field. yet, these aspects only serve as motivation for future work. ## conclusion in this work, we demonstrated an unsupervised approach for creating an annotated dataset for medical ner for the german language defined by the wiki-data ontology structure using exclusively open data resources. the proof of concept of the proposed method in practical scenarios was shown on a set of external datasets, yielding surprisingly well results. we also discussed further potential but currently underexplored factors such as improved sparql queries as future work. relevant assets are published on github 6 , including a web interface that enables external users to create new corpora from custom sparql queries for independent experiments.