A Meta-dataset of G erman Medical Corpora: Harmonization of Annotations and Cross-corpus NER Evaluation (2023)

Abstract: Over the last years, an increasing number of publicly available, semantically annotated medical corpora have been released for the German language. While their annotations cover comparable semantic classes, the synergies of such efforts have not been explored, yet. This is due to substantial differences in the data schemas (syntax) and annotated entities (semantics), which hinder the creation of common meta-datasets. For instance, it is unclear whether named entity recognition (NER) taggers trained on one or more of such datasets are useful to detect entities in any of the other datasets. In this work, we create harmonized versions of German medical corpora using the BigBIO framework, and make them available to the community. Using these as a meta-dataset, we perform a series of cross-corpus evaluation experiments on two settings of aligned labels. These consist in fine-tuning various pre-trained Transformers on different combinations of training sets, and testing them against each dataset separately. We find that a) trained NER models generalize poorly, with F1 scores dropping approx. 20 pp. on unseen test data, and b) current pre-trained Transformer models for the German language do not systematically alleviate this issue. However, our results suggest that models benefit from additional training corpora in most cases, even if these belong to different medical fields or text genres.

## introduction recently, an increasing amount of medical text datasets for the german language with semantic annotations has been released to the public @xcite . these corpora come in unequal data formats and with widely varying definitions of annotated entities, e.g., based on ontologies like the umls @xcite , top level hierarchies in snomed ct @xcite , or other medical terminologies such as icd-10. the employed annotation guidelines have at the same time, adapted german versions of widely used transformer models have become publicly available @xcite , more recently also specific to biomedical texts @xcite . while these models have been evaluated on many individual datasets, their performance on truly unseen data remains unknown. to enable cross-corpus evaluations, we create a meta-dataset of german medical corpora by harmonizing them under the schema proposed in the big-bio framework @xcite , as illustrated in fig. 1 . this way, we also make the datasets easily available to the community as data loaders in the hugging face library @xcite . an example for this schema is depicted in fig. 2 . while schema harmonization addresses the issue of syntactic interoperability, the semantics of annotated entities may still differ, as definitions of entity classes have been derived from different medical ontologies. therefore, we propose two possible alignments of labels across the four german medical corpora and conduct a series of experiments, evaluating different combinations of training corpora and pre-trained transformers. in this work, we focus on the entity annotations related to medications, as these are the only ones that can be aligned consistently across all corpora. their definitions are still partly extracted from different medical ontologies. our goal is to determine whether multiple datasets with similar annotated entities contribute to creating models that can be used outside the domain they were trained in. here we refer to the domain of a corpus broadly as the set of characteristics conforming it, mainly text genre, medical field, and annotation policy. such experiments have been successfully conducted for the english language and resulted in robust ner taggers, albeit for entity classes different from the more clinically motivated german-language corpora @xcite . to the best of our knowledge, no such harmonization has been conducted for the german language and clinical entity classes. the remainder of this work is organized as follows: in section 2, we review literature on german clinical corpora and biomedical data harmonization. in section 3, we describe our methods, i.e. the corpora used, data harmonization steps, and performed cross-corpus evaluation. we present our results in section 4 and discuss them in section 5. our work concludes with an outlook in section 6. ## related work in the following, we set our contribution in the context of related work. ## materials and methods in the following, we present the characteristics of each corpus and an overview of the harmonization and annotation alignment processes. we provide a description of the experimental setup and the evaluation methods used to analyze the results. ## results the seqeval (traditional) f 1 scores of the first set of experiments are shown in we use abbreviations with the first three letters to refer to the datasets, i.e. grascco is gra. we recall the experiments by the row number in ## discussion in this section, we discuss our findings and perform a fine-grained error analysis. ## conclusion and outlook reaching agreements and establishing standards for clinical entity annotation is vital to facilitate inter-corpus operability. so it is adhering to similar formats and schemas to structure the informa-tion. this can help to combine the sporadically released open medical text datasets for non-english languages to build more robust models. our work aimed to support this goal by harmonizing all currently available, semantically annotated german medical corpora within the bigbio framework and make the implemented data loaders available to the community. our experiments show that the currently available corpora in german serve for poorly generalizable models. our results also suggest that mixing multiple corpora in training is beneficial on single test splits and widely improves robustness, thus highlighting the importance of easing crosscorpus integration. the comparison of different pre-trained transformers does not shed conclusive results. both general-purpose and biomedicalspecific instances seem to perform similarly on unseen data. the results presented in this paper correspond to a single training iteration on all combinations of the pre-defined train-test splits of the data. for future work, we consider performing proper crossvalidation experiments by dividing each corpus into folds and using all resulting combinations in order to obtain more stable results and confidence intervals. however, this approach increases the number of experiments from 34 trained models to 340. we have obtained preliminary results of such evaluation, and the findings are consistent with the ones presented in this work. other options for further research include extending our comparison of multiple transformers, or even considering generative approaches to ner. investigating whether other label alignments are meaningful once more comparable datasets become available would help to reinforce our results outside of medication annotations. given the currently available corpora, the only other entities that might be comparable are the short version of diagnosis/pathology and therapeutic procedure from ggponc 2.0 with diagnosis and treatment from bronco150, respectively. however, here the differences in semantics are even more pronounced than in the case of medication. annotation campaigns using unpublished corpora, which concern other medical fields and text genres, suggest that we might be able to harmonize other semantic classes in the future. for instance, the condition category in the fine-grained annotation scheme proposed by @xcite for clinical notes in nephrol-ogy roughly corresponds to the findings class in ggponc 2.0. to conclude, our study calls for more representative large german clinical corpora to generate robust ner taggers that can be used for real-world scenarios, together with a consensus on the semantics and annotation guidelines to equate labeled entities through the datasets.