Cross-domain G erman Medical Named Entity Recognition using a Pre-Trained Language Model and Unified Medical Semantic Types (2023)

Abstract: Information extraction from clinical text has the potential to facilitate clinical research and personalized clinical care, but annotating large amounts of data for each set of target tasks is prohibitive. We present a German medical Named Entity Recognition (NER) system capable of cross-domain knowledge transferring. The system builds on a pre-trained German language model and a token-level binary classifier, employing semantic types sourced from the Unified Medical Language System (UMLS) as entity labels to identify corresponding entity spans within the input text. To enhance the system’s performance and robustness, we pre-train it using a medical literature corpus that incorporates UMLS semantic term annotations. We evaluate the system’s effectiveness on two German annotated datasets obtained from different clinics in zero- and few-shot settings. The results show that our approach outperforms task-specific Condition Random Fields (CRF) classifiers in terms of accuracy. Our work contributes to developing robust and transparent German medical NER models that can support the extraction of information from various clinical texts.

## introduction information extraction from the large volume of unstructured text generated in hospitals and clinics has the potential to facilitate clinical research and enhance personalized clinical care. especially the narrative notes, such as radiology reports, discharge summaries and clinical notes provide a more detailed and personalized history, assessments, medication and symptoms, offering a better context for clinical decision-making @xcite . in the field of natural language processing (nlp), the problem of automatically and accurately extracting specific terms from text data is approached as a named entity recognition (ner) task. ner methods ranging from rule-based to deep learning methods are the core technologies for automatically identifying medical instances from clinical narratives, such as diseases, diagnosis, drugs, and treatments @xcite @xcite @xcite @xcite . building clinical ner systems for non-english languages, e.g. german in our case, is challenging due to data scarcity. only a few real-world annotated resources in german are publicly available @xcite . this problem can be overcome by cross-domain transfer learning, where models transfer knowledge learned from data-rich relevant domains to domain-specific target tasks with less or no annotated data @xcite @xcite @xcite . we propose a simple but effective transfer learning framework based on a german bert foot_0 encoder that is given a prompt consisting of a semantic type from umls semantic network foot_1 , followed by a separator token and the medical text, e.g. " @xcite clinical drug @xcite zofran 4mg for nausea.". on top of the encoder is a binary token classifier which predicts a probability for each token to determine whether it belongs to the given semantic type or not. our approach, denoted as bert-sner (code foot_2 ) and depicted in figure 1 , is based on three insights from recent research in transfer learning: i) pre-trained language models (plms), e.g. bert @xcite , facilitate downstream tasks in specific domains @xcite @xcite . ii) prompting plms is becoming increasingly popular for solving lowresource ner tasks, as it can successfully exploit generic knowledge learned in the pre-training tasks @xcite @xcite . iii) the unified medical language system (umls) metathesaurus @xcite is a useful knowledge source for mining medical terms in both biomedical and clinical documents @xcite @xcite @xcite . the lack of domain-specific annotations is our motivation to develop models that can easily be adapted after pre-training on non-domain-specific annotated data. in our transfer learning experiments, we first derive training data from the open-source muchmore corpus 4 to train bert-sner. muchmore consists of german abstracts from 41 medical journals and entities are annotated with 134 umls semantic types (archive 2001 5 ). for more details on the annotation process of this corpus, please refer to @xcite . after that, we map the entity labels of the two clinical target tasks to umls semantic types to be consistent with the annotations in the muchmore corpus and perform zero-and few-shot experiments with 10, 50 and 100 shots for the two clinical target tasks. the contributions of our work can be summarized as follows: 1) our approach addresses lowresource german clinical ner tasks effectively. 2) we identify effective ways of transferring opensource medical knowledge for improving the performance of german clinical ner models. ## approach our approach explores the feasibility of knowledge transfer between different datasets by incorporating umls semantic term types to unify the entity labels. we compare the resulting ner system to a baseline architecture of bert encoder combined with a task-specific conditional random fields (crf) classifier @xcite , i.e. bert-crf @xcite @xcite @xcite . in contrast to bert-crf models, bert-sner does not require the introduction of new taskspecific parameters for solving the cross-domain target tasks, which benefits few-shot fine-tuning, while the bert-crf models fail if there are less than 100 samples in the target domain available. entity type description semantic type diag a disease, a symptom or a medical observation that can be matched with the german modification of the international classification of diseases. sign or symptom; disease or syndrome; finding treat a diagnostic procedure, an operation or a systemic cancer treatment that can be found in the operation. diagnostic procedure; therapeutic or preventive procedure med a pharmaceutical substance or a drug that can be related to the anatomical therapeutic chemical classification system. pharmacologic substance; clinical drug we use two datasets from different german clinical domains as target tasks: the berlin-tübingen-oncology corpus bronco @xcite and ex4cds @xcite . bronco consists of german discharge summaries for cancer patients annotated with medical entities of interest, such as medication (med), diagnosis (diag) and treatment (treat). ex4cds is a corpus of textual explanations for supporting system predictions of three possible outcomes (rejection, infection, graft failure) after kidney transplantation in the nephrology clinic. it focuses on entities that indicate the patient's health state as well as laboratory measures after a process. ## results and discussion the binary classifier of bert-sner predicts a probability for each token in the input sequence affected by the preceding semantic type and the sentence that follows. the classification result for each token is determined by setting a threshold. if the predicted probability is less than the threshold, the token is assigned to class 0, otherwise to class 1. the lower the threshold, the higher the false positive prediction rate, and conversely, a high threshold may result in a lower recall rate. we determine the threshold value for each entity label by finding the optimal precisionrecall trade-off on the validation set of both target tasks based on the calculation results using the sklearn.metrics.precision_recall_curve function. figure 2 tables 5 and 6 present f-scores per individual entity types. when comparing the results between zero-and few-shot, we find some semantic types that can not be generalized well to the target domains, such as (treat -> diagnostic procedure) in bronco vs (diaglab -> diagnostic) in ex4cds, and (med -> clinical drug) in bronco vs (medication -> clinical drug) in ex4cds. in contrast, domain-specific entity types, healthstate, labvalues and process, which are unseen or infrequent in the source task, can benefit the most from the increasing number of shots in the self-domain. these results suggest that in future work, there is a need to investigate more deeply the semantic differences of domain-specific entities matched to the same unified semantic type when experimenting with the bert-sner system for more diverse clinical domains. in addition, we need to examine more the impact of the amount of training data from the muchmore corpus on individual entity types in target tasks. ## related work our work focuses on solving low-resource ner tasks in the clinical domain leveraging additional resources from related domains, and in non-english languages. a common solution is to perform downstream tasks for non-english languages, especially typologically close to english through cross-lingual transfer from large-scale pre-trained multilingual bert models @xcite @xcite @xcite or english language models @xcite . @xcite and @xcite attempt to use synthesised data through translation from english resources @xcite to train a german medical ner model. most of the previous works in this field have focused on a single task and it's unclear if these task-specific approaches can easily be extended to other clinical datasets with different label sets. sequence-to-sequence (seq2seq) plms with prompt-based methods in another line have been shown to be useful for solving low-resource ner problems @xcite @xcite @xcite . other previous work of this line @xcite @xcite utilized ner data from a resource-rich domain to fine-tune the seq2seq models on ner tasks before applying them to low-resource ner tasks. although no new parameters are introduced to the pre-trained seq2seq language model when formulating the ner tasks in a generative framework, these methods require much effort for finding the optimal prompts and framework to transform an input sequence of tokens (words or characters) into an output sequence of entity labels. unlike seq2seq ner models, our bert-sner model uses semantic types as prompts in front of the input directly, and the binary classifier is more efficient in terms of computational requirements, inference time and post-processing needs. ## conclusion our results suggest that transferring knowledge from publicly available medical resources with bert-sner is more effective than with bert-crf in low-resource scenarios. the overall benefit of the bert-sner in real-world use cases is that it can be used as an initial model to effectively develop domain-specific models in a variety of clinical applications, as it requires much less fine-tuning data than training a ner model from scratch. in future work, we will explore transfer learning more to generalize bert-sner to more different clinical ner tasks in low-resource situations. to apply bert-sner to new clinical applications without annotated samples, we will use active learning strategies such as least confidence oracle @xcite to query the most informative samples to obtain annotations for fine-tuning.