Building A G erman Clinical Named Entity Recognition System without In-domain Training Data (2024)

Abstract: Clinical Named Entity Recognition (NER) is essential for extracting important medical insights from clinical narratives. Given the challenges in obtaining expert training datasets for real-world clinical applications related to data protection regulations and the lack of standardised entity types, this work represents a collaborative initiative aimed at building a German clinical NER system with a focus on addressing these obstacles effectively. In response to the challenge of training data scarcity, we propose a Conditional Relevance Learning (CRL) approach in low-resource transfer learning scenarios. CRL effectively leverages a pre-trained language model and domain-specific open resources, enabling the acquisition of a robust base model tailored for clinical NER tasks, particularly in the face of changing label sets. This flexibility empowers the implementation of a Multilayered Semantic Annotation (MSA) schema in our NER system, capable of organizing a diverse array of entity types, thus significantly boosting the NER system’s adaptability and utility across various clinical domains. In the case study, we demonstrate how our NER system can be applied to overcome resource constraints and comply with data privacy regulations. Lacking prior training on in-domain data, feedback from expert users in respective domains is essential in identifying areas for system refinement. Future work will focus on the integration of expert feedback to improve system performance in specific clinical contexts.

## introduction clinical named entity recognition (ner) plays a central role in extracting valuable information from medical texts as essential features for developing clinical decision support systems. in this work, we concentrate on the german language and its application within clinics in germany. clinical documents can originate from a variety of sources. each source has its unique characteristics, making it challenging to develop a one-size-fits-all ner system @xcite @xcite @xcite . in developing german clinical ner systems, challenges arise when strict privacy rules are applied to data sources and the complexities associated with expert annotation of training data @xcite . previous related research efforts in german language have tackled these challenges using a range of techniques, from rule-based approaches to transfer learning methods in low resources scenarios @xcite @xcite . in this paper, we describe the development and assessment of an adaptive german clinical ner system without prior training on in-domain data. this goal is motivated by the principles of interactive machine learning (iml) @xcite , particularly when dealing with the challenges of annotating complex medical texts. in this work, we investigate innovative transfer learning techniques and support the evolution of dynamic annotation schemas by engaging expert users from the medical field. we aim to address the constraints posed by limited data resources. the system has been developed as part of a cooperative project involving a machine learning lab and a university institute for medical informatics. our system is equipped with a dedicated web-based user interface (web-ui) for correcting system-generated annotations, which is instrumental in our case study involving cardiology. qualified experts who are granted access to review the specific documents sourced from the hospital's internal database in medical informatics can utilize this tool to interact with system-generated outputs via a standalone website. the main objective of the case study is to conduct a comprehensive analysis of the performance of the ner system when applied to a non-distributed dataset without violating privacy regulations. this analysis helps identify areas where adjustments are needed to refine the annotation schema and offers valuable insights to guide further fine-tuning of the model's performance to maintain its relevance to domain-specific nuances, context, and entity variations. figure 1 displays our collaborative research environment. all system-related modules and model checkpoints are deployed at the hospital endpoint to ensure strict data security. the main contributions of our work in the field of german clinical ner are as follows: this forms the basis for a dynamic and expandable semantic annotation schema as new clinical use cases emerge over time. as the ner system is deployed in specific clinical contexts, we can refine the annotation schema to align with the unique requirements of each use case it serves, ensuring the system's effectiveness and relevance across diverse clinical scenarios with minimal modification needed. our collaborative effort is achieving ner system's adaptability across clinical domains and improving the robustness of the ner system's performance through the engagement of domain experts in applications. ultimately, we aim to address the impact of strict privacy rules on data accessibility and annotation quality and contribute to research on the development of clinical information extraction systems in the german healthcare sector. codes and demonstration are available in github repository https://github.com/ sitinggz/bert-sner-cardio . ## related work efforts to address the scarcity of in-domain training data in german ner training have led to the exploration of two main strategies. one strategy involves translating annotated english corpora, such as the n2c2 dataset @xcite and ddi dataset @xcite , to synthesize domainrelated german training datasets @xcite . however, the accuracy and feasibility of the resulting ner models remain limited by nuances and contextual differences between languages and clinical domains, which affect their applicability to real german clinical texts. another line of work involves the manual annotation efforts of curating generic medical datasets @xcite by annotating on open-source corpora derived from medical journals. furthermore, there are ongoing endeavours to develop domain-specific german clinical data sets that adhere to the english reference guidelines @xcite . task-specific datasets, like roller et al. (2022) are rare but valuable for enhancing ner model performance by capturing domain-specific nuances. however, their adoption is hindered by the resource-intensive manual annotation workload. llorca et al. (2023); liang et al. (2023a) attempted to generalise various entity labels from different annotated german datasets to achieve a reasonable amount of cross-domain training data. while @xcite introduced harmonised versions of german medical corpora through the big-bio framework @xcite , contributing to the creation of common metadata sets for improved ner model performance across different medical text sources. however, the study found limited generalisation of ner models across different datasets. it remains challenging to effectively leverage diverse medical corpora for entity recognition in german texts. @xcite augmented training data by mapping the original entity labels from source datasets to semantic types based on the ulms ontologies and utilized a german bert encoder with a binary token classifier to efficiently recognize medical entities by prompting with various semantic types followed by the same medical text. the novel training framework has shown effective cross-domain knowledge transfer and enhanced performance in low-resource german ner tasks. while previous approaches offer potential solutions for the data scarcity issue, none have been able to develop a cross-domain adaptive ner system. our work represents a step towards a more efficient and flexible long-term solution, as we combine the ner training framework from @xcite with a multilayered semantic annotation schema, specifically targeting ner challenges within clinical information extraction and adapting to evolving clinical use cases. ## case study in the absence of an annotated test dataset that closely resembles most real-world scenarios, evaluating the system's performance necessitates user evaluation. in our case study, the ner system is deployed on a german clinical corpus from the cardiovascular domain, where doctor letters undergo anonymization and time-shifting to comply with privacy regulations @xcite . expert annotators can review and modify the system's output through the web-ui (see figure 3 and 4), facilitating ongoing refinement based on valuable user feedback within the applied domain. this user-centric evaluation approach ensures that the system is continuously optimized to meet the specific needs and requirements of the clinical context, thereby enhancing its practical utility and effective-figure 4 : ui snippet of annotation revision. annotators are requested to revise the system-generated annotations by semantic groups. tokens from one sentence (the 8th sentence in this example) are placed in the header of the revision table, one token per column. the annotators can make changes on the entity types labelled to a single token (selecting one column) or to multiple tokens (selecting multiple columns) by clicking on the most appropriate entity type (diagnostic procedure in the second row) under the selected semantic group (procedure in the example). label o: null indicates that the token is not recognised as an entity according to the current annotation schema. ## conclusion in summary, our work aims to overcome the challenges involved in developing a ner system for german clinical nlp applications without indomain training data. we propose using advanced transfer learning methods and focusing on direct adaptability to new datasets with crl and msa. our case study, which involves close collaboration with domain experts in specific clinical applications, yields invaluable insights that contribute to the overall improvement of the system. these insights are essential for tailoring the system to meet the specific information extraction requirements in target domains. our work represents a significant advancement in clinical information extraction, alleviating limitations associated with data scarcity and cross-domain transferability. in future research, we will concentrate on incorporating expert feedback into the adaptation pipeline and models' finetuning, ultimately creating a continuous learning ecosystem tailored to the distinct clinical context. this effort aligns with our primary objective of advancing ner technology to effectively address challenges related to data scarcity, medical text diversity, and ever-changing label sets.