D anish Clinical Named Entity Recognition and Relation Extraction (2023)

Abstract: Electronic health records contain important information regarding the patients’ medical history but much of this information is stored in unstructured narrative text. This paper presents the first Danish clinical named entity recognition and relation extraction dataset for extraction of six types of clinical events, six types of attributes, and three types of relations. The dataset contains 11,607 paragraphs from Danish electronic health records containing 54,631 clinical events, 41,954 attributes, and 14,604 relations. We detail the methodology of developing the annotation scheme, and train a transformer-based architecture on the developed dataset with macro F1 performance of 60.05%, 44.85%, and 70.64% for clinical events, attributes, and relations, respectively.

## introduction electronic health records (ehr) contain important information regarding the patients' medical history including diagnoses, medications, treatment plans, allergies, and test results. however, much of this information is stored in unstructured narrative text. while this information could be used to guide diagnostic decision making and treatment plans, the unstructured format makes it infeasible to fully exploit in clinical practice and research. natural language processing (nlp) algorithms could be used to transform the unstructured narrative text of the ehr into structured information * equal contribution and give medical doctors (md) a fast overview of even a medical history spanning multiple years. nlp models' ability to process and extract information from written text keeps improving with benchmark-breaking models being published on a regular basis. for example, transformer-based models such as gpt-3 @xcite , @xcite , and electra @xcite have recently shown promising results for many nlp tasks, e.g. named entity recognition and relation extraction (ner). in ner, models are trained to tag words with predefined entities and find the relations between them. in clinical ner, entities such as diseases, treatments, drugs, and tests have been extracted automatically from ehrs. however, many of the developed datasets are only in english and for specific clinical specialities or note types @xcite @xcite . this paper describes the methodology for developing the first danish clinical ner dataset. the dataset consists of text paragraphs from danish ehrs spanning multiple departments and note types. first, the paper describes the clinical dataset, the strategy for choosing entities tailored to extract important information from ehrs, and the annotation scheme. next, we train a transformer-based architecture on the developed ner dataset. ## related works the annotation schemes and extracted clinical entities and relations vary. @xcite extracted medications, their status (active, discontinued, neither), and attributes. the i2b2 2009 challenge @xcite and n2c2 2018 track 2 @xcite only extracted medications and their attributes. examples of attributes are name, dosage, mode of administration, frequency, duration, reason, strength, form, and adverse drug effects. semeval-2016 task 12 @xcite extracted time entities; event entities and their contextual modality, degree, polarity, and type; and temporal relations between time and event entities (before, overlap, before-overlap, after). semeval-2015 task 14 @xcite ) and clef ehealth 2013 task 1 @xcite extracted disorder mentions and mapped them to their umls/snomed concept unique identifier. the former also classified attributes such as the disorder's subject, course, body location, and severity, and whether it was negated, uncertain, conditional, or generic. the i2b2 2010 challenge @xcite ) extracted entities (medical problems, treatments, tests), assertions (present, absent, possible, conditional, hypothetical future, and associated with someone other than the patient), and relations between medical problem entities and each of medical problem, treatment, and test entities. the i2b2 2012 challenge @xcite extracted clinically relevant events. their type was classified as concept (problem, test, treatment), clinical department, evidentials indicating source of information, or occurrences (events that happen to the patient). polarity was classified as positive or negated, and modality as happens, proposed, conditional, or possible. temporal expressions were extracted with their type (date, time, duration, frequency), value, and modifier indicating whether the temporal expression was exact or not. temporal relations indicating the type of connections between events and temporal expressions were also extracted. ## methods this section describes the data, annotation scheme, and model used for danish clinical ner. ## results this section presents the agreement of the annotators on the gold set and the results of the danish clinical ner models. ## discussion and limitations this paper presented a methodology for developing a dataset for danish clinical ner. it presented an annotation scheme for annotation of all clinical events, their attributes, and relations that are relevant for the medical history. the dataset included text paragraphs from danish ehrs spanning multiple departments and note types. we trained and adapted pure ner deep learning models to extract clinical events (overlap match macro f1 68.20%; exact match macro f1 60.05%), attributes of clinical events (macro f1 44.85%), and relations between clinical events (macro f1 70.64%). the results are promising for danish clinical ner but need improvement. a discussion of possible improvements to the methodology, limitations, and future work is provided below. the clinical event extraction model had similar performance on all classes with accuracies between 53% (results) and 69% (anatomies). there was little contamination between classes as most errors were caused by failure to extract or false extraction of a clinical event. there was some contamination between symptoms and diseases with 12% of diseases being classified as symptoms and 5% of symptoms being classified as diseases. this supports claims by annotators that diseases and symptoms in some cases are difficult to differentiate and that extra attention must be given to differentiate these in the annotation guidelines. the attribute extraction model had large differences in performance with accuracies between 23% (non-patient) and 84% (negation). there were more misclassifications of the non-patient attribute as doubt (40%) than correct classifications. the future and doubt attributes had significant contamination between them with 25% and 11% misclassifications as the other class, respectively. the many misclassifications between nonpatient and doubt attributes, and especially future and doubt attributes, could indicate that the model would improve if the non-patient, doubt, and future attributes were merged to a single class of uncertain attributes. this would most likely not harm the usefulness of the model to mds significantly. the fact that more prior attributes were misclassified as current (41%) than correct classifications (36%) likewise indicates that these two attributes could be merged into a single class of clinical events that occurred. this would, however, decrease the usefulness of the model as it is important for mds reviewing the medical history to know if a clinical event is prior or current. the relation model extracted 93% of the "has result" relations, and 62% and 69% of the "is treated with" and "has location" relations, respectively. the differences are likely caused by the fact that the "has result" relation only connects diagnostics to results while the two other relations have three different one-way relationships. in this paper, we only explored one type of ner model and tested a limited set of architectures and hyperparameters. future work could include testing other architectures and enriching the model input with more information, e.g. the output of a text parser, which could help differentiate attributes dealing with the time-aspect. the six annotators had an average micro f1 (overlap match) of 85.62%, 81.71%, and 77.79% for clinical events, attributes, and relations, respectively. merging certain attributes and more emphasis on differences between symptoms and diseases could increase these scores. the danish clinical ner dataset is not made publicly available due to it containing sensitive information. we advise interested researchers to contact us for sharing possibilities. ## conclusions this paper presented methodology and annotation scheme for developing the first danish clinical ner dataset. the corpus consists of 11,607 paragraphs annotated for six entity types, six attributes, and three relations. the corpus was used to finetune language models which showed promising results for classifying the entities, attributes, and relations of the dataset.