Applying and Sharing pre-trained BERT -models for Named Entity Recognition and Classification in S wedish Electronic Patient Records (2021)

Abstract: To be able to share the valuable information in electronic patient records (EPR) they first need to be de-identified in order to protect the privacy of their subjects. Named entity recognition and classification (NERC) is an important part of this process. In recent years, general-purpose language models pre-trained on large amounts of data, in particular BERT, have achieved state of the art results in NERC, among other NLP tasks. So far, however, no attempts have been made at applying BERT for NERC on Swedish EPR data. This study attempts to fine-tune one Swedish BERT-model and one multilingual BERT-model for NERC on a Swedish EPR corpus. The aim is to assess the applicability of BERT-models for this task as well as to compare the two models in a domain-specific Swedish language task. With the Swedish model, recall of 0.9220 and precision of 0.9226 is achieved. This is an improvement to previous results on the same corpus since the high recall does not sacrifice precision. As the models also perform relatively well when fine-tuned with pseudonymised data, it is concluded that there is good potential in using this method in a shareable de-identification system for Swedish clinical text.

## introduction electronic patient records (epr), also called clinical text, contain valuable information about patients' symptoms, physicians' assessments, diagnoses, treatments and treatment outcomes. advancements in natural language processing (nlp) and machine learning have made it possible to use large amounts of clinical text to assist physicians and medical researchers in detecting early symptoms of disorders, predicting adverse effects of treatments, etc, see chapter 10 in @xcite . however, clinical text contains information that can reveal the identity of patients and other mentioned individuals, so called protected health information (phi). methods have been developed to detect this information and obscure it in order to protect people's identities @xcite . one important note to make is that de-identified text cannot be guaranteed to be safe to release and must still be handled with great care. a good de-identification system can, however, help facilitate an efficient anonymisation process. in this study phi refers only to the named entities which may reveal a person's identity, such as name, age and location. in this sense, detecting and identifying the phi before obscuring it is a named entity recognition and classification (nerc) problem. when it comes to data-driven nerc, models based on recurrent neural networks (rnns) and long short-term memory (lstm) networks have been successfully used for several languages @xcite . in the last two years, however, transformer-based language models such as bert have achieved stateof-the-art results in several nlp task on commonly used data sets @xcite . bert is a general-purpose language model developed by @xcite . in essence, bert is a neural network based on transformers. transformers are a type of deep learning model designed to handle sequential data, such as natural language text. since their introduction in 2017 @xcite , transformers have been widely used across a variety of nlp tasks, not least on clinical text @xcite . the benefit of transformerbased models over previous architectures is that they do not require the sequential data to be processed in order, allowing for parallelization of the training process. this has made it possible to develop large pre-trained models such as bert, which have been fitted on larger amounts of data than was previously feasible. since the first bert-model was released in 2018, several models with modified architecture and different data used in pre-training have been released, including the multilingual m-bert foot_0 . m-bert is pre-trained on texts in 104 languages, including swedish. in 2019, the national library of sweden released a swedish bert model, kb-bert foot_1 , pre-trained exclusively on swedish texts. to use a pre-trained bert-model for a downstream task, it needs to be fine-tuned for that task. both kb-bert and m-bert have shown success in the nerc task for swedish when fine-tuned with the publicly available stockholm-umeå corpus consisting of swedish texts from the 1990's @xcite . to our knowledge, however, no previous attempt has been made at using these models for nerc in swedish epr data. in this study, we attempt to improve nerc performance on swedish electronic patient records by fine-tuning kb-bert and m-bert with domainspecific data. more specifically, our aim is to achieve high recall, which is a priority in the deidentification task, without sacrificing precision. a risk with de-identification methods based on machine learning is that a model trained on sensitive data could be re-engineered, revealing the data. in a bert-model, there are no links between words in the vocabulary, making it infeasible to retrieve the patient records used for fine-tuning. however, due to names and other personal identifiers appearing in the model's vocabulary, there may be legal issues with releasing a model fine-tuned on patient records. therefore, in an additional experiment, the models are fine-tuned using pseudonymised patient records to see how nerc performance on authentic records is affected. the outline of this paper is as follows. first, section 2 presents some previous studies on nerc in clinical text and specifically previous results on the data set at hand. then, section 3 describes the data used in this study, gives some more detail on the two bert models and goes through how the fine-tuning and evaluation are performed. section 4 presents the results for both models. finally, the results are discussed in section 5. ## related research there are several publicly available bert-models pre-trained specifically for the biomedical and clinical domains. in 2019, @xcite released biobert foot_2 , a bert-model pre-trained on pubmed articles as well as wikipedia articles and books. the authors present an f 1 -score of approximately 0.87 on the commonly used i2b2 2010 data set for clinical text nerc. in a different 2019 project, @xcite continued to pre-train the pretrained bert-model released by @xcite on pubmed abstracts and clinical notes. this model, named bluebert foot_3 , reaches an f 1 -score of approximately 0.77 on the i2b2 data set. the same year, @xcite released clin-icalbert foot_4 pre-trained on clinical texts but also specifically on discharge summaries. the combined bio+discharge summary model reaches an f 1 -score of 0.88 on the i2b2 2010 data set. all of these models are only pre-trained on english texts. for non-english clinical text nerc, some advancements were made in connection to the 2019 shared task meddocan which consisted of performing nerc on spanish electronic patient records with annotated phi. in a submission to the contest @xcite used m-bert, which is also pre-trained on spanish text @xcite with a decoding crf layer for token classification. they also applied some post-processing techniques, achieving f 1 -score and recall of approximately 0.93. when it comes to swedish, several attempts have been made at performing nerc on the annotated data set of electronic patient records stockholm epr phi corpus. in one study by @xcite the authors extended the annotated data set with data generated using a semi-supervised learning method with the aim of increasing recall without sacrificing precision. the highest recall reported was 0.8920, at which point the precision was 0.9420. these results were achieved using a conditional random field (crf) model. @xcite managed to increase the recall to 0.9209 using the same model by under-sampling negative tokens, thus tokens not belonging to a phi. however, this came at the cost of significant decrease in precision to 0.8819. regarding the application of models trained on pseudonymised clinical data for nerc on authentic data, there is a study by @xcite where the authors achieved at highest recall of 0.5510 using a lstm network. the experiment was repeated with a classic crf and the recall decreased to 0.4983. ## data and methods this section describes the data, tools and methods used in this study. first, the epr data set is described in section 3.1. then, section 3.2 describes the bert-models used and how they were fine-tuned. lastly, section 3.3 describes how the models were evaluated in a number of experiments. ## results the results presented in this section were achieved with the best hyper-parameter values found, see section 3.3. note that the hyper-parameter optimization was not exhaustive and this may have significant effects on the results. ## discussion and conclusions the results show that the fine-tuned kb-bert achieves recall on the same level as that reported in @xcite on the same data set, see