Deidentifying a N orwegian Clinical Corpus - an Effort to Create a Privacy-preserving N orwegian Large Clinical Language Model (2024)

Abstract: The study discusses the methods and challenges of deidentifying and pseudonymizing Norwegian clinical text for research purposes. The results of the NorDeid tool for deidentification and pseudonymization on different types of protected health information were evaluated and discussed, as well as the extension of its functionality with regular expressions to identify specific types of sensitive information. The research used a clinical corpus of adult patients treated in a gastro-surgical department in Norway, which contains approximately nine million clinical notes. The study also highlights the challenges posed by the unique language and clinical terminology of Norway and emphasizes the importance of protecting privacy and the need for customized approaches to meet legal and research requirements.

## introduction today with the european general data protection regulation (gdpr) law, and the norwegian law for the processing of personal information lov om behandling av personopplysninger (personopplysningsloven) it is notoriously difficult to get access to electronic patient record texts to perform research. first of all, one needs to submit an application to the norwegian regional committees for medical and health research ethics (rek) and after that approval, one needs to ask personvernsombud (pvo) at the local hospital to access the data. one way to make it easier is to process the data before using it for research by sanitising the data, that is to deidentify and then pseudonymise it, very similar to what has been carried out in @xcite . ## related research the field of deidentifying and pseudonymizing clinical text for research purposes has been a subject of extensive research, with much of the previous work based on shared tasks related to datasets such as i2b2 (now n2c2). in @xcite , and most studies model deidentification as a named entity recognition (ner) task, @xcite . making these datasets available to researchers has facilitated a lot of progress on this task over the years; starting with traditional nlp methods @xcite , then with deep learning using word embeddings, @xcite , then more recently to deep learning methods using contextual embeddings or large language models, @xcite . these benchmark datasets partially solved the need for standardised evaluation metrics to facilitate the comparison and improvement of different deidentification methods. regarding the generation of pseudonyms or surrogates there is a nice description carried out by @xcite where the authors elaborate on the replacement at different generalisation levels. more recent research have shown promising results in the field. among others, @xcite conducted a study on the automatic deidentification of medical documents in spanish. the study developed two different deep learning-based methodologies for the task and also developed a data augmentation procedure to increase the number of texts used to train the models. @xcite carried out deidentification and pseudonymisation of 17.9 gb of swedish clinical text using a swedish clinical bert model called swedeclin-bert. the process of deidentification took over two weeks, while pseudonymisation, replacing the found entities with pseudonyms or surrogates, was ready in a couple of days since it is a rule based approach. in total 83,914,340 sensitive entities were found in 49,715,558 sentences encompassing 2.8 billion words. @xcite reviewed the recent research for ensuring the correct usage of regular expressions, which is crucial for identifying specific types of sensitive information. however, there is a growing focus on addressing the challenges posed by the diverse and nuanced nature of clinical narratives, including variations in language use, context, and medical jargon. for instance, different institutions have different standards on how they treat their electronic health record narratives. this highlights the importance of documenting deidentification processes in diverse contexts, such as norway in this case. ## application of method a server republic of gamers with the operating system debian linux installed and equipped with two gpus (asus geforce rtx 3090), 64 gb of internal memory (ram) (2 x 32gb 3200 mhz ddr4), 8 tb gen4 x4 m.2 nvmes ssd hard disc etc and not connected to the internet was used for the deidentification task. the server was also encrypted and situated in a server room only accessible to researchers who were specially authorised to work with the data and signed a confidentiality agreement. the server also remains offline during the project. the process to deidentify and pseudonymise the corpus took approximately one week. the results can be seen in the evaluation of the current version of nordeid is based on a comparison between human annotations and predictions made by the model. the evaluation dataset consists of 19 clinical notes that encompass about 13,000 tokens, annotated in the conll format, a popular schema for text annotation used in natural language processing. the annotations target various entities of phis listed in the performance of the model is quantitatively measured using standard metrics: precision, recall, and f 1 -score. precision measures the proportion of correct positive identifications made by the model, recall assesses the model's ability to identify all relevant instances, and the f 1 -score provides a harmonic mean of precision and recall, offering a balance between the two. ## analysis the evaluation results are presented in a detailed format as shown in the model achieves its highest f 1 -scores with full_date (0.76), phone_number (0.73), and first_name (0.68). these results indicate a strong ability to recognise and accurately tag full dates and names in clinical notes. although the model shows no capability in correctly classifying family_number, organisation, and so-cial_security_number, nordeid was able to identify those entities as phis, according to the confusion matrix (figure 2 ). by looking at the average scores (micro average, macro average and weighted average), the model demonstrates moderate effectiveness with a weighted average f 1 -score of 0.53. while this indicates potential utility in a clinical setting, there is notable room for improvement. figure 2 shows the entity confusion matrix. each row of the matrix represents the instances in an actual class, while each column represents the instances in a predicted class. for phi types such as health_care_unit, full_date, first_name, and last_name, there is a higher number of true positives. this shows a strong alignment with human annotations. certain types of phis, such as organisation and social_security_number, have higher false positives and false negatives. this suggests the challenge in classifying these phis correctly. there are also a high number of misclassifications between location and health_care_unit, as well as between date_part and full_date. this could be due to the similarity in format and context between these types of phis. ## challenges a major challenge was encountered in the beginning when trying to identify and classify sensitive data. this required a detailed understanding of the type and extent of sensitive information applicable to norwegian texts, which included a wide variety of personal identifiers and confidential medical details. the difficulty was further increased by the subtle differences in language and clinical terminology that are unique to norway. an example of illustrating this challenge is personnummer, which refers to a norwegian social security number. this number is highly sensitive since it is unique to each person and contains information such as date of birth and gender. the model needed to distinguish between actual personnummer instances and other similar-looking numerical sequences. there are also different ways to represent this number depending on how healthcare professionals annotate notes. for example, a personnummer of 01010112345 can be rewritten as 010101-12345 or 010101 12345 or 01 jan 01 12345. the task of data management, such as cleaning and formatting, can be challenging. this included ensuring the data was in the correct text format, linking the data correctly, and dealing with issues when tokenizing norwegian clinical texts. these processes were essential to ensure the data was accurate and reliable before using them with the model. annotation of clinical texts requires a lot of resources. this process required both time and expertise, particularly in the medical domain. therefore, providing enough resources for annotation is a major challenge in ensuring the overall efficiency and precision of the deidentification procedure. implementing the model to deidentify norwegian clinical texts is also computationally intensive. it requires substantial computational resources, including processing power and memory, which was a limiting factor. operating in an offline environment also introduced additional constraints, particularly in setting up the environment for model training and debugging code. this scenario limits the ability to take advantage of cloud computing resources and requires reliance on local computational capabilities. finally, there is a potential challenge in mitigating biases in the training and the output produced by nordeid. since the model relies on existing datasets for training, there is a risk of carrying the biases that exist within these datasets. therefore, ensuring the unbiased and equitable functioning of the model in the deidentification of norwegian clinical texts is essential and should not be overlooked. ## discussion each deidentification system needs to be customised on which phis to remove depending on the research task and type of data or what each country has for laws or rules. lawyers and physicians do not always agree on which phi is sensitive. for example, health care unit can be valuable to keep sometimes, age can also be important in certain research, most clinical researchers want to keep the class name while computer scientists consider it is much safer to replace identified phis with pseudonyms or surrogates, @xcite . in the example with age it can be replaced with an age close to the actual age, for example, @xmath0. ## conclusion in conclusion, this paper has discussed the challenges and methods involved in deidentifying and pseudonymizing norwegian clinical text for research purposes. the use of the nordeid tool and regular expressions for identifying specific types of sensitive information proved effective in the deidentification process. the research highlighted the importance of privacy preservation and the need for tailored approaches to meet legal and research requirements. the significance of mitigating potential biases in the training and output of deidentification models were also emphasized.