Utility Preservation of Clinical Text After De-Identification (2022)

Abstract: Electronic health records contain valuable information about symptoms, diagnosis, treatment and outcomes of the treatments of individual patients. However, the records may also contain information that can reveal the identity of the patients. Removing these identifiers - the Protected Health Information (PHI) - can protect the identity of the patient. Automatic de-identification is a process which employs machine learning techniques to detect and remove PHI. However, automatic techniques are imperfect in their precision and introduce noise into the data. This study examines the impact of this noise on the utility of Swedish de-identified clinical data by using human evaluators and by training and testing BERT models. Our results indicate that de-identification does not harm the utility for clinical NLP and that human evaluators are less sensitive to noise from de-identification than expected.

## introduction the training data for clinical nlp models are sensitive because they often contain information that can reveal the identity of real patient. this makes sharing data and models difficult. one way of decreasing the privacy risks of using clinical data is to de-identify them. a popular way of doing this is by automatically detecting and removing protected health information (phi). this is often done using named entity recognition models that can find these sensitive data in clinical texts. the de-identified clinical data can be used both for clinical research but also as training data for machine learning algorithms. these approaches will be described in the next section. one concern is that de-identification will deteriorate the quality of clinical texts. the risk is that this will not only harm down-stream nlp tasks, but also make the data less useful for other research purposes. we have also identified a hesitancy from lawyers who fear that de-identification can harm the safety of the data. in this paper, we evaluate the extent to which de-identification harms perceived utility using human evaluators. we then evaluate the impact of de-identification on the utility of the datasets for building clinical nlp models. ## related research this related research section presents studies regarding the quality of the de-identification system both in terms of safety and privacy but also for down-stream tasks as for medical research. in a study by @xcite 86 patient records in english were de-identified. eight physicians and 11 medical students that have treated and written these records 1-3 months earlier could not recognise their patients. some of physician suspected that they could recognize their patient on some clinical details but after a control it was found that the wrong patient had been identified. @xcite propose a sanitation process that removes information that might make the patient record sensitive. this is not done by replacing typical phi like names, but instead by replacing sensitive diseases such as clamydia, aids, or hiv with less sensitive terms such as virus. the ideas is to aggregate information but this limits the utility of the patient records. dalton-locke et al. (2020) used de-identified patient records from mental health clinics to perform research regarding the housing service of patients suffering from mental illnesses. structured data had previously been used for this research. the researchers compared the two approaches and found it feasible to use de-identified patient records and de-identified structured data jointly for this research. the system called cris is a combination of a de-identification algorithm and a security model has been approved for use in mental health research by the ethics board. this allows researchers to ex-tract data from the the patient record system without requiring individuals' informed consent @xcite . in an other study, dalianis (2019) constructed a pseudonymization system for swedish clinical text. the pseudonymization system replaced phi with pseudonyms or surrogates. all tags that could identify a phi were removed so the records looked realistic and neutral. the system was evaluated by two computer scientist that had worked with clinical text mining, specifically with this type of text. the text they evaluated had not been seen by the scientists before. they read 98 patient records where half were pseudonymized and the other half were not. on average, 91 percent of the pseudonymized records were judged as original. @xcite carried out deidentification and pseudonymization of over 323,000 danish patient records and then carried out a manual review of 369 de-identified and pseudonymized patient records with a total length of over 71,000 words, this revealed seven words where quasi-identifiers 1 had not been de-identified and it revealed 109 words where it was incorrectly de-identified, this reduced the medical correctness and readability according to the authors. a finding by the authors was if they use abbreviation lists and also medical lists the number of false positives would probably been diminished. @xcite evaluated the performance of the conditional random field (crf) algorithms on down-stream tasks based on clinical training text that have been de-identified with increasing degrees of recall. the authors used four different de-identification strategies: pseudonymization (replace with surrogats), masking, keeping the class name and removing the entire sentence containing the phi. pseudonymization was the most effective strategy for preserving down-stream utility. masking and replacing the phi with the class name had a larger negative impact. the most severe impact was seen when employing the sentence removal strategy. however, a balanced recall (not high recall) on all four strategies did not affect the down-stream performance significantly. ## data the clinical data used in this study originates from the karolinska university hospital and is stored 1 a quasi-identifier is a identifier that indirectly can identify a patient such an street name or a zip code. in the research infrastructure called health bank -swedish health record research bank foot_0 @xcite . the data encompasses 2 million patient records foot_1 . three clinical data set have been de-identified using the bert model created by @xcite . the experiments fine-tune models using both the unaltered data and the de-identified data. the following datasets were used: stockholm epr gastro icd-10 corpus a corpus of 795,839 tokens in 6,062 discharge summaries encompassing 4,985 unique patients with gastrointestinal diseases. each discharge summary is associated with multiple icd-10 codes which have been divided into blocks. the dataset is a described in @xcite . the task is to predict the correct icd-10 block for each discharge summary. ## discussion the human evaluators in this study disliked the idea of replacing phi with surrogate values. instead, they preferred replacing sensitive entities with their phi class. however, the resulting dataset not only fails to use the protecting effects of hips (hiding in plain site) @xcite . it also makes it obvious to an adversary that any phi they encounter is in fact a real phi that the system failed to replace. human evaluations also uncover the problem of dealing with abbreviations, since they can be mistaken for organizations. this can be dealt with by adding word lists to deal specifically with abbreviations. the human evaluators did not find the pseudonymized text difficult to read. on the contrary, one evaluator had difficulties distinguishing between real and pseudonymized data. this indicates that pseudonymized health records retain much of their utility for non-nlp research. comparing the utility of pseudonymized and real datasets, we find no harmful effects from deidentification on down-stream performance. this confirms and builds upon previous results in @xcite ) that also showed that utility was retained after de-identification. ## conclusion de-identification works best when the underlying ner classifier has both high precision and high recall. a high recall is crucial to ensure that as many phi as possible are detected. at the same time, having a low precision may introduce noise into the data which can harm its utility. in this study, we show that existing deidentification systems can effectively be used to make datasets safer. we also show that the noise introduced in this process does not harm downstream performance in clinical nlp tasks. the qualitative evaluations also show that humans have trouble distinguishing between pseudonymized and real data. we also uncover a discrepancy between the nlp community and our human evaluators regarding the perceived value of hiding sensitive data using hips.