Clinical Text Anonymization, its Influence on Downstream NLP Tasks and the Risk of Re-Identification (2023)

Abstract: While text-based medical applications have become increasingly prominent, access to clinicaldata remains a major concern. To resolve this issue, further de-identification and anonymization of the data are required. This might, however, alter the contextual information within the clinical texts and therefore influence the learning and performance of possible language models. This paper systematically analyses the potential effects of various anonymization techniques on the performance of state-of-the-art machine learning models based on several datasets corresponding to five different NLP tasks. On this basis, we derive insightful findings and recommendations concerning text anonymization with regard to the performance of machine learning models. In addition, we present a simple re-identification attack applied to the anonymized text data, which can break the anonymization.

## introduction although clinical text processing is gaining more and more attention, access to data remains a significant challenge as it typically contains sensitive, patient-related information. thus, personal information needs to be removed by applying one of the many existing de-identification and anonymization techniques and controlling access to the data (see, e.g., @xcite , @xcite al. (2019)) following the hipaa safe harbor (hipaa, 2022) method, we define de-identification as the removal of protected health information (phi) that directly relates to an individual, such as name, address, birth date, etc. however, de-identification does not guarantee anonymity for data subjects. on the other hand, anonymization is defined as any irreversible procedure, which is applied to the data, such that no information can be linked to any specific individual anymore @xcite , making the data subjects anonymous and no longer identifiable. de-identification might be sufficient to conceal sensitive patient data for many existing nlp tasks and datasets. conversely, to train models on a broader patient record provided by healthcare practitioners, including text and structured information, to support more complex medical problems, anonymization must be considered to protect patients' privacy. initiatives to make holistic patient data available for research are currently in planning @xcite . in this work, we only consider text data, which is one crucial aspect of a patient history. each text anonymization technique has different characteristics and brings modifications to the source text which might affect the machine learning potential. therefore, in this work, we explore the following questions: rq1: what happens when ml models are trained on anonymized corpora and tested on non-anonymized data? rq2: in which ways does this affect the learning procedure of nlp tasks and the final performance of the models? rq3: to share data for a specific nlp task, which techniques would be best, based on their characteristics, anonymization strength, and effects on model performance? rq4: how effective are these techniques against re-identification?" to explore those questions, this work conducts a systematic analysis regarding the influence of text anonymization and their effects on the performance of (state-of-the-art) ml models. in course of this, we train and test context-sensitive language representation models using various datasets corresponding to different nlp tasks. the main contributions of this paper include a set of findings and recommendations regarding text anonymization for nlp tasks in the context of ml, as well as, a fictitious re-identification experiment investigating the (in)effectiveness of the different techniques. ## related work a range of different text anonymization approaches exist in the literature, which modify the text struc-ture within a dataset, delete, replace, or introduce synthetic information, making it harder to identify or infer factual information about the patient. the following approaches have been explored and adapted for this work: suppression @xcite is a technique that either completely removes certain words or sentences or masks them with a neutral label denoting their suppression. perturbation @xcite modifies data through permutation or data swapping, in the case of text, similarly to data augmentation, by flipping characters or changing the order of words. substitution @xcite replaces certain information with other related or more general terms. aggregation (k-anonymity) @xcite groups individual data subjects together, e.g., by their attribute values, to make it more difficult to identify a single individual. only limited work has been done to describe the systematic influence of text anonymization on the performance of ml models. @xcite , explore the effect of different phi concealment strategies on named entity recognition (ner) tasks, @xcite explore the performance of concept extraction using de-identified data, as well as @xcite explore the effects pseudonymizing/removing phi data. the three papers mentioned above conclude that de-identification does not have a (strong) negative effect on the model performance regarding downstream nlp tasks. finally, although not clinical text, @xcite show that anonymization can cause significant negative changes in the sentiment analysis performance on twitter data. this work, however, goes beyond existing related work, as we conduct the first analysis regarding the anonymization of clinical text and the effects thereof on ml models. in this regard, we report the results and findings obtained mainly through seven different techniques we tested, on six datasets, corresponding to five different nlp tasks. ## data and methods the experiments in this work are based on the following datasets and tasks: while the first four datasets include annotated discharge summaries, the last two datasets include pairs of sentences extracted from mimic-iii @xcite . due to limited space, we refer the reader to the source papers. using those datasets, different text anonymization techniques are applied to the training split. the following techniques are implemented, based on suppression, perturbation, substitution, and aggregation, as described above: de-identification (dei) although the documents already are pseudonymized, de-identification through masking might have an influence on the performance which we want to examine. in this case, using the tool philter @xcite , all phi data such as synthetic names and dates in the text are replaced by "xxxx". mask numbers (mask) all occurrences of numbers in a given text, both in numerical or alphabetical form, are replaced using "xx". in this case, any numerical data that can hint to the patient, such as drug dosages, types of diabetes, quantities, hours, etc. is masked. shuffle sentences (shuf) sentences in a given text are shuffled. random swap (swap) a certain percentage of words are randomly chosen and swapped all over the document. the two previous procedures make it harder (to different degrees) to infer factual information about the data subject since the logical relationships between the sentences, such as temporality and causality, are broken @xcite . ## experiments for our experiments, we rely, if possible, on the original setup and configuration as described in the original publications. given the clinical corpus, the data is split into training and test data. next, anonymization is applied to the training data. for each anonymization technique, a model is trained and then evaluated on the original (not anonymized) text of the test split. the model is trained and evaluated five times to get reliable results in each experiment. if the anonymization technique is not deterministic and produces a different anonymized dataset each time, we repeat the text anonymization five times. this results in 25 runs. the results of each approach are averaged and compared to the base model's performance (without anonymization). we test the models on the original data for two main reasons: first, it's closer to a real-world scenario, where the (publicly available) data is anonymized to train an ml model, and the test data consists of the non-anonymized local patient data at a health care facility. second, this allows us to compare the effects of different techniques fairly. all experiments are conducted with bert base and bio+clinical bert. the experiments corre- ## discussion based on the outcomes of the previous experiments, we draw the following insights regarding clinical text anonymization: first, de-identification is an indispensable technique as it removes all direct identifiers. it provides the lowest level of anonymity and causes minimal performance loss but it must be combined with other anonymization techniques. based on our results and analysis, we can deduce that some anonymization techniques affect the performance of the models on specific tasks more than others. therefore, there is no single one-fits-all anonymization technique that can always be recommended. the optimal technique needs to be selected depending on the (security) requirements, the sensitivity of the data as well as the nlp task. overall, the results indicate a correlation between the performance loss and the strength of the anonymization technique, but each technique comes with different costs that should be considered. text aggregation is the strongest of the presented techniques. it offers relatively good security against re-identification but leads to the most substantial performance loss. moreover, the technique has various disadvantages: a) it leads to fewer training examples as data is merged, b) and longer text documents which might cause problems with standard bert models, which can only process up to 512 input tokens. finally, c) a patient data repository loses relevant information as data is mixed up with other patients. moreover, the simple fictitious re-identification experiment showed that patients could potentially be re-identified through a similarity search attack. although the scenario is hypothetical, it highlights the importance of providing additional security mechanisms for future health data repositories. one of the problems of our attack was that although sensitive data was removed and text modified, most of the text (words) remained the same. to make anonymization more secure, coming up with more advanced techniques might be necessary, such as modifying the overall text without changing the main content and meaning (a re-formulation task). this could be possibly overcome by generating synthetic data from real examples. ## conclusion this work presented a first structured analysis regarding text anonymization and its effects on the performance of state-of-the-art machine learning models. extensive experiments have been conducted including seven different anonymization techniques on multiple datasets, which cover five different clinical nlp tasks. on the grounds of this experimentation, we can analyse the results and extract valuable insights regarding the effects of different types of anonymization on machine learning performance with respect to a given task. for short, we did not find a universal one-fits-all anonymization technique that would perform best in all tasks. instead, the particular decision depends on several factors such as the type and strength of the anonymization technique, the underlying nlp task, desired level of anonymity, etc. in addition, we conducted a simple re-identification experiment to examine the robustness of each technique on a fictitious data repository. our initial results show that depending on the setup, the tested anonymization may not be strong enough to prevent re-identification. 109