Anonymization Through Substitution: Words vs Sentences (2024)

Abstract: Anonymization of clinical text is crucial to allow the sharing and disclosure of health records while safeguarding patient privacy. However, automated anonymization processes are still highly limited in healthcare practice, as these systems cannot assure the anonymization of all private information. This paper explores the application of a novel technique that guarantees the removal of all sensitive information through the usage of text embeddings obtained from a de-identified dataset, replacing every word or sentence of a clinical note. We analyze the performance of different embedding techniques and models by evaluating them using recently proposed evaluation metrics. The results demonstrate that sentence replacement is better at keeping relevant medical information untouched, while the word replacement strategy performs better in terms of anonymization sensitivity.

## introduction with the increasing adoption of electronic health record (ehr) systems, clinical data has become available in large amounts to be used by healthcare practitioners @xcite . however, it often contains sensitive information about patients and healthcare professionals that needs to remain private when being shared in order to comply with data protection regulations such as the health insurance portability and accountability act (hipaa) (u.s. department of @xcite in the united states of america and the general data protection regulation (gdpr) @xcite in the european union. many systems for the anonymization of clinical text have been developed throughout the years, ranging from solutions relying on hand-crafted rules and patterns @xcite @xcite to more complex systems based on machine and deep learning @xcite @xcite @xcite @xcite . although some of these systems show impressive results, their lack of adoption in realworld scenarios remains a barrier to sharing clinical data and its usage for secondary purposes. one should also consider whether perfect anonymization, i.e., removing all the sensitive information while keeping the non-sensitive information intact, is an achievable goal @xcite . while traditional named entity recognition (ner) based methods have shown impressive performance in anonymization tasks, achieving recall rates of over 90%, they still have limitations. abdalla et al. @xcite emphasized this issue, noting that relying solely on precision and recall for evaluating de-identification algorithms carries the risk of missing sensitive information. to tackle this challenge, they introduced an innovative solution. instead of solely relying on ner, they proposed a method that utilizes proximity measures between word embeddings. this approach replaces each token in a clinical note with a semantically similar one, ensuring the removal of all sensitive information. however, this method raises concerns about potential information loss and readability issues. ribeiro et al. @xcite have implemented this strategy on the incognitus toolbox, naming it k-nearest embeddings obfuscation (kneo). this work follows their approach and aims to compare two different strategies for the replacement -using word or sentence embeddingsby evaluating them on new and adapted metrics for anonymization sensitivity and clinical information loss. the remainder of this paper is structured as follows: section 2 provides an overview of word and sentence embeddings. section 3 outlines the evaluation metrics to compare the proposed strategies, and section 4 describes the used methodology. additionally, section 5 provides a discussion and anal-ysis of the obtained results, and section 6 lays out the conclusions. lastly, section 7 provides insights into some limitations of the analyzed solutions. ## embeddings finding representations of text is a necessary step in most natural language processing (nlp) tasks @xcite . word embeddings are commonly used by representing each word as a fixed-length vector of real numbers that captures useful syntactic and semantic properties @xcite . these representations allow the words to be the subject of mathematical operations that wouldn't otherwise be possible @xcite , aiding in finding similarities between text pieces. similarly to word embeddings, sentence embeddings are representations of entire sentences as fixed-size vectors in a continuous vector space. these embeddings capture the semantic meaning and context of the entire sentence, encoding information about word usage, syntax, and semantics. sentence embeddings models are trained on large text corpora and learn to encode sentences into meaningful vector representations. ## evaluation metrics we evaluate the performance of our strategies using the evaluation metrics proposed by @xcite . the authors divide the metrics into two categories: anonymization sensitivity metrics and clinical information retention metrics. the first category, whose focus is on the masking of sensitive entities, contains the following metrics: string matching-based recall (smr), average levenshtein index of dissimilarity (alid), levenshtein recall (lr), levenshtein recall for direct identifiers (lrdi) and levenshtein recall for quasi identifiers (lrqi). the clinical information retention metrics, jaccard similarity coefficient (jsc) and normalized softmax discounted cumulative gain (nsdcg), are based on the usage of a biobert @xcite model, which has been pre-trained on a hierarchical classification task of icd-10 code categories. these evaluation metrics and their formulas are described in detail in the previously mentioned paper. ## methodology the following methodology allows the comparison between the proposed strategies and models. two anonymization strategies, word and sentence substitution, were evaluated using one and four models, respectively. ## results and discussion figure 1 illustrates the performance obtained by each model on the different evaluation metrics by averaging the results obtained for all the test notes. we can observe that word replacement obtains better results on all the anonymization metrics except for alid but performs worse regarding clinical information retention. this is an expected outcome, as it is related to the way the anonymization is being performed. for example, when anonymizing a clinical note with the sentence "the patient's name is john doe", the word replacement strategy will replace every word in the sentence. however, when using sentence replacement, it could be the case that it is replaced with a different sentence that contains common elements, such as "john" or "doe," thus negatively impacting the performance of these metrics. the same rationale explains the better performance of sentence replacement in the information retention metrics. for instance, if the name of a medical condition appears in the clinical note we want to anonymize, replacing every word will result in that medical condition no longer being there. as for the sentence replacement, it is possible for the substitute sentence also to contain the name of the said medical condition. interestingly, replacing every word of the given clinical notes did not achieve a score of 100% in any of the metrics involving recall. this can be attributed to the fact that some sensitive entities can appear as subwords of other non-sensitive entities. additionally, mimic-iii also contains some labeling errors. regarding the anonymization sensitivity metrics, there is no discernable difference in performance between the doc2vec and the sentence-transformer models. it is interesting to notice that doc2vec and all-minilm-l6-v2, being the two models that produce vectors with the lowest number of dimensions, outperformed the two models that produce vectors with a much higher number of dimensions. this is because each dimension captures different semantic and syntactic attributes of the text, which may not be totally useful for the anonymization itself. on the two information retention metrics, however, the sentence-transformer models perform better than the doc2vec model. in this case, the dimensionality of the produced vectors most likely influences the results, as these metrics rely on the similarity of the original and anonymized version of the note. the avsolatorio/gist-large-embedding-v0 model obtains the best performance in both metrics. it is an expected result, as it is the model that produces vectors with the highest number of dimensions, which results in a better capturing of similar sentences. additionally, this pre-trained model is one of the best-performing models on the biosses benchmark. as for the pritamdeka/s-pubmedbert-ms-marco model, its lower performance might indicate that the pubmed text it was trained on differs from the clinical text contained in the mimic-iii database. while no strategy was better across all metrics, our strategies are based on the premise that the replacement group contains no sensitive information, and therefore, neither will the anonymized version of a clinical note. the lower performance the sentence replacement strategy obtains on the information retention metrics can originate from the overlap of fake sensitive entities in the replacement group and the test set. for example, a fake entity appearing in a note we are anonymizing may have already appeared in a sentence for the embedding space generation, which influences the sentence replacement process. although it is a fake entity, its presence in the anonymized version will have an influence on the results. had we utilized a dataset with real sensitive information, this overlap would likely have decreased and boosted the anonymization sensitivity results. as such, we look at sentence replacement as the better approach. ## conclusions this work presents a comparison between two different and novel techniques for the anonymization of clinical notes. five different models were tested and evaluated across several evaluation metrics aimed at anonymization sensitivity and clinical information retention. the discussed results indicate that both replacement techniques have their unique strengths and are viable alternatives to the traditional ner (named-entity recognition) approaches when the removal of sensitive information is a priority over data usefulness, as the latter are never capable of detecting all the sensitive information.