Unlocking the Potential of Large Language Models for Clinical Text Anonymization: A Comparative Study (2024)

Abstract: Automated clinical text anonymization has the potential to unlock the widespread sharing of textual health data for secondary usage while assuring patient privacy. Despite the proposal of many complex and theoretically successful anonymization solutions in literature, these techniques remain flawed. As such, clinical institutions are still reluctant to apply them for open access to their data. Recent advances in developing Large Language Models (LLMs) pose a promising opportunity to further the field, given their capability to perform various tasks. This paper proposes six new evaluation metrics tailored to the challenges of generative anonymization with LLMs. Moreover, we present a comparative study of LLM-based methods, testing them against two baseline techniques. Our results establish LLM-based models as a reliable alternative to common approaches, paving the way toward trustworthy anonymization of clinical text.

## introduction clinical data contains sensitive information about patients and healthcare professionals. unauthorized disclosure of this data can compromise patient privacy by linking the disclosed patient information with other accessible data sources @xcite . therefore, information systems must comply with regulations such as the general data protection regulation (gdpr) @xcite and the health insurance portability and accountability act (hipaa) (u.s. dept of @xcite , which grant data protection rights to the citizens of the european union (eu) and the united states (us). according to the international organization for standardization (iso), data anonymization is "the process by which personal data are irreversibly altered so that a data subject can no longer be identified directly or indirectly, either by the controller or in collaboration with any other party" (iso figure 1: illustration of the followed workflow. clinical notes can be anonymized through various methods, including llm-based approaches. a fair evaluation is carried out using novel metrics, compatible with every anonymization strategy. large language models (llms) have the potential to be useful tools in anonymizing clinical notes due to their ability to process and interpret vast amounts of unstructured data, produce multilingual text, and leverage extensive general knowledge that may aid in this task @xcite . however, the increasing size of these models raises concerns regarding the inherent sensitivity of this type of data, particularly when using external computing on cloud-based platforms or relying on proprietary models, such as openai's gpt-4 @xcite , which can only be reached through external apis, potentially compromising confidentiality. to address this challenge, this work explores the potential of using open-source llms that can be locally deployed on cheaper and readily available infrastructure. by running these models locally, healthcare providers can retain full control over their data, significantly mitigating risks associated with external data transfer and storage. furthermore, local deployment allows for the fine-tuning of models, enhancing the effectiveness of the data anonymization process by adapting the model to the specific types of notes produced by each hospital. this approach ensures the protection of sensitive information and aligns with the growing need for healthcare systems to adopt more secure and regulatory-compliant technologies in handling and analyzing data. to support our approach, this work advances the state-of-the-art by proposing six new evaluation metrics to fairly measure the quality of each model and the clinical information retention in the anonymization process. figure 1 illustrates the overall workflow followed in this paper. ## de-identification framework and tools the need for effective and reliable clinical text deidentification methods has led to the development of various tools and frameworks. following this trend, @xcite developed incogni-tus, a comprehensive toolbox that delivers conventional and state-of-the-art techniques for automated clinical text de-identification, including a presidio-based architecture @xcite , and a de-identification module based on k-nearest neighbor obfuscation (kneo) @xcite . the goal of this section is to target the background components of the incognitus framework, which are used as a baseline for comparison with the llm-based methods analyzed in this study. ## evaluation metrics in conventional methods such as ner-based techniques, the computation of commonly used evaluation metrics such as recall, precision, and f1-score is relatively straightforward. since each token is associated with a label, and classification models output a prediction for each token, one simply needs to compare the true and predicted values to conclude the correctness of each prediction. nevertheless, using other types of anonymization methods, such as those based on generative models, raises challenges. as these methods output directly anonymized textual content, which may not be written the same way as the input, the link between tokens and labels gets lost. because the locations of sensitive entities may change from the original to the de-identified version, directly calculating these metrics is no longer possible. moreover, in generative models not all removed tokens were considered sensitive and thus using the concept of false positive, i.e., a replaced or erased token that did not constitute a sensitive entity, would lead to an unfair judgment of performance. a potential strategy to identify entities that went unnoticed during anonymization is a total string matching search for the exact content of every sensitive entity in the original content. however, this strategy is limited, as simple alterations to the sensitive entities compromise their detection and, subsequently, the trustworthiness of further calculations based on these matches. to address these issues, we propose four new metrics independent of token-target links that can be used to evaluate any anonymization method fairly. these are built upon the concept of levenshtein distance (ld) @xcite . furthermore, two of these metrics assess anonymization by focusing more on privacy concerns. it is important to note that text anonymization inevitably entails a tradeoff between minimizing privacy risks and retaining data utility. therefore, we also propose two metrics to evaluate clinical information retention. ## methodology the methodology was designed to enable a fair comparison between the performance of different techniques for clinical note anonymization. a total of seven anonymization solutions were compared: two baseline techniques offered by the incogni-tus toolbox @xcite , a fine-tuned bert model, clinicalbert @xcite , and four prompt-based methods that leverage microsoft's phi-2 and meta's llama-3 llms @xcite (including two zero-shot learning strategies and two finetuned models). ## results and discussion figure 2 presents the performances of the different strategies, as given by the average of each evaluation metric measured across all the test notes. firstly, we focus on metrics of anonymization sensitivity, particularly smr and lr. the overall performances measured by both metrics are consistent with one another, with the exception of kneo which is the best-performing strategy only according to smr. this result was expected given that this anonymization method replaces every single token, ensuring that no sensitive entity remains unaltered and thus reporting better performances when evaluated by a metric that looks for total string matching. since lr is not as sensitive to slight alterations in the spelling of entities, one can infer that this metric considered that some changes carried out by kneo were insufficient to achieve anonymization. this hypothesis is corroborated by the fact that kneo obtained the lowest alid, which indicates that the replacements made are less substantial compared to other anonymization methods. another noteworthy result is that no anonymization method was able to achieve 100% in any recall measure, i.e. smr, lr, lrdi, and lrqi. further inspection of the data showed that some sensitive entities consist of a pair of letters (name initials), which can easily appear in the middle of nonsensitive words. in addition, there are also some inconsistencies in the labeling of the mimic iii dataset, e.g., isolated numbers that are labeled as dates when they merely refer to quantities. these occurrences were misclassified as errors, which explains the absence of perfect recalls. moreover, although ld-based metrics may identify incomplete de-identification occurrences, they can also be misleading when certain words are similar despite being unrelated (e.g., the name "tim" and the first three letters of the word "time" produce an lra of 1). note that these situations can affect every method, and therefore the comparisons between different techniques are valid nevertheless. as for alid, its results would never reach 100% because, even in perfect anonymization, there is always a residual similarity between an entity and any other token in the note. as a result, the observed consistency between the values of the leading approaches clinicalbert shows the overall best performance on the anonymization sensitivity metrics, while presidio has the lowest results in 4 out of 5 measurements. even though presidio is specifically tailored for text de-identification, it still lacks critical clinical concepts to achieve better performance on clinical-related de-identification. when it comes to information retention, the fine-tuned generative models reported the highest values in both jsc and nsdcg, followed by clinicalbert. the most prominent result is the significant difference between the low values achieved by kneo and those of the remaining methods. this exposes the unfortunate yet somewhat expected outcome related to the significant loss of information associated with the application of kneo, reflected both in a lower jsc (19.1% compared to an average of 72.6%), and in a lower nsdcg (30.4% compared to an average of 84.6%). considering that the icd-10 classification model used was trained to classify code categories and not specific codes, this is an even more concerning outcome, which shows that the kneo strategy needs improvement before being considered a reliable method. apart from kneo, our second baseline, presidio, also underperformed on these two metrics. as a result, in terms of clinical information retention, its performance can be compared to the performance of zero-shot generative models, which also were not specifically pre-trained on the clinical domain. looking specifically at the generative methods, all of them were consistent in terms of recall. however, one can notice that an increase in the number of parameters of the model (e.g., phi-2 has approximately 2 billion, llama-3-8b has 8 billion parameters, and so on) has a slight positive impact across all metrics. the number of parameters positively correlates with the amount of information distilled into the model, which can enable the model to better generalize across multiple tasks. another important point to note is that the tested generative models improve when fine-tuned, while zero-shot models struggle to identify the structure of clinical notes. although recall metrics are not heavily compromised, zero-shot models often end up anonymizing entities that should not be omitted. for that reason, the precision and clinical information retention of the model are weaker. on the other hand, fine-tuned models have a better understanding of the structure of the clinical text and are able to retain crucial information while anonymizing sensitive entities. therefore, while increasing the number of parameters improves overall performance, fine-tuning is essential for maximizing the model's precision and its ability to not lose important clinical information. as an example, even the smallest fine-tuned model, phi-2, was able to beat the largest zero-shot model, llama-3-70b, on both clinical information retention metrics, while keeping competitive recall results. ## conclusions this work presents a comprehensive comparative study between traditional methods for the automated anonymization of clinical text and new techniques that leverage the power of llms. two different approaches from the incognitus anonymization toolbox and five methods based on llms were tested across seven different performance metrics, including six newly proposed metrics designed to tackle the challenges inherent to generative methods. the results introduce anonymization techniques based on llms as a promising alternative to the current methods, representing a step forward toward unlocking the true potential of clinical text data for secondary usage.