Named Entity Inference Attacks on Clinical LLM s: Exploring Privacy Risks and the Impact of Mitigation Strategies (2025)

Abstract: Transformer-based Large Language Models (LLMs) have achieved remarkable success across various domains, including clinical language processing, where they enable state-of-the-art performance in numerous tasks. Like all deep learning models, LLMs are susceptible to inference attacks that exploit sensitive attributes seen during training. AnonCAT, a RoBERTa-based masked language model, has been fine-tuned to de-identify sensitive clinical textual data. The community has a responsibility to explore the privacy risks of these models. This work proposes an attack method to infer sensitive named entities used in the training of AnonCAT models. We perform three experiments; the privacy implications of generating multiple names, the impact of white-box and black-box on attack inference performance, and the privacy-enhancing effects of Differential Privacy (DP) when applied to AnonCAT. By providing real textual predictions and privacy leakage metrics, this research contributes to understanding and mitigating the potential risks associated with exposing LLMs in sensitive domains like healthcare.

## introduction various fields have seen the benefits of applying transformer-based large language models (llm) to nlp tasks @xcite . the medical domain is one such field that has applied llms to various tasks and achieved state-of-the-art performance @xcite . due to the increased number of training parameters; training such models can be expensive in terms of computation, data, and time. to alleviate these issues, pre-training is done via a general language modelling task, and this "base" model is distributed to be fine-tuned @xcite . the result of the pre-training and fine-tuning process is a language model that achieves a high level of performance for a specific task within a specific domain. anoncat is a roberta-based llm that has been fine-tuned for the task of de-identifying clinical textual data @xcite . the purpose of anoncat is to protect patient privacy within healthcare records and to provide a framework that is adaptable between hospitals, departments, and other healthcare agencies. anon-cat is available through the medcat github foot_0 @xcite . textual data containing sensitive personal information can be encoded in the model during pretraining @xcite and fine tuning @xcite , and this may be exploitable by inference attacks. clinical textual data will often have highly sensitive attributes that a model will see during training, such as names, dates of birth, medications, family, and lifestyle. motivated attackers may be able to infer such sensitive attributes via white-box (direct access to the model) @xcite and black-box (access to model outputs only) attacks @xcite . inference attempts are more commonly applied to generative models in comparison to alternative textual models (such as masked language models) @xcite . efforts have been made to reduce the amount of training that can be leaked from inference attacks; such as regularization, differential privacy, confidence masking, and knowledge distillation @xcite . in particular, differential privacy (dp) is a common defence against data leakage from llms @xcite , where individual data points are aimed at being obfuscated while maintaining the statistical information of the underlying dataset. in this work, our aim is to look at anoncats susceptibility to a "name inference attack", a variant of an attribute inference attack. we also provide two methods to measure the privacy of the model. a name inference attack is an attempt by a motivated attacker to infer the named entities of a given de-identified text. we look to answer the following questions: attribute inference attacks are an alternative method in which an attacker can infer sensitive features from samples @xcite . these samples are assumed to be from the training set, or at least statistically similar to training samples. another method of attack is embedding inversion, where, given the embedding parameters, sensitive tokens or phrases can be recovered @xcite . these methods generally do not target the most sensitive of training information -such as names and dates of birth. some works look at inferring sensitive information at a "group" level as opposed to a single sample, which achieves a higher leakage of relative privacy @xcite . attackers also have multiple avenues to expose vulnerabilities and gain access to training data. white-box and black-box attacks cover large amounts of potential attacks, with varying levels of access to victim models and source weights @xcite . datasets used in the attack are similarly varied according to their task and availability @xcite . to combat this, work has been done to enable the application of dp in deep learning on a large scale, where privacy is maintained and the impact on predictive performance is minimised @xcite . this has been extended to the realm of nlp, where dp has been deployed in an attempt to preserve the privacy encoded in hidden states while maintaining the utility of the model @xcite . efforts have also been made to ensure the privacy of fine-tuning datasets through techniques applied during the fine-tuning process @xcite . ## methods given a sample x which is comprised of it's sensitive and non-sensitive attributes (in this case tokens) such that: @xmath0, x s ] where x ns refers to its non-sensitive attributes and x s refers to its sensitive counterparts. we define the attack algorithm in alg. 1. the hidden states h(x ns ) provided by φ are used as input for the attribute inference attack, where the trained parameters of φ are frozen so as not to poison the attack model with ground truth from the attack dataset. ## conclusion we have demonstrated the "named inference attack", an attribute inference attack that focuses on generating the names that were used as part of the training process. we demonstrated our attack on de-identification models trained using "anoncat", showing that we can predict approximately 2% of names from an attack dataset when using only the most likely generated label. finally, we compared the performance of the attack with models with differing levels of privacy, such as a black-box attack or differential privacy. various works have presented different methods of inference attacks on machine learning models @xcite @xcite . all of these works show a small, but potentially significant, data leakage. the same has been demonstrated in this work, with perhaps the most sensitive attribute -names. when only the most likely prediction is generated, name inference attacks perform similarly (~2%) to other works that attempt to infer sensitive attributes in similar masked language models @xcite . although generating multiple predictions for a single input is not standard practice in traditional machine learning models, this approach can be particularly useful in attribute inference attacks. by generating more names for a single input, the model's performance improves, potentially increasing the risk of sensitive attribute disclosure. this may also show that generating text via a causal language model is a difficult task compared to other tasks where output labels are limited. this type of attack is measured in terms of absolute leakage. conventionally, leakage is measured in relative terms compared to random guessing @xcite @xcite . the attribute space for the type of attack demonstrated here has too many possibilities. random guessing can be assumed to have a performance of 0%, and thus absolute performance is a suitable metric. consensus on an acceptable level of information leakage may be difficult to reach. although any level of leakage is not ideal, different fields may have different tolerances for privacy leakage. ultimately, acceptable leakage is contextually defined by the interaction of technical limits, risk assessments, regulatory requirements, and specific downstream use. whilst there is no direct 'acceptable' level of leakage or privacy, the uk's information commissioner's office has previously suggested in correspondence that 95% accuracy of the de-id model itself would be acceptable given that these models are being deployed into environments with many additional security and privacy constraints. hospitals such as university college london hospitals are using these guidelines as part of their information governance. there is a minor improvement in privacy during the black-box attack compared to a white-box attack using the binary classification metric. the rolling hamming distance shows greater privacy provided by limiting access to model weights. differential privacy shows a trade-off between model utility and privacy. as inference attack performance degrades in line with privacy budget increase, the predictive performance decreases when attempting to de-identify text. the small differences in attack model performance between different budgets may indicate that the inherent difficulty of inference attacks on masked language models may only require a smaller allocation of a privacy budget compared to other models. consideration should be given to the goals and objectives of potential attackers, especially in fields such as healthcare, where there is low tolerance for information leakage. little has been formalised about hypothetical attackers conducting inference attacks, and less about real-world attackers performing real attacks. are they seeking to infer as much private information as possible or targeting specific individuals? are their motivations financial, political, or something else? this work can validate models and apis, enabling their secure external exposure while using real-world data. by understanding the risk of sharing data and models, information governance teams can define tolerable thresholds of privacy risk, facilitating access to resources for fields such as healthcare and research. in our experiments, we assume that the attack training data follows a distribution similar to the victim model's data. although this assumption cannot be guaranteed, it provides some security, as an information leakage ceiling of 2 -8% reduces the confidence of potential attackers. moreover, if a large-scale attack were to take place, it would be difficult for such an attack to isolate the true positives from the false positive results. however, further attacks that target both true and false posi-tives may achieve some success. future work could explore vulnerabilities beyond names, such as addresses, ages, and other sensitive attributes that may also be inferable. identifying these risks is critical to protecting privacy and equipping policy makers to make informed decisions. this work has focused on inferring names that have been used in the process of training anoncat; where the pre-training step is a masked language model. other models can be explored in future work, such as generative language models, which have become more prevalent as conversational ais become more common. for a fully secure environment, we recommend that red-team inference attacks not be the sole focus of security considerations. this approach should be used in conjunction with other measures to ensure both model and data privacy. anoncat is deployed within secure data environments and enhanced with additional security measures, such as restrictive access controls and active monitoring of access and usage.