Beyond Reconstruction: Generating Privacy-Preserving Clinical Letters (2025)

Abstract: Due to the sensitive nature of clinical letters, their use in model training, medical research, and education is limited. This work aims to generate diverse, de-identified, and high-quality synthetic clinical letters to enhance privacy protection. This study explores various pre-trained language models (PLMs) for text masking and generation, employing various masking strategies with a focus on Bio_ClinicalBERT. Both qualitative and quantitative methods are used for evaluation, supplemented by a downstream Named Entity Recognition (NER) task. Our results indicate that encoder-only models outperform encoder-decoder models. General-domain and clinical-domain PLMs exhibit comparable performance when clinical information is preserved. Preserving clinical entities and document structure yields better performance than fine-tuning alone. Masking stopwords enhances text quality, whereas masking nouns or verbs has a negative impact. BERTScore proves to be the most reliable quantitative evaluation metric in our task. Contextual information has minimal impact, indicating that synthetic letters can effectively replace original ones in downstream tasks. Unlike previous studies that focus primarily on reconstructing original letters or training a privacy-detection and substitution model, this project provides a framework for generating diverse clinical letters while embedding privacy detection, enabling sensitive dataset expansion and facilitating the use of real-world clinical data. Our codes and trained models will be publicly available at https://github.com/HECTA-UoM/Synthetic4Health.

## introduction electronic clinical letters play a crucial role in healthcare communication. however, their sensitive nature makes them challenging to share and limits their adoption in clinical education and research @xcite figure 1 : an example of the objective: generating more clinical letters from the original anonymised clinical letter segment with clinical soundness 2016; @xcite . although public datasets such as mimic and i2b2 provide deidentified clinical data, they are often restricted to specific regions and institutions, limiting their representativeness of diverse clinical conditions @xcite . to address these challenges, synthetic clinical letter generation has attracted growing interest. while existing methods primarily rely on structured data, natural language generation (nlg) models provide a promising alternative by integrating linguistic and clinical knowledge @xcite @xcite . unlike previous studies, we go beyond training de-identification models to detect and substitute private information. this work focuses on leveraging nlg methods to generate synthetic clinical letters while indirectly minimising privacy risks. although the dataset we used has been anonymised, we additionally apply a privacy detection and masking process as an additional verification step to further enhance the security of synthetic letters. our findings contribute to bridging the gap in privacyaware clinical letter generation, facilitating a more effective approach to processing real-world clinical letters and addressing data scarcity in the medical domain. a brief example of our objective is shown in figure 1 . to achieve this, we investigate different model architectures, segmentation strategies, and masking techniques and evaluate their effectiveness both qualitatively and quantitatively. additionally, we assess their usability in downstream nlp tasks such as named entity recognition (ner). we ensure compliance with ethical guidelines by using only de-identified clinical data and adhering to all data use agreements. ## related work biomedical patient data privacy protection has been an important task for clinical research, especially when it comes to big data era. developing privacy-preserving decision support tools has been a challenge for statisticians and clinical researchers @xcite @xcite . recent studies in clinical natural language processing (nlp) explored various tasks, including ner, de-identification, and nlg. several tools, such as scispacy @xcite , are designed to enhance domain-specific entity recognition, while philter @xcite combines both traditional and modern nlp models to identify and remove protected health information (phi). transformerbased architectures are widely used in clinical nlg, particularly in text rewriting, discharge summary generation, and data augmentation, @xcite . for instance, lt3 @xcite improves label-to-text generation, while deid-gpt @xcite employs gpt-4 to identify and generate substitute words for private information. @xcite demonstrate that masked language models (mlms) outperform causal language models (clms) in text masking tasks. existing studies either focus on training models, utilize existing llms identify to identify private information, or concentrate solely on nlg without much attention in privacy. however, few studies integrate clinical text generation with privacy-preservation and diversity considerations, which is the focus of this study. ## methodology to generate clinical letters that retain the original clinical narrative without being exact duplicates, we employed various plms. sensitive data is masked by and substituted with contextually predicted tokens using plms. additionally, we evaluate different masking strategies to de-identify potentially sensitive information as an additional validation step. we also considered how non-sensitive elements, such as stopwords, indirectly influence the effectiveness of de-identification. a brief workflow is presented in figure 2 . ## conclusion this study explores de-identified synthetic clinical letters that preserve document structure and clinical narratives while enhancing diversity. encoderonly models outperformed encoder-decoder models, with base models performing comparable to