Generating Synthetic Free-text Medical Records with Low Re-identification Risk using Masked Language Modeling (2025)

Abstract: The abundance of medical records holds great promise for enhancing healthcare and advancing biomedical research. However, due to privacy constraints, access to such data is typically limited to internal use.Recent studies have attempted to overcome this challenge by generating synthetic data through Causal Language Modelling. Yet, this approach often fails to ensure patient anonymity and offers limited control over output diversity—unless additional computational cost is introduced.In response, we propose a method for generating synthetic free-text medical records based on Masked Language Modelling . Our approach retains key medical details while introducing variability in the generated texts and reducing the risk of patient re-identification. With a relatively lightweight architecture of approximately 120 million parameters, the system ensures low inference costs.Experimental results show that our method produces high-quality synthetic data, achieving a HIPAA-compliant PHI recall of 96% and a re-identification risk of only 3.5%. Furthermore, downstream evaluations reveal that models trained on the synthetic data perform comparably to those trained on real-world data. Our trained models are publicly available on Github as SynDeidMLM (at https://github.com/SamySam0/SynDeidMLM ) (meaning syn thetic and de-id entified data generation using MLM ).

## introduction the widespread adoption of electronic medical record (emr) systems has led to the accumulation of substantial volumes of patient data, offering considerable opportunities to improve healthcare delivery and biomedical research @xcite . however, access to such data is heavily restricted due to privacy concerns, aiming to safeguard patients' personal information @xcite . one promising alternative is the use of synthetic data, which allows the generation of documents-such as discharge summaries-that retain medically relevant information while reducing privacy risks. this approach enables broader data sharing for purposes like healthcare system testing @xcite , medical training @xcite , and the development of artificial intelligence tools @xcite . much of the previous work on synthetic medical text generation has primarily relied on causal language modelling (clm), with comparatively limited attention paid to masked language modelling (mlm). while clm approaches have shown promise in replicating the statistical patterns of real-world clinical data, they present several challenges-specifically, ensuring privacy protection, managing diversity in generated texts, and mitigating the computational cost of generation. recent findings by @xcite demonstrate that masked language modelling can perform on par with causal language modelling across a wide range of synthetic generation tasks, while offering greater flexibility in controlling contextual information. building on these insights, this paper introduces a system designed to generate synthetic english-language medical texts-such as discharge notes, admission records, and doctor-todoctor communications-using masked language modelling. the system integrates a cutting-edge de-identification tool capable of automatically detecting protected health information @xcite , thereby removing the need for manual pre-processing. it also incorporates two named entity recognition (ner) models to help retain essential clinical information and strike a balance between diversity and fidelity in the generated output. importantly, the system is based on an encoder-only, non-autoregressive architecture, significantly reducing both its size and inference cost. the code will be released for public access. ## 200 2 related work in their recent study, @xcite proposed a generative adversarial network (gan) to produce synthetic electronic health records. while effective in some respects, their method struggled to manage the similarity between synthetic and original data and failed to accurately capture temporal dependencies within medical histories. building on similar techniques, @xcite presented a system for generating synthetic medical records with a low risk of re-identification. despite encouraging results, the authors noted that limited diversity in the generated samples reduced their usefulness for tasks like oversampling. they also assumed that synthetic generation alone sufficiently mitigates re-identification risk, signalling the continued need for explicit de-identification mechanisms. in one of the most recent contributions to synthetic medical data research, @xcite assessed gpt-3.5's ability to generate discharge summaries. their findings revealed that gpt-3.5 often closely reproduced input concepts, thereby heightening the risk of re-identification. additionally, the generated texts were often unnatural, omitting key medical details while introducing irrelevant or misleading information. clinicians involved in the evaluation acknowledged the presence of correct content but highlighted weaknesses in narrative structure, variety, and supporting details. another concern raised was the model's lack of data governance, as it is not maintained by the institution that owns the original data. taken together, these studies highlight common challenges in synthetic medical text generation: persistent privacy concerns and limited control over output variability. in response, our work advocates for the use of masked language modelling, which offers enhanced control over the content being generated while reducing privacy risks and maintaining lower computational costs. ## system design the system architecture, illustrated in figure 1 , is designed to generate synthetic medical records-including discharge summaries, admission notes, and clinician correspondence-through a two-stage pipeline: a masker and a mask-filling system. the masker identifies which parts of the text should be hidden or retained, outputting a partially masked version of the original text. the mask-filling system then replaces the masked sec-tions with context-aware content, producing one or more synthetic variants of the original record. ## experimental setup this section presents the dataset used to train and evaluate the mlm model, alongside the four system variants assessed in our experiments. ## conclusion in this work, we proposed a system using masked language models to generate synthetic clinical text, addressing challenges of data scarcity and privacy. the system includes a masker (with deidentification, medical ner, and pos tagging) and a mask-filling module (supporting both simultaneous and iterative strategies). key findings show that: (1) the system produces diverse yet clinically meaningful text, (2) offers control over diversity and fidelity without reducing readability, (3) performs well in downstream ner tasks-comparable to real data, (4) ensures strong privacy protection (hipaa-phi recall of 0.96; re-id risk of 0.035). the full system, syndeidmlm, is available at https://github.com/samysam0/syndeidmlm .