Generation and Evaluation of Synthetic Endoscopy Free-Text Reports with Differential Privacy (2024)

Abstract: The development of NLP models in the healthcare sector faces important challenges due to the limited availability of patient data, mainly driven by privacy concerns. This study proposes the generation of synthetic free-text medical reports, specifically focusing on the gastroenterology domain, to address the scarcity of specialised datasets, while preserving patient privacy. We fine-tune BioGPT on over 90 000 endoscopy reports and integrate Differential Privacy (DP) into the training process. 10 000 DP-private synthetic reports are generated by this model. The generated synthetic data is evaluated through multiple dimensions: similarity to real datasets, language quality, and utility in both supervised and semi-supervised NLP tasks. Results suggest that while DP integration impacts text quality, it offers a promising balance between data utility and privacy, improving the performance of a real-world downstream task. Our study underscores the potential of synthetic data to facilitate model development in the healthcare domain without compromising patient privacy.

## introduction the development of computer-aided tools in medicine, including natural language processing (nlp), requires real patient data for model training. however, this development has been significantly limited due to the lack of availability of patient data due to privacy concerns, restricted access to hospital data, a scarcity of labeled data, barriers to sharing pretrained models, and a lack of capable computational resources in many healthcare settings @xcite . the lack of specialised datasets when developing nlp models can lead to biased or ungeneralizable models @xcite . recent literature highlights that open-source, synthetic datasets could mitigate data scarcity and lead to robust ai model training, particularly in nlp @xcite . however, very few studies tackle the generation of synthetic free text in the medical domain, with no known studies focusing on gastroenterology text reports. while synthetic data presents a viable solution to dataset scarcity, ensuring the privacy of patient data in the original dataset used for training remains essential. recent findings suggest that simply deidentifying the training set by removing names and unique identifiers is insufficient to prevent patient re-identification @xcite . despite this, it is not common practice to include a robust data privacy framework when generating synthetic medical data @xcite . to maintain stringent patient confidentiality, our approach incorporates differential privacy (dp), a framework that mathematically guarantees the level of inability to identify an individual's data within a dataset @xcite . our approach is motivated by the fact that only a limited number of academic papers investigate the application of differential privacy in the generation of synthetic data within healthcare @xcite . the quality and utility of these generated reports must also be rigorously assessed to ensure their practical application in clinical settings. utility, in our context, refers to the degree to which the synthetic data can be used to perform real-world tasks, such as text classification. it is crucial to compare how differential privacy impacts the quality and utility of synthetic data and whether it can be used to enhance performance on various tasks. these tasks can be supervised, such as text classification, unsupervised or semi-supervised, like task adaptive pre-training tasks (tapt). ## aims generating synthetic medical notes with differential privacy. the generation of synthetic text data with differential privacy (dp) is an emerging field with limited research. while (yue et al., 2022) has provided a comprehensive framework for generating synthetic text data with dp, none of the investigated datasets include medical data. sarkar et al. (2024) also propose a framework for synthetic data generation with dp. ## methods a summary of the overall pipeline is depicted in figure 1 . ## discussion and conclusion we have fine-tuned a pre-trained large language model with differential privacy to generate privacypreserved synthetic endoscopy reports. we leveraged a highly specific in-house training set of over 90,000 endoscopy free-text reports. using our pipeline, we generated a set of 10,000 diverse synthetic endoscopy reports, available for further research on a per-query basis. the utility of the synthetic reports was assessed by attempting to improve a clinically useful high performing classification baseline. the synthetic reports were used to augment the training set of the baseline and to pretrain the baseline classifier using a task-adaptive pretraining framework. a pre-training experiment with real patient data was also conducted for direct comparison. the privacy preservation of dp-generated reports is quantified by assessing their similarity to the original data compared to that of synthetic data generated without dp. we observed an important difference in rouge-l scores between the dp and non-dp generated synthetic reports. therefore, we can conclude that, in our study, dp effectively prevents the replication of sensitive training samples. a set of outliers was also introduced in the training set, and we concluded that unique clinical findings could not be regenerated by the models.