Assessing Authenticity and Anonymity of Synthetic User-generated Content in the Medical Domain (2024)

Abstract: Since medical text cannot be shared easily due to privacy concerns, synthetic data bears much potential for natural language processing applications. In the context of social media and user-generated messages about drug intake and adverse drug effects, this work presents different methods to examine the authenticity of synthetic text. We conclude that the generated tweets are untraceable and show enough authenticity from the medical point of view to be used as a replacement for a real Twitter corpus. However, original data might still be the preferred choice as they contain much more diversity.

## introduction medical text is difficult to share, even for research purposes, as it contains information about patients that might reveal an individual's identity. this makes natural language processing in that domain difficult. moreover, there have been concerns about sharing even publicly available data from social media in recent years. this is partially due to legal reasons (e.g., x (twitter)) but also due to privacy concerns. while data sensitivity can be at least addressed by de-identification (removal of personal health identifiers) and anonymization (irreversible removal of all information that possibly links back to an individual) @xcite , privacy aspects constitute an additional barrier (see @xcite @xcite ). synthetic data generation bears much potential and a way out of this misery, particularly with the rise of generative models. various attempts within and outside the medical domain generate synthetic clinical data and show that large datasets can be easily generated and models trained on them can compete with models trained on real data @xcite figure 1 : example of a source (top) and a pseudo (bottom) tweet translated into english. the original japanese text can be found in the appendix. @xcite @xcite . apart from that, the use of generative data has advantages such as structural similarity, information relevance, and subjective assessment @xcite . furthermore, synthetic dataset have been shown to be useful in, e.g., epidemiology research, medical education and training, and algorithm testing @xcite . for example, @xcite employ generative adversarial networks (gans) @xcite to generate (english) electronic health records (ehrs) while @xcite synthesize tabular medical data such as laboratory values. amin-nejad et al. (2020) use gpt-2 @xcite amongst other models to create datasets of discharge summaries in english. these data are then used as either pure training/fine-tuning material @xcite or to augment existing resources @xcite , resulting in a better performance of the trained models when compared to the low-resource setup in which they are usually fine-tuned. @xcite report that linguistic phenomena are reproduced while privacy is preserved in their generated datasets of french clinical case reports. the usefulness of the authors' synthetic corpus is extrinsically investigated by fine-tuning models for a clinical named entity recognition task. the perfor-mance of the models yields promising results. although the use of generated data in corpora is highly significant from the viewpoint that medical language resources are difficult to make public, few studies evaluate the anonymity or authenticity of the data, including medical aspects. @xcite investigate the privacy-preserving characteristics and utility of synthetic ehrs by introducing a new measure based on pointwise differential training privacy (pdtp) @xcite . another work is provided by @xcite , who propose a framework to investigate the "realism" of synthetic ehrs. they compare the generated information with the rules, constraints, and concepts used in the original ehr data. their approach, however, does not apply to unstructured user-generated texts. in contrast to related work, this paper examines the authenticity of synthetic user content with respect to health-related topics in japanese. more precisely, we examine whether artificially generated user tweets about potential adverse drug effects (ade) are authentic and privacy-preserving and, therefore, might be a valuable alternative resource for future research. ## dataset the baseline of this work is a synthetic corpus of japanese tweets in the context of drug intake and symptoms @xcite . the source data (original) was collected using 68 medication names as keywords from a japanese drugname dictionary 1 in the twitter api. during preprocessing, urls and user names were removed. only tweets containing mentions of drugs and symptoms were kept in the data. t5 @xcite , a transformer-based encoder-decoder model, was fine-tuned on these data and finally used to generate 10,000 tweets per medication name. the created texts were filtered and manually annotated with 22 different adverse drug reactions. more details about the synthetic data can be found in the appendix and in @xcite . in this paper, we are examining how authentic these synthetic data are. in the following, we distinguish between source tweets, i.e., the original data, and pseudo-tweets, i.e., the synthetics tweets. ## method we analyze the data in different ways to measure the authenticity and validate the anonymity of the pseudo-tweets. first, we examine the source and the pseudo data on the word level and compare the vocabulary of both datasets. next, we analyze if the distribution of our target events in the synthetic data is similar to that in the source data. finally, we directly compare a subset of synthetic and source tweets manually as well as automatically with respect to naturalness, comprehensibility, medical correctness and anonymity. ## results in the following section, we briefly present the results of our analyses. more details, particularly tables and figures, can be found in the appendix. ## conclusion in this work, we analyzed synthetically-generated tweets in the context of drug intake and adverse drug reactions. the data was compared to (real) user-generated messages regarding authenticity, privacy preservation, and medical correctness. the results show that the synthetic data has characteristics similar to the source data. from a linguistic point of view, the data shows less variation, but it contains a similar number of data with questionable medical correctness (as the original), and has a similar authenticity. in addition to that, pseudo data could serve as a "safety net" as it might be less likely to provide identifiable information. finally, we believe that the findings are generally valid for different languages; however, larger and more complex models than t5 might increase the authenticity and correctness level but might easily reproduce sensitive information it has seen during training. regarding ethical considerations, the following three methods were implemented to avoid privacy issues in the original twitter data: deleting the usernames in training data for the model, deleting the exact duplicates in the generated text from the source, and, with manual work of annotators, checking all of the synthetic data and making sure no identifying information remains. models using original data were trained locally. we further acknowledge that questions used to judge tweets with gpt-4 and the corresponding responses are (1) not reproducible as soon as an updated version of gpt-4 is released, and (2) might result in different responses when the questions are slightly modified or set up differently. finally, to assess the authenticity and diversity of the data, many more linguistic measures could be applied. this paper only presents a few as a complement to the medically inspired investigations.