Evaluating Differentially Private Synthetic Data Generation in High-Stakes Domains (2024)

Abstract: The difficulty of anonymizing text data hinders the development and deployment of NLP in high-stakes domains that involve private data, such as healthcare and social services. Poorly anonymized sensitive data cannot be easily shared with annotators or external researchers, nor can it be used to train public models. In this work, we explore the feasibility of using synthetic data generated from differentially private language models in place of real data to facilitate the development of NLP in these domains without compromising privacy. In contrast to prior work, we generate synthetic data for real high-stakes domains, and we propose and conduct use-inspired evaluations to assess data quality. Our results show that prior simplistic evaluations have failed to highlight utility, privacy, and fairness issues in the synthetic data. Overall, our work underscores the need for further improvements to synthetic data generation for it to be a viable way to enable privacy-preserving data sharing.

## introduction the widespread availability of public digitized text has greatly facilitated the advancement of natural language processing (nlp). text processing could also be extremely valuable for processing high-stakes private data, like healthcare records @xcite , social workers' notes @xcite , or legal documents @xcite . however, the need to maintain data privacy hinders the responsible development and deployment of models in these domains. building nlp tools often requires sharing data externally with contractors or researchers, as agencies like child protective services typically do not have in-house ai expertise. while data sharing has been accomplished through data use agreements with individual teams or laboriously redacting identifiable information from text (e.g., @xcite ), these approaches have limitations. limited sharing still requires increasing the number of people who have access to sensitive data, and it precludes the development of public benchmarks, which have proved crucial for standardizing model development. redaction fails to fully prevent reidentification, as even lower dimensional data is often possible to re-identify with just small amounts of auxiliary data @xcite . furthermore, redacted data is not useful for tasks requiring sensitive information, such as developing a model to identify contact information for potential caretakers of a child @xcite . in our work, we propose and conduct useinspired evaluations of the feasibility of using synthetic data to address these limitations. recent work has proposed sharing synthetic text generated from differentially private language models in place of real data @xcite @xcite . differential privacy (dp) offers an appealing solution, as it provides a theoretical guarantee of privacy preservation that is controllable through a specified privacy budget. although the bulk of work in developing dp approaches has been centered around models trained on tabular and image-related data, there has been increasing interest in applying dp to unstructured text data @xcite @xcite . while initial results of synthetic data are promising, prior work has lacked grounding in realistic applications, for example, running experiments with public internet data that language models may already have been exposed to during pre-training. in contrast, we conduct experiments on text data from two high stakes domains: healthcare and child protective services, and we rigorously evaluate the synthesized text for its utility, privacy, and potential fairness implications. for utility and privacy, we introduce novel well-motivated evaluation criteria ("silver" coreference modeling and entity-centric metrics). to the best of our knowledge, no prior work has investigated fairness considerations in this domain. we evaluate several approaches for privacypreserving synthetic data generation, including finetuned models and an in-context-learning approach. while these evaluations reveal some promising opportunities for synthetic text, they further expose utility degradation, privacy leakage (even when using dp), and issues with group fairness. these results indicate that prior simplistic evaluations have overestimated current viability of synthetic data. our primary contributions include a rigorous and reproducible evaluation framework that exposes limitations underestimated in prior work, and empirical results over real high-stakes data. overall, our work demonstrates that contrived metrics do not necessarily translate to more realistic scenarios, emphasizing the need for thorough in-domain evaluation to understand potential strengths and limitations of synthetic data. ## discussion overall, our results are consistent with prior work in that we find only small performance degradation when training a model on dp-generated synthetic text as compared to real data for relatively less fine-grained (e.g. icd-9 @xmath0, in ## related work the majority of research on enabling shareable sensitive data has focused on text anonymization, replacing or redacting private information like names and addresses from text. while some approaches redact and replace sensitive information using deterministic rule-based systems @xcite @xcite @xcite , others employ masked language models @xcite . differentially private mechanisms have also been integrated into text sanitization processes, such as differentially private perturbation of text embeddings @xcite or sampling of replacement tokens @xcite building on the principle of metric-local dp @xcite . although these methods are computationally inexpensive and domain-agnostic, they have weak privacy guarantees and limited capacity to modify text @xcite @xcite . recently, datasets comprised entirely of synthetic data have become potentially viable @xcite . our work differs from similar approaches to synthetic data generation in its focus on actual high stakes data and thorough grounded evaluation @xcite @xcite . notably, al aziz et al. ( 2021 ) do similarly investigate healthcare data, but they do not evaluate potential privacy leakage, and their utility measures do not adequately capture errors in text fluency and consistency which is crucial for finer-grained applications. a separate but overlapping line of work has focused on improving privacy in nlp models & protection against membership inference, rather than in the generated data. this work has similarly trained nlp models with differential privacy but has evaluated direct performance of these models on downstream tasks @xcite . nevertheless, this line of work is not directly comparable to ours, as it focuses on training models directly on private data, while our approach promotes the shareability of data and imposes fewer restrictions on sharing models trained using private synthetic data. ## conclusions although synthetic data generated with differential privacy is an appealing way to improve responsible ai development, off-the-shelf dp does not achieve sufficient privacy, utility, or fairness over real high-stakes data. these failings suggest numerous opportunities for future work on improving the coherence of synthetic data and the application of privacy preservation to this task. our evaluation methods offer a way to foster this research, with grounding in real applications, rather than contrived settings, where performance is liable to being over-estimated.