Training Data Extraction From Pre-trained Language Models: A Survey (2023)

Abstract: As the deployment of pre-trained language models (PLMs) expands, pressing security concerns have arisen regarding the potential for malicious extraction of training data, posing a threat to data privacy. This study is the first to provide a comprehensive survey of training data extraction from PLMs.Our review covers more than 100 key papers in fields such as natural language processing and security. First, preliminary knowledge is recapped and a taxonomy of various definitions of memorization is presented. The approaches for attack and defense are then systemized. Furthermore, the empirical findings of several quantitative studies are highlighted. Finally, future research directions based on this review are suggested.

## introduction pre-trained language models (plms) are widely used in natural language processing. statistical models that assign probabilities to token sequences have been studied, and large neural networks are increasingly being used for pre-training with large datasets. this scaling has led to fluent natural language generation and success in many other downstream tasks @xcite . in some cases, parameter updates are not required for downstream tasks @xcite . with increasing applications of plms, security concerns have increased considerably @xcite @xcite . studies have revealed the risk of language models exhibiting unintentional memorization of training data, and occasionally outputting memorized information @xcite @xcite . in particular, @xcite identified that personal information can be extracted by generating numerous sentences from plms and performing membership inference @xcite . these attacks on plms are referred to as training data extraction and are undesirable because of privacy, decreased utility, and reduced fairness concerns @xcite . however, with the evolution of plms, limited progress has been achieved in addressing these concerns, and security technology is yet to mature. this study is the first to provide a comprehensive survey of training data extraction from plms. starting with the pioneering work, we reviewed more than 100 previous and subsequent studies. specifically, we screened papers citing @xcite 1 based on the relationships, the number of citations, and their acceptance. first, section 2 presents preliminary knowledge. we then discuss several topics with the following contributions: ## preliminaries about plms this section describes the basics of modern plms. first, we explain the methodology used for training language models and generating texts. next, the standard practical schema is introduced. ## definitions of memorization memorization is the concept that plms store and output information about the training data. there is a wide variety of research on memorization, with diverse definitions and assumptions. we illustrate a taxonomy of definitions in figure 1 . ## training data extraction attacks this section systematizes the attack procedure. most studies follow @xcite . they revealed that hundreds of verbatim text sequences can be extracted from the training data. given a plm, the procedure consists of two steps, candidate generation, and membership inference, as displayed in figure 2 . ## training data extraction defenses this section systematizes approaches to defense. we can mitigate privacy risks before, during, and after creating plms as displayed in figure 2 . the classification was reconstructed using references @xcite @xcite . extensive studies have been conducted on the hazardous generation of plms @xcite @xcite @xcite . however, this study focused on training data extraction. ## empirical findings this section presents empirical findings on training data extraction from plms. initial studies were limited to qualitative evaluations, but subsequent studies @xcite @xcite @xcite @xcite have focused on quantitative evaluations. in particular, based on one of the first comprehensive quantitative studies @xcite , we report on the impact of the model size, the string duplication in the training set, and the length of prompts. they used various sizes of gpt-neo model family @xcite , which are the autoregressive language models pre-trained by the pile dataset @xcite . four model sizes, namely 125 million, 1.3 billion (b), 2.7 b, and 6 b parameters, were considered. the number of duplicate strings was determined by analyzing the pile dataset. a subset of 50,000 sentences from the pile dataset was used for evaluation, and the distribution of duplicates was considered. the beginning of each sentence was cut out at a certain number of tokens and considered as a prompt. the amount of memorization was calculated as the fraction of generations that exactly reproduce the true string for their prompt averaged over all prompts and sequence lengths. ## conclusion & future directions we have reviewed over 100 papers for the first comprehensive survey on training data extraction from plms. the final section provides suggestions for future research directions. we hope that this study highlights the importance of training data extraction from plms and accelerates the discussion.