Provably Confidential Language Modelling (2022)

Abstract: Large language models are shown to memorize privacy information such as social security numbers in training data. Given the sheer scale of the training corpus, it is challenging to screen and filter these privacy data, either manually or automatically. In this paper, we propose Confidentially Redacted Training (CRT), a method to train language generation models while protecting the confidential segments. We borrow ideas from differential privacy (which solves a related but distinct problem) and show that our method is able to provably prevent unintended memorization by randomizing parts of the training process. Moreover, we show that redaction with an approximately correct screening policy amplifies the confidentiality guarantee. We implement the method for both LSTM and GPT language models. Our experimental results show that the models trained by CRT obtain almost the same perplexity while preserving strong confidentiality.

## introduction language models (lm) have rich real-world applications in, among others, machine translation @xcite , ai chatbots @xcite , question answering @xcite , and information retrieval @xcite . the advent of transformers @xcite has fostered a dramatic advancement in the capabilities of generative neural language models, yet they come at a cost to privacy, as the amount of excess parameters in the lm enables it to memorize certain training samples. recent works show that sensitive user information from the training dataset, such as address and name, can be extracted verbatim from text generation models by querying the lm as an api @xcite @xcite . how to train a highperforming language model without memorizing sensitive text has become a major research challenge. existing solutions to this problem primarily leverage differential privacy (dp) @xcite . differentially private learning algorithms ensure that an attacker could not infer whether a data point is used for training, let alone extracting the sensitive information within that data point. however, there are several mismatches between the problem of privacy that dp addresses, and our problem of preventing the memorization of sensitive text (henceforth referred to as confidentiality). first, confidential information in a natural language dataset is sparse (e.g., the bulk of an email might not carry confidential information). dp's undiscriminating protection for all sentences could be unnecessarily conservative which limits the utility of the trained model. second, what needs to be protected is the content of the sensitive text, rather than the data context. for example, in the sentence "my ssn is 123-45-6789.", it is the actual ssn that we hope to conceal rather than the general information that someone entered her ssn in a chatbot dialogue. thirdly, the same sensitive content could appear in many data points, which makes the protection of the content more challenging than protecting one data sample. these differences motivate us to treat the problem of confidentiality protection in lm separately with new definitions. besides dp, we also consider classical techniques of redaction and deduplication. redaction refers to the process of removing sensitive or classified information from a document prior to its publication in governmental and legal contexts. deduplication is the procedure of detecting and removing identical and nearly identical texts from a corpus. the main challenge of applying these techniques is that it is hard to manually redact a gigantic dataset and automated tools are far from being perfect. figure 1 : an example from simulated dialog dataset customersim. the yellow highlights are confidential content (middle). left shows the text after redaction by a sequence labeling policy π. however, if the policy is not perfect, there exists false negative or false positive samples as shown on the right. the contribution of this paper is fivefold. to the best of our knowledge, we are the first that rigorously establish the role of deduplication and redaction in achieving provably stronger confidentiality (or the related differential privacy) guarantees; and the first that achieve provably confidentiality in transformer models with only a mild utility loss. next, we briefly introduce the relevant background and discuss the related work to put our work in context. language modeling is a fundamental problem in natural language processing @xcite @xcite . consider a text sequence that consists of multiple tokens from a vocabulary v, i.e., @xmath0, w 2 , . . . , w n ), where w i is the i-th token. the goal of language modeling is to construct a generative model of the distribution pr(w), by applying the chain rule pr(w) = n @xmath1. we let f θ (w i |@xmath2. a language model is trained to maximize the probability of the data in a training set w, by minimizing the negative log-likelihood over each training example with the loss function recurrent neural networks (rnns) used to be a common choice for the neural network architecture to estimate the probability distribution pr(w). @xcite . more recently, large-scale transformer-based language models have replaced rnns in state-of-the-art models for all sorts of nlp tasks @xcite . nevertheless, common language models are vulnerable to privacy attacks and possibly expose information about their sensitive training data @xcite . differentially private (dp) learning methods (see, e.g., @xcite has been applied to language models as a blanket solution for a number of privacy and security risks. @xcite trained an rnn language model with dp guarantees in a federated learning setup. @xcite pre-trained bert under dp on datasets with hundreds of millions of examples. these paper also demonstrated that dp can effectively prevent data-extraction attacks in practice even for algorithms with dp guarantees that are considered too weak from a theoretical-perspective (e.g., ϵ = 8 or 16). however, the strong protection of dp often results in a substantial drop in the utility of the trained model, which makes them less desirable in practice. in fact, it was recently shown that it is necessary for deep learning models to memorize certain training data to achieve high accuracy (feldman, 2020), which suggests that dp or any other techniques that require the model to not memorize any training data will perform poorly in the highdimensional, power-law distributed real datasets. this motivates us to consider weakened models that only prevent memorizing the sensitive part of the text. recent works @xcite show that deduplication enables language models to emit memorized text less frequently with same or better accuracy. however, deduplicating training datasets can not prevent all unintended memorization. we combine deduplication and redaction and then apply both techniques to the training process of lm to achieve confidentiality with provable guarantee. the closest to us is perhaps the work of @xcite , who proposed selective differential privacy (s-dp), which requires indistinguishability between two datasets that differ only on a sensitive message. correspondingly, they propose an algorithm (selective dp-sgd) for training rnn that adds noise only to the part of computation that involves sensitive tokens. to define s-dp and to run selective dp-sgd, one needs to have access to a policy function f which determines which token is sensitive. this requirement limits the applicability of their approach to those applications where such perfect f is known. we note that even for name-entity recognition the state-of-the-art model is far from being perfect, and which part of the text is considered sensitive is often ambiguous even for human annotators. we will see that naively running selective dp-sgd with an approximate policy function does not provide a meaningful confidentiality guarantee and is vulnerable to practical data extraction attacks. finally, we note that in the case when a perfect policy function is available, we can simply use it for redaction, which provides a perfect s-dp with ϵ = 0. a big part of our contribution is to refine s-dp to a (slightly different) definition called "confidentiality" and to demonstrate that we use an approximate screening policy to amplify the confidentiality parameter. ## the crt method and theory in this section, we develop our method with provable confidentiality. ## end redact and dedup could be implemented manually, but with the large text corpus nowadays it is more common that these procedures are implemented using automated tools. for example, dedup could be implemented efficiently with just one pass of data using a bloom filter @xcite (or other hashing tricks that also catches nearduplicates). bloom filter in particular, enjoys the nice property that it could have false positives but never any false negatives. redact π could be realized by a named entity recognition (ner) model or a personal-identifiable information (pii) detector. finally, crt combines the two pre-processing steps with normal optimizer and dp-sgd, the standard algorithm for deep learning with differential privacy. a pseudo-code of the algorithm is given in algorithm 1. besides using a sequence labeling policy π with balanced precision/recall as part of the redaction process. the algorithm uses another, more conservative, policy π c with nearly perfect recall to decide on the data points that do not contain sensitive text. in the situation when such π c isn't available, we simply choose π c (s, x) = 1 for all tokens x in a sentence s and the second part becomes the vanila dp-sgd. it is also important that every data point that contains @xmath3. ## experiments we evaluate crt by training two types of language model, lstm and gpt-2, on two datasets: 1) mul-tiwoz 2.2, a well-known human-written dialogue dataset and 2) customersim, a simulated dialogue dataset for conversation generation. multiwoz 2.2 is an already-public dialogue dataset written by crowd-workers, which collects over 10,000 annotated dialogues spanning 8 domains @xcite . we use this dataset to show how crt works in real-world applications. following us department of labor's guidance foot_3 on personal-identifiable information (pii), we treat all confidential information (e.g. email address, reference number, telephone number, etc.) as secrets. for the sequence labeling policy π and conservative policy π c , we build upon an ner model to do redaction. see appendix a.4 for more details. customersim. following s-dp @xcite , we simulate a dialog dataset called customersim with synthetic user information. the dialog flow is simulated based on a fixed agenda and the language generation is template-based @xcite . customersim consists of 10 thousand examples and over one million tokens. we treat user name, address, phone number, order, and tracking number as secrets, and use a regular expression tester (regex) to detect them for the redaction process. experiment details. for lstm model, we follow the setting in s-dp to choose a one-layer lstm. because s-dp requires hidden states of the sensitive input to be protected, it doesn't support more layers nor bidirectional lstm. since the advent of transformers @xcite significantly improves the capabilities of generative language models, we also test transformer-based language model gpt-2 @xcite from huggingface @xcite . as for deduplication, we use sha-1 @xcite hash function to encode sequences to sha-1 hash code and then remove identical sequences based on the same hash code. for bayesian confidentiality, we treat the uniform distribution over the secret sequences as the distribution µ. more experiment details can be found in appendix a.3. baselines. for lstm model, we compare four different training approaches: (1) vanilla sgd (denoted by "non-private-lstm"), (2) selective dpsgd (denoted by "s-dp-lstm") (3) dpsgd (denoted by "dpsgd-lstm") and (4) confidentially redacted training (denoted by "crt-lstm"). while for gpt-2 model, we compare three different training approaches: (1) vanilla sgd (denoted by "non-private-gpt"), ( 2 ) dpsgd (denoted by "dpsgd-gpt") and ( 3 ) crt (denoted by "crt-gpt"). our implementation of s-dp-lstm all the models are trained five times to reduce randomness, and the parameters are tuned based on the validation set performances. ## conclusion in this paper, we propose confidentially redacted training (crt), a method to train language models while protecting the secret texts. we introduce a new definition of confidentiality which quantifies the risk of leaking sensitive content. we prove the effectiveness of crt both theoretically and empirically on multiple datasets and language models.