Learning and Evaluating a Differentially Private Pre-trained Language Model (2021)

Abstract: Contextual language models have led to significantly better results, especially when pre-trained on the same data as the downstream task. While this additional pre-training usually improves performance, it can lead to information leakage and therefore risks the privacy of individuals mentioned in the training data. One method to guarantee the privacy of such individuals is to train a differentially-private language model, but this usually comes at the expense of model performance. Also, in the absence of a differentially private vocabulary training, it is not possible to modify the vocabulary to fit the new data, which might further degrade results. In this work we bridge these gaps, and provide guidance to future researchers and practitioners on how to improve privacy while maintaining good model performance. We introduce a novel differentially private word-piece algorithm, which allows training a tailored domain-specific vocabulary while maintaining privacy. We then experiment with entity extraction tasks from clinical notes, and demonstrate how to train a differentially private pre-trained language model (i.e., BERT) with a privacy guarantee of \epsilon=1.1 and with only a small degradation in performance. Finally, as it is hard to tell given a privacy parameter \epsilon what was the effect on the trained representation, we present experiments showing that the trained model does not memorize private information.

## introduction recent advancements in natural language processing (nlp), mainly the introduction of the transformer architecture and contextual language representations, have led to a surge in the performance and applicability of large language models @xcite . such models rely on pre-training on massive selflabeled corpora to incorporate knowledge within * tel-hai college, israel. work was done while at google. the language representation. additionally, when presented with a new dataset and task, such models often gain from an additional pre-training stage, where they are trained to solve a language modeling task on the new training data. while the pre-training steps are crucial for good model performance on downstream tasks, it can come at the expense of the privacy of the persons mentioned in the data. as these models learn to predict words using their context, they often memorize individual words and phrases. such memorization can lead to information leakage when using the trained models or the language representation. this problem is particularly acute in medical domains, where sensitive patient data might leak @xcite . one solution for pre-training the model while preserving patients' privacy is to train the model with a differential privacy guarantee @xcite . such guarantee is achieved through a training process which introduces random noise, allowing the modeler to bound the effect an individual has on the model. however, for a sufficiently small privacy parameter , this usually comes at the expense of model performance. also, differentially private training schemes were only shown to work for recurrent language models, and not for more recent systems that are based on the transformer architecture @xcite . apart from their size (110m trainable parameters for bert), transformer-based language models introduce an additional privacy concern. when using pre-trained language models on new datasets, we can often improve performance by learning a new domain-specific vocabulary, and re-training the model with the new tokenizer (section 5). unfortunately, commonly used transformer-based models such as bert rely on the wordpiece tokenization algorithm @xcite , which uses the distribution of words in the data and can therefore potentially leak private information as well. finally, even if we successfully train a contextual embedding model with a sufficiently small guarantee, it is hard to test and evaluate the resulting privacy-preserving properties of the model. one also has difficulty understanding whether the differentially-private training procedure affected the language representation other than by measuring performance on a downstream task. for example, it could be that other valuable information was also lost during the differentially private training. in this work, we provide a detailed solution to training a differentially-private vocabulary and contextual embedding model, and to better understanding the resulting representation. we present a method for training bert, a contextual embedding model, on medical data with a strong privacy guarantee @xmath0.1 in total (including the private word-piece and pre-training algorithms) and with only a small degradation in performance (section 2.1). to do that, we introduce the first differentially private wordpiece algorithm, designed to generate a new domain-specific vocabulary while maintaining user privacy (section 3.2). following that, we successfully generate a differentially private bert model, which uses the new vocabulary to improve results on the downstream tasks. possibly the most major technical challenge in pre-training a differentially-private contextual embedding model is the fact that the training batch size has to be very large (128k), all the while training on specific hardware (tpus) in which the batch size is limited. we overcome this obstacle by spreading each training batch over time during the training process, along with other useful manipulations we discuss in section 2.1. finally, after training the differentially-private bert on clinical notes, we follow common wisdom @xcite and provide privacy tests showing that information leakage has been prevented in this process (section 5). we hope that this work will further improve user privacy, and will spur more theoretical and empirical research in the intersection of differential privacy and natural language processing. ## previous work since the introduction of the differentially-private stochastic gradient descent (sgd) algorithm @xcite , it is possible to train deep neural networks (dnn) with privacy guarantees. specifically, there have been sev-eral attempts to train dnn-based language models with such guarantees, though with mixed results in terms of performance on downstream tasks @xcite . to better understand the trade-offs between the performance and privacy of deep language models, we survey here the literature on differentially-private training and on methods for measuring privacy in language models. ## training differentially private contextual language models training differentially private language models becomes exceedingly difficult with model size. hence, attempting to train a transformer model such as bert using the dp-sgd algorithm, without any modifications, is bound to result in a significant performance degradation @xcite . moreover, as the wordpiece algorithm, the process that tokenizes the textual input of bert, is not differentially private, re-training it to fit a domain-specific vocabulary will not guarantee that there is no information leakage regardless of the dp-sgd training. in this section, we formulate the problem of training a dp bert model on medical text, and explain the process of constructing a differentially private vocabulary. we then discuss the importance of parallel training and very large batch sizes in training such large language models, and provide a method for sufficiently increasing such crucial parameters. ## experimental setup we design our experiments to demonstrate the ability of the dp training scheme to achieve similar results to the non-dp training scheme on the same data. we focus on the medical domain as it has strict privacy requirements and its language is distinct enough so that additional pre-training should be useful. we start by describing the data used for the dp training and relevant implementation details. we then present the entity extraction task used for the supervised task training and evaluation. finally, we discuss the relevant baselines, chosen to demonstrate the efficacy of the dp training scheme. pre-training data. for the dp pre-training, we supplement the original training data used in @xcite with the mimic-iii dataset, a commonly used collection of medical information that contains more than 2 million distinct notes @xcite . mimic-iii covers 38,597 distinct adult patients and 49,785 hospital admissions between 2001 and 2012. the clinical notes in this dataset are widely used by nlp researchers for a variety of clinically-related tasks @xcite , and were previously used for pre-training bert models specifically for the medical domain @xcite . using the combined dataset, we train our dp-bert model using the training scheme described in section 3. ## results in this section we empirically evaluate the tradeoffs between a model's privacy and its usefulness. previously, in section 3, we have shown how to pretrain a contextual embedding model such as bert with any, possibly substantial, privacy guarantee. we naturally expect that a stronger privacy guarantee would entail that less information is preserved during pre-training, which in turn would degrade performance on downstream tasks. thus, we aim to ascertain the exact trade-off between these two goals in order to be able to choose a model that has both good performance and a satisfactory privacy guarantee. we provide two sets of experiments to help better understand this trade-off as well as to provide practitioners with tools to understand the effects of dp pre-training. first, we use the pre-trained dp model and fine-tune it on the entity extraction task on both i2b2-2010 and i2b2-2011, demonstrating the ability of the differentially private language model to benefit from the pre-training step. then, we test the ability of the model to memorize private information and show that it is protected against commonly used privacy attacks. aggregating both results, we argue that medically-relevant information is preserved in the dp model all the while private information is not revealed. for all our model variants, unless explicitly stated otherwise, the parameters are as discussed in section 3, with batch size @xmath1, noise multiplier σ = 2.72, and 1m training steps. ## discussion and future work in this paper, we have shown a procedure for learning and evaluating a differentially-private contextual language model. we have defined the problem of learning such a model with end-to-end privacy guarantees and have discussed the pitfalls that might lead to poor downstream performance. allowing for vocabulary modifications, we have introduced a novel wordpiece algorithm and proved that it is differentially-private. then, to overcome the difficulties associated with learning dp contextual language models, we have offered practical measures for circumventing them, most notably through vastly increasing batch sizes. finally, to increase the trust of the dp trained contextual language model, we have utilized a secret sharer evaluation test and showed that our trained language model does not memorize private information. while these results are definitely encouraging, more research is needed. our results are confined to the medical domain, where privacy needs are perhaps most stringent. showing the efficacy of this training and evaluation pipeline on other domains would certainly increase the trust in it. additionally, we have not fully explored potentially tighter bounds on our dp wordpiece algorithm. in future work, we plan to provide more theoretical and empirical support for end-to-end privacy guarantees. finally, the observed performance gain due to the vocabulary training presents an interesting question for the larger nlp community. understanding the importance of vocabulary vs. linguistic style when performing additional pre-training could improve the domain adaptation capabilities of existing nlp systems. in future work, we plan to expand our dp training to additional domains, allowing us to test the power of vocabulary modifications via the dp wordpiece training in increasing across domain performance. yukun zhu, ryan kiros, richard s zemel, ruslan salakhutdinov, raquel urtasun, antonio torralba, and sanja fidler. 2015. aligning books and movies: towards story-like visual explanations by watching movies and reading books. in iccv.