Learning and Evaluating a Differentially Private Pre-trained Language Model (2021)

Abstract: Contextual language models have led to significantly better results on a plethora of language understanding tasks, especially when pre-trained on the same data as the downstream task. While this additional pre-training usually improves performance, it can lead to information leakage and therefore risks the privacy of individuals mentioned in the training data. One method to guarantee the privacy of such individuals is to train a differentially-private model, but this usually comes at the expense of model performance. Moreover, it is hard to tell given a privacy parameter \epsilon what was the effect on the trained representation. In this work we aim to guide future practitioners and researchers on how to improve privacy while maintaining good model performance. We demonstrate how to train a differentially-private pre-trained language model (i.e., BERT) with a privacy guarantee of \epsilon=1 and with only a small degradation in performance. We experiment on a dataset of clinical notes with a model trained on a target entity extraction task, and compare it to a similar model trained without differential privacy. Finally, we present experiments showing how to interpret the differentially-private representation and understand the information lost and maintained in this process.

## introduction recent advancements in natural language processing (nlp), mainly the introduction of the transformer architecture and contextual language representations, have led to a surge in the performance and applicability language models. such models rely on pre-training on massive self-labeled corpora to incorporate knowledge within the language representation. additionally, when presented with a new dataset and task, such models often gain from an additional pre-training stage, where they * work was done while at google. are trained to solve a language modeling task on the new training data. while the pre-training steps are crucial for good model performance on downstream tasks, it can come at the expense of the privacy of the persons mentioned in the data. as these models learn to predict words using their context, they often memorize individual words and phrases. such memorization can lead to information leakage when using the trained models or the language representation. this problem is amplified in medical domains, where patients data might leak and expose protected health information (phi). one solution for pre-training the model while preserving patients' privacy is to train the model with a differential privacy guarantee. however, for a sufficiently small privacy parameter , this usually comes at the expense of model performance. also, it was only shown to work for recurrent language models, and not for more recent systems that are based on the transformer architecture @xcite . apart from their size (our model has 109m trainable parameters), transformer-based language models introduce an additional privacy concern, as their reliance on wordpiece based tokenization algorithm can also potentially leak private information. moreover, even with a sufficiently small guarantee, it is hard to test and evaluate the resulting privacy-preserving properties of the model. one also has difficulty understanding whether the differentially-private training procedure affected the language representation other than by measuring performance on a downstream task. for example, it could be that other valuable information was also lost during training. in this work we provide here a detailed solution to training a differentially-private contextual embedding model, and to better understand the resulting representation. we start by presenting a method for training bert, a contextual embedding model, on medical data with a strong privacy guarantee @xmath0. possibly the most major technical challenge in doing so is the fact that the training batch size has to be fairly large, all the while training on specific hardware (tpus) in which the batch size is limited. we overcome this obstacle by distributing each training batch over time during the training process, along with other useful manipulations (section 2.1). as these models gain from retraining the wordpiece algorithm on the target dataset, we propose a differentiallyprivate wordpiece algorithm, preventing additional information leakage through the model's vocabulary (section 3.2). after training the differentially-private bert on clinical notes, we follow common wisdom and provide privacy tests to show that information leakage has been prevented in this process (section 5). we further provide adversarial attacks that can help understand the privacy guarantees in terms of memorized words and phrases. these tests, when combined, provide a useful toolbox for understanding how "private" is the differentially-private model. ## previous work since the introduction of the differentially-private stochastic gradient descent (sgd) algorithm @xcite , it is possible to train deep neural networks (dnn) with privacy guarantees. specifically, there have been several attempts to train dnn-based language models with such guarantees, though with mixed results in terms of performance on downstream tasks @xcite . to better understand the trade-offs between the performance and privacy of deep language models, we survey here the literature on differentially-private training and on methods for measuring privacy in language models. ## training differentially private contextual language models training differentially private language models becomes exceedingly difficult with model size. as such, attempting to train a transformer model such as bert using the dp-sgd algorithm and without any modifications will usually lead to a significant performance degradation @xcite . moreover, as the wordpiece algorithm, the process that tokenizes the textual input of bert, is not differentially private, training will not guarantee that there is no information leakage. in this section, we formulate the problem of training a dp bert model on medical text, and explain the process of constructing a differentially private vocabulary. we then discuss the importance of parallel training and very large batch sizes in training such large language models, and provide a method for sufficiently increasing such crucial parameters. ## experimental setup we design our experiments to demonstrate the ability of the dp training scheme to improve performance while preserving privacy. we focus on the medical domain as it has strict privacy requirements and its language is distinct enough such that additional pre-training should be useful. we start by describing the data used for the dp training and relevant implementation details. we then present the entity extraction task used for the supervised task training and evaluation. finally, we discuss the relevant baselines, chosen to demonstrate the efficacy of the dp training scheme. pre-training data. for the dp pre-training, we supplement the original training data used in @xcite with the mimic-iii dataset, a commonly used collection of medical information that contains more than 2 million distinct notes @xcite . mimic-iii covers 38,597 distinct adult patients and 49,785 hospital admissions between 2001 and 2012. the clinical notes in this dataset are widely used by nlp researchers for a variety of clinically-related tasks @xcite , and were previously used for pre-training bert models specifically for the medical domain @xcite . using the combined dataset, we train our dp-bert model using the the training scheme described in section 3. at this point, we use a word-piece vocabulary generated from mimic-iii without privacy guarantees. entity-extraction task. for the supervised task training, we use i2b2-2010, a dataset from the i2b2 national center for biomedical computing for the nlp shared tasks challenges @xcite . this dataset contains clinical notes tagged for concepts, assertions, and relations. in this task, 170 patient reports are labeled with three concepts: test, treatment, and problem. the total number of entities in each category are as follows: baselines. we compare our differentially private bert model, denoted as dp bert, to several non private baselines: bert (wikipedia + books) we train a bertlarge model, as in @xcite , using the default hyperparameters. bert-m (wikipedia + books + mimic-iii) we supplement the original training from @xcite with the mimic-iii clinical notes corpus. in addition, we also use a (non-differentially private) wordpiece vocabulary generated from mimic-iii. biobert we use the training data presented in @xcite , and use it to train bert. we tested version v1.1 which it trained using the original dataset + 1m pubmed abstracts. in section 5 we compare several differentially private models, discuss their differences and highlight the effect of certain parameters (as discussed in section 3) on the ee task performance. ## results in this section we empirically evaluate the tradeoffs between a model's privacy and its usefulness. previously, in section 3, we have shown how to pretrain a contextual embedding model such as bert with any, possibly substantial, privacy guarantee. we naturally expect that a stronger privacy guarantee would entail that less information is preserved during pre-training, which in turn would degrade performance on downstream tasks. thus, we aim to ascertain the exact trade-off between these two goals in order to be able to choose a model that has both good performance and a satisfactory privacy guarantee. we provide two sets of experiments to help better understand this trade-off as well as to provide practitioners with tools to understand the effects of dp pre-training. first, we use the pre-trained dp model and fine-tune it on the aforementioned ee task. then, we test the ability of the model to memorize private information and show that it is protected against commonly used privacy attacks. aggregating both results, we argue that medicallyrelevant information is preserved in the dp model all the while private information is not revealed. ## discussion and future work in this paper, we have shown a pipeline for learning and evaluating a differentially-private contextual language model. we have defined the problem of learning such a model with end-to-end privacy guarantees and have discussed the pitfalls that might lead to poor downstream performance. to overcome the difficulties associated with learning such models, we have offered practical measures for circumventing them, most notably through vastly increasing batch sizes. then, to increase the trust of the dp trained contextual language model, we have utilized a secret sharer evaluation test and showed that our trained language model does not memorize private information. while these results are definitely encouraging, more research is needed. our results are confined to the medical domain, where privacy needs are perhaps most stringent. showing the efficacy of this training and evaluation pipeline on other domains would certainly increase the trust in it. additionally, we have not yet measured the model's performance with the dp wordpiece algorithm. in future work, we plan to provide more theoretical and empirical support for end-to-end privacy guarantees. finally, the observed performance gain due to the vocabulary training presents an interesting question for the larger nlp community. understanding the importance of vocabulary vs. linguistic style when performing additional pre-training could improve the domain adaptation capabilities of existing nlp systems. in future work, we plan to expand our dp training to additional domains, allowing us to test the power of vocabulary modifications via the dp wordpiece training in increasing across domain performance.