DP - BART for Privatized Text Rewriting under Local Differential Privacy (2023)

Abstract: Privatized text rewriting with local differential privacy (LDP) is a recent approach that enables sharing of sensitive textual documents while formally guaranteeing privacy protection to individuals. However, existing systems face several issues, such as formal mathematical flaws, unrealistic privacy guarantees, privatization of only individual words, as well as a lack of transparency and reproducibility. In this paper, we propose a new system ‘DP-BART’ that largely outperforms existing LDP systems. Our approach uses a novel clipping method, iterative pruning, and further training of internal representations which drastically reduces the amount of noise required for DP guarantees. We run experiments on five textual datasets of varying sizes, rewriting them at different privacy guarantees and evaluating the rewritten texts on downstream text classification tasks. Finally, we thoroughly discuss the privatized text rewriting approach and its limitations, including the problem of the strict text adjacency constraint in the LDP paradigm that leads to the high noise requirement.

## introduction protection of privacy is increasingly gaining attention in today's world, both among the general public and within the fields of machine learning and nlp. one very common methodology for applying privacy to an algorithm is differential privacy (dp) @xcite . in simple terms, dp provides a formal guarantee that any individual's contribution to a query applied on a dataset is bounded. in other words, no individual can influence this query 'too much'. one particular method of applying dp to the domain of nlp is differentially private text rewriting, in which an entire document is rewritten with dp guarantees by perturbing the original text representations. for instance, given a document "i would like to fly from denver to los angeles this thursday", the system may rewrite it as "show me flights to cities in california this week". if one is training a model on intent classification for airline travel inquiry systems, either document would be a useful data point. in this way, we avoid using the original text that has uniquely identifiable qualities of a specific author, and instead create a privatized 'synthetic' example. this is in fact a form of local differential privacy (ldp), which is a stronger form of dp that is not limited to a specific dataset. the benefits of an ldp text rewriting system are immense, where the output privatized dataset can be used for any downstream analysis. we also avoid the problem of having to manually determine what specific tokens in a document are private, applying ldp to the entire document. however, there is a significant difficulty in creating such a system, with a lot of perturbation required to achieve any reasonable privacy guarantees, leading to poor downstream utility. in addition, there are several issues in existing dp text rewriting systems, such as formal flaws having been discovered in their methodology @xcite , older types of models used (e.g. single-layer lstm, as in @xcite ), high privacy budgets, as well as a lack of transparency in the claimed privacy guarantees, outlined in @xcite . to address these issues, we propose dp-bart, a dp text rewriting system under the local dp paradigm that improves upon existing baselines and consists of several techniques that can be directly applied to a pre-trained bart model @xcite , without having to design and train such a model from scratch. despite being a large transformer architecture, it can be easily used for data privatization, not requiring many resources. our methodology consists of a novel clipping method for the bart model's internal encoder representa-tions, as well as a pruning and additional training mechanism that reduces the amount of dp noise that needs to be added to the data during the privatization process. we summarize our contributions as follows. first, we present our dp-bart model and its related methodologies, aimed at reducing dp noise and reaching a better privacy/utility trade-off. for comparison, we use a reimplementation of the current primary baseline for this task, the adept model. second, we run experiments to investigate the privacy/utility trade-off of these models, using five unique datasets that gradually increase in size, evaluating rewritten texts on downstream text classification tasks. finally, we thoroughly examine the feasibility of the ldp text rewriting setting, investigating issues of the high noise requirement due to the strict text adjacency constraint, trade-offs between privacy and dataset size, what exactly is the object of privatization, required computational resources, as well as limitations of the approach as a whole and possible alternatives. ## related work we present a theoretical background on differential privacy, the bart model, and pruning for neural networks in appendix a. applying differential privacy to neural network training and model publishing has converged to using a mainstream method, namely dp-sgd @xcite . however, the task of text privatization is still broadly unexplored, with many unanswered questions remaining, such as dealing with the unstructured nature of text and explainability of the privacy guarantees provided to textual data @xcite . @xcite ) explored text rewriting with global differential privacy, sampling from a generative language model trained with dp. there are only a few approaches that directly tackle the problem of differentially private text rewriting with ldp. @xcite developed the adept system, which is an rnn-based text autoencoder that incorporates dp noise to its encoder output hidden state. as described by @xcite , adept had a formal error in calculating the laplace noise scale, which resulted in it violating differential privacy. a more recent text rewriting system is dp-vae @xcite , which added constraints to the vanilla vae model latent space @xcite to obtain a bounded sensitivity on its mean and variance parameters. despite the high difficulties of the task, the paper reports surprisingly high performance for high privacy standards. since their experimental description lacks some key details and the code base is not public, we cannot reproduce their approach. in addition, there are a number of word-level dp systems @xcite @xcite , where individual word embeddings are perturbed with dp, with new words then sampled close to these privatized vectors. as mattern et al. (2022b) point out, there are several shortcomings of such approaches, including a lack of obfuscating syntactic information and the inability to provide proper anonymization. in essence, these methods do not privatize a full utterance, but only single words. ## methods we outline this section as follows. first, we briefly describe the baseline method we use, being a modified version of the adept system by @xcite . next, we investigate two main issues with applying a local dp system such as adept to a transformer model, namely extreme sensitivity and computational infeasibility, described in sections 3.2.1 and 3.2.2, respectively. we then demonstrate several novel mechanisms which tackle these issues and provide numerous benefits in the privacy/utility trade-off for the local dp setting. section 3.3 describes the clipping by value module, with an additional analysis on determining optimal settings for it provided in appendix b. sections 3.4 and 3.5 then describe the neuron-based pruning methods which significantly reduce the amount of noise that needs to be added to the model for a given privacy budget and increase model robustness to noise through further noisy training. low-level specifics on the pruning methods are further provided in appendix f. ## results figure 3 shows our downstream test f 1 results for all datasets, at varying values of ε. we report results for the analytic gaussian mechanism, which nearly always outperformed those of the laplace mechanism. we present results in tabular form with mean and standard deviations in appendix k. additionally, we present sample rewritten texts in appendix l. we outline the main patterns as follows. dp-bart-pr+ performs best dp-bart-pr+ reaches the best privacy/utility trade-off for the majority of datasets, having the highest scores at the lower ε values. dp-bart-pr results are secondbest for most datasets, performing better than dp-bart-clv and adept, which are low for the majority of configurations. the overall results hierarchy can be clearly seen in the snips dataset, where at ε = 500, dp-bart-pr+ reaches f 1 0.65, dp-bart-pr at 0.39, while both dp-bart-clv and adept are below f 1 0.15. original vs. rewritten results for the original setting are generally on-par with those of the rewrite-no-dp setting. for instance, snips original f 1 is 0.98, and ε = ∞ with rewriting is also at f 1 of 0.98 for dp-bart-pr, being very similar for the other three models. one exception to this is imdb, which has a drop from original f 1 0.86 to 0.72 for all models. this can be explained by the fact that the original settings use longer sequence lengths, while both rewrite-no-dp and rewrite-dp settings are limited to a sequence length of 20. this is not a problem for datasets such as atis and snips, since their documents are generally very short, mostly limited to brief user inquiries. for a dataset such as imdb, however, which consists of detailed reviews by individuals, limiting the sequence length results in a loss of valuable information. epsilon vs. dataset size regardless of dataset size, we can see a drop in results for all models as ε is decreased. with the models incorporating pruning, this drop appears at later ε values, such as dp-bart-pr+ on the amazon dataset moving down from f 1 0.82 at ε = 250 to f 1 0.33 at ε = 100, and dp-bart-pr from f 1 0.79 at ε = 500 to f 1 0.33 at ε = 250. a similar pattern can be seen for the snips dataset, despite being far smaller than amazon, while the drugs.com dataset shows low results throughout, for all model types. the smallest dataset, atis, also performs poorly, which can be explained by the large number of classes and few data points for learning the task in the noisy setting. we can generally see that a larger dataset size does not necessarily mean better results at lower ε values, although the significantly larger amazon dataset does show the best results. ## discussion reducing noise for text rewriting with ldp we have shown that it is possible to reduce the amount of noise in the ldp setting of privatized rewriting, in order to obtain more useful rewritten texts for downstream tasks. to compare dp-bart-clv vs. dp-bart-pr, we can examine the resulting ℓ 2 sensitivity from equation 8 ( pre-training and computational resources ultimately, a very effective way to prepare a model for privatized text rewriting would be to pre-train it from scratch, being fully in control of hyperparameters such as the dimensionality n of the encoder output vectors z, which determines the ℓ 1 and ℓ 2 sensitivities from equations 7 and 8, respectively. in addition, the whole model could be pre-trained with added noise and clipping mechanisms, potentially being even more robust than our approach in dp-bart-pr+, where we incorporate further noisy training. we noticed for dp-bart-pr+ that the lower the ε value we use, the more additional training iterations the model needs to properly reduce the validation loss. this demonstrates that, also in the setting of pre-training from scratch, we would need to train for more iterations in order to reach lower ε values. this can pose serious challenges, however, for reasons of computational demand discussed in section 3.2.2. dp-bart-pr+ can therefore be seen as a sweet spot approach, where we only need a few additional training iterations and can still achieve a significant dimensionality reduction through pruning, as well as additional robustness to noise. what is being privatized it is very important to be clear on exactly what information is being privatized when performing text rewriting with ldp. since we are working with dp at the document level, the entire document is a 'data point', hence any choice and combination of words for a given sequence would be a unique identifier. we thus avoid the problem of having to choose what specific tokens are 'private' within the document. this is crucial, since stylistic aspects of an author can be very abstract, with subtle syntactic and vocabulary choices. another significant benefit of such an approach, is that we are not limiting ourselves to any specific downstream analysis (e.g. sentiment of a document), being task agnostic. however, this also means that, for any given document, any other document is neighboring, since we are in the ldp setting. this leads us to a serious discussion on the limitations of such an approach in section 8. an additional question arises of whether one dataset may have multiple documents associated with one individual. there are several ways to go about dealing with this. one standard approach in differential privacy is to linearly scale the ε parameter. thus, if there are k documents associated with a given individual, then a privacy budget of kε is accounted in total @xcite . another option would be to simply append all texts associated with one individual into a single 'document', rewriting this using just a single ε privacy budget. ## conclusion we have proposed dp-bart, a novel methodology for ldp-based privatized text rewriting, which outperforms existing methods. we have demonstrated our method's privacy/utility trade-off, the relations between the privacy budget and dataset size, and discussed limitations of the privatized text rewriting approach as a whole. future research directions include utilizing large-scale pre-training to potentially reach a better privacy/utility trade-off, as well as investigating domain specific text rewriting for relaxing the strict requirements of the ldp approach. the public data used for our experiments consists of news, web text, stories and books @xcite . we expect that expanding this to include more data and more varied domains will lead to better performance in a greater diversity of texts and downstream tasks. ldp for text rewriting for every output document, any two inputs, no matter how similar or distinct, are considered neighboring. if we have a small sequence length of 20 tokens, with a relatively small vocabulary of 1000 words, then the total number of possible combinations is 1000 20 , which is 10 60 ! while we compress these documents into a latent vector with a limited range and dimensionality, the strict adjacency constraints are still present. we can therefore expect an inevitable utility drop when using more reasonable ε values (e.g. ε = 1). with more sophisticated architectures, we have shown that it is possible to push this ε value down to some extent. however, our lowest ε is still too high to carry over into real-world applications of privacy preservation. as outlined by @xcite , values of ε for different applications in the dp literature can range from 0.01 to 10. choosing the right ε value depends on the specific queries that are computed and the nature of the data @xcite . for our case, the value of ε can be interpreted in the following manner. the ε-ldp mechanism that we are applying to our data makes any two input texts rewritten to be indistinguishable up to a factor of e ε . more formally, for any two input texts x and y to our ldp model m: it may therefore make sense to take a slightly less strict approach to text adjacency, for instance moving into domain specific text rewriting. for example, text rewriting could be carried out for a specific dataset, with the notion of adjacency restricted to any two individuals within that dataset, hence requiring much less perturbation. the strength of the privacy guarantee, in this case, would then be very dependent on the size of the dataset @xcite . akhil shiju and zhe he. 2022. classifying drug ratings using user reviews with transformer-based language models. in 2022 ieee 10th international conference on healthcare informatics (ichi), pages 163-169. ashish vaswani, noam shazeer, niki parmar, jakob uszkoreit, llion jones, aidan n gomez, łukasz kaiser, and illia polosukhin. 2017. attention is all you need. advances in neural information processing systems, 30. teng wang, xuefeng zhang, jingyu feng, and xinyu yang. 2020. a comprehensive survey on local differential privacy toward data statistics and analysis. sensors, 20(24):7030. benjamin weggenmann, valentin rublack, michael andrejczuk, justus mattern, and florian kerschbaum. 2022. dp-vae: human-readable text anonymization for online reviews with differentially private variational autoencoders. in proceedings of the acm web conference 2022, pages 721-731, virtual event. acm. zekun xu, abhinav aggarwal, oluwaseyi feyisetan, and nathanael teissier. 2020. a differentially private text perturbation method using regularized mahalanobis metric. in proceedings of the second workshop on privacy in nlp, pages 7-17, online. association for computational linguistics.