Granularity is crucial when applying differential privacy to text: An investigation for neural machine translation (2024)

Abstract: Applying differential privacy (DP) by means of the DP-SGD algorithm to protect individual data points during training is becoming increasingly popular in NLP. However, the choice of granularity at which DP is applied is often neglected. For example, neural machine translation (NMT) typically operates on the sentence-level granularity. From the perspective of DP, this setup assumes that each sentence belongs to a single person and any two sentences in the training dataset are independent. This assumption is however violated in many real-world NMT datasets, e.g., those including dialogues. For proper application of DP we thus must shift from sentences to entire documents. In this paper, we investigate NMT at both the sentence and document levels, analyzing the privacy/utility trade-off for both scenarios, and evaluating the risks of not using the appropriate privacy granularity in terms of leaking personally identifiable information (PII). Our findings indicate that the document-level NMT system is more resistant to membership inference attacks, emphasizing the significance of using the appropriate granularity when working with DP.

## introduction with increasing concerns about the privacy of individuals and data leakage from nlp systems @xcite , a method that has gained popularity in privacy-preserving nlp is differential privacy (dp) @xcite . however, the exact manner in which dp is applied to a textual dataset has numerous pitfalls. the unit of privacy is one among them, i.e. the granularity at which we assume an individual 'data point' (e.g. sentences, documents, and so forth) @xcite , with an assumption of independence among data points @xcite ). 1 our code is available at https://github.com/ trusthlt/granularity-is-crucial-dp . one particular task that has recently raised many privacy concerns is neural machine translation (nmt). applying dp at the sentence level for nmt may break the independence assumption if more than one sentence is associated with a single individual @xcite , as depicted in figure 1 . in such cases, scaling up the unit of privacy to the document level by grouping related sentences overcomes the violated privacy protection which 'pretends' all sentences are independent, i.e. the status quo. the main objective of this paper is to compare the use of dp for nmt systems and datasets at the sentence and document levels, focusing on the level of privacy protection offered. first, we investigate the trade-off between privacy and utility for different levels of granularity (sentence vs. document) during the training process of an nmt system. specifically, we examine how performance is affected when applying dp with varying levels of privacy guarantees and granularity. secondly, we aim to evaluate the risks of not using a proper privacy granularity during the training process of an nmt system through data extraction attacks on these systems. our contributions are as follows. (1) we propose a novel approach to apply differential privacy to nmt systems at the document level, uti-lizing the dp-nmt framework @xcite and the mlongt5 model @xcite . the evaluation results show that the document-level nmt system is extremely sensitive to the privacy budget (ε), which can significantly affect performance and lead to a drop in utility. we therefore suggest training the document-level nmt system on a larger, non-sensitive dataset, such as wmt22 @xcite , to achieve a better trade-off between privacy and utility on the downstream dataset with dp. (2) we apply the lossbased membership inference attack (mia) @xcite to detect private information in nmt systems at both the sentence and document levels. based on this mia, we create an evaluation schema for personally identifiable information (pii) to estimate the percentage of potential information leakage. our results show that the document-level nmt system is more robust against the loss-based mia than the sentence-level nmt system, demonstrating the importance of using the proper granularity when working with dp. ## conclusion we have examined the privacy leakage of sentencelevel vs. document-level approaches using the loss-based membership inference attack on the mlongt5 model. the results show that a sentencelevel model has more risks of privacy leakage than a document-level model. specifically, a sentencelevel model is more likely to overfit compared to a document-level model, which can lead to more confident guessing of sentence-level training instances. furthermore, regarding the privacy/utility tradeoff in the document-level model, optimizing transformer models for long texts, especially with dp-sgd, is a challenging task that requires more data than our downstream dataset. we demonstrate our solution to this problem with an augmented training technique, using a large public dataset, in our case wmt22, to achieve an acceptable utility, then fine-tune on the downstream dataset to achieve the best privacy/utility trade-off. for future work, we aim to design a better mia that takes into account the correlation aspect of nlp datasets.