Paper: Split-and-Denoise: Protect large language model inference with local differential privacy (ICLR-2024)

Reviewer comment (Theory):
Recent work by "Mattern et al. - The Limits of Word Level Differential Privacy" has pointed out limitations of dx-privacy. The author should address these weaknesses and concerns related to dx-privacy in the paper.

Author response:
**Q6. Limitations of $d_{\chi}$-differential privacy**

The method proposed by "Mattern et al. - The Limits of Word Level Differential Privacy" couldn’t be directly applied to our framework, as it involves repharasing the sentence and thus is not compatible with the denoise model. Indeed, $d_{\chi}$-DP has been used frequently for text inputs in recent research [1]. Below we present some discussions on the weakness of the $d_{\chi}$-DP bound:

-	Linear growth of privacy budget with token length. In the experiment, we limited the input length within 512 tokens. We also conducted inference attack on the privatized token representations to verify the privacy protection capability. Furthermore, in Section A.11, we show that our approach retains the performance even under extremely low privacy budgets, indicating that the user could set a very low privacy budge in case of possible leakage for long text inputs.
-	Lack of syntactic changes. The syntactic properties of the original text could be hided when the privacy budget is set to a very low level.
-	Limitation for usefulness of the text data on local DP. To preserve the utility of the subsequent output, we deploy a denoise model on user side to conduct error correction.

[1] Feyisetan, O., Balle, B., Drake, T., & Diethe, T. (2020, January). Privacy-and utility-preserving textual analysis via calibrated multivariate perturbations. In Proceedings of the 13th international conference on web search and data mining (pp. 178-186).

Reviewer comment (Evaluation):
The denoise model is pre-trained and known by the server. Could the server utilize the Denoise model to obtain less noisy user embeddings and potentially compromise privacy? This issue should be explored.

Author response:
**Q1. Privacy concerns of denoise model.**

The training of denoise model relies solely on the public dataset, and thus the denoise model is irrelavant to the private data. We added the clarification at the end of Section 3.4. Furthermore, the input to the denoise model requires the knowledge of specific noise matrix, which is not accessible by the server.

Reviewer comment (Experiments):
The impact of knowing the noise level or not should be more thoroughly analyzed.

Author response:
**Q2. The impact of knowing the noise level or not**

We added theoretical (see Section 3.4) and empirical studies (see Section A.10) to assess the impact of noise level awareness. For theoretical analysis, we showed that the denoising ability is inherently in tension with the level of privacy protection and provided a lower bound on the MSE of denoised results. For empirical analysis, we run the experiment that deploys the denoise model on server side using only privatized token representations as input. The results show that SnD significantly outperforms the server-side denoise method by over 10% in almost all cases for MRPC task.

Reviewer comment (Experiments):
The paper does not adequately demonstrate the privacy aspects regarding textual input. There is a lack of evidence that the proposed method can effectively defend against potential inversion attacks compared to an unprotected scheme.

Author response:
**Q3. Empirical evaluation of privacy protection & different $\eta$s**

Refering to general response to Q1, we added experiment considering the inference attacks in Section 4.3 and 4.4.3. We found that the three classes of models show different robustness exhibit distinct robustness against inference attacks, and thus the privacy budget varies across the models. The $\eta$ is selected that meets the following two criteria simultaneously: (a) the accuracy for embedding inversion attack is below 0.01, and (b) the accuracy for attribute inference is below 0.53 (the accuracy for random guessing is around 0.5). 

Regarding the interpretation of $\eta$, it can be viewed as a parameter controlling the indistinguity between the distribution of the randomized mechnisms with two possible inputs $x$, $x'$. The log-likelihood ratio of observing any particular output $y$ given two inputs $x$, $x'$ is bounded by $\eta d(x, x')$, where $d(x, x')$ is the distance metric of the two inputs. In this paper, we adopted the Euclidean norm as the distance metric.

Reviewer comment (Writing):
The paper creates some confusion due to the mixed usage of "eta" in both its symbol and word forms.

Author response:
**Q4. Typo errors and expression issues**

Thanks for pointing out. We corrected the typos and used consistent $\eta$ in the paper.

Reviewer comment (Experiments):
What does eta mean for privacy? Why can it be set differently for GPT, Bert, and T5 models?

Author response:
**Q3. Empirical evaluation of privacy protection & different $\eta$s**

Refering to general response to Q1, we added experiment considering the inference attacks in Section 4.3 and 4.4.3. We found that the three classes of models show different robustness exhibit distinct robustness against inference attacks, and thus the privacy budget varies across the models. The $\eta$ is selected that meets the following two criteria simultaneously: (a) the accuracy for embedding inversion attack is below 0.01, and (b) the accuracy for attribute inference is below 0.53 (the accuracy for random guessing is around 0.5). 

Regarding the interpretation of $\eta$, it can be viewed as a parameter controlling the indistinguity between the distribution of the randomized mechnisms with two possible inputs $x$, $x'$. The log-likelihood ratio of observing any particular output $y$ given two inputs $x$, $x'$ is bounded by $\eta d(x, x')$, where $d(x, x')$ is the distance metric of the two inputs. In this paper, we adopted the Euclidean norm as the distance metric.

Reviewer comment (Experiments):
Can this be extended to training (finetuning)?

Author response:
**Q5. Extension to training/finetuning**

That’s a good question. While the privatization method described in our paper can be utilized during the training stage, directly applying the denoise model to the training/finetuning stage would pose a significant challenge. It would be another field of research to protect LLM training with local differential privacy while maintaining utility.

Reviewer comment (Experiments):
The overhead introduced by the local encoder in the proposed approach is not clearly presented. While Section 3.5 provides a basic complexity analysis, a more comprehensive evaluation involving specific experiments is recommended to properly assess the feasibility of the approach. Additionally, it remains unclear how the denoise model, pre-trained on the server, can adapt to varying noise levels determined by clients.

Author response:
**Q1. The overhead of proposed approach**

We added experiment in Appendix A.9 to evaluate overhead of SnD. Refer to the general response to Q3.

**Q2. Denoise model for varying $\eta$ levels**

In Appendix A.6 we added the specification of training the denoise model. Simply speaking, we devide the $\eta$ in to three groups according to the correlation between the privatized and plain token representation tokens, and train a seperate denoise model for each group. During inference, the user could choose the corresponding denoise model based on their privacy requirement.

Reviewer comment (Experiments):
There is a lack of detail in the baseline description. The only information provided about the benchmark method is that “the token embeddings are perturbed by the user before sending them to the server.” More information about TokEmbPriv should be provided, including why this method was chosen as the sole benchmark in the evaluation. Furthermore, it should be clarified why other methods, such as a vanilla model, were not considered for comparison.

Author response:
**Q3. Baseline description & lack of baseline**

Refering to the answer to Q2 in general response, we added two baselines in the experiments. We also present more illustration of the baseline methods in “prior works” and Appendix A.1. The accuracy performance for a vanilla model is presented in the case of $\eta=\infty$ in table 1, 2, and 3, where no noises are added and denoise model is not deployed on user side.

In the latest version, we have added the description and implementation details for the baseline methods in Section A.6.5.

Reviewer comment (Evaluation):
The evaluation of the denoising process is somewhat unclear. It should be explicitly stated and explained why a higher Cosine Similarity score between the initial and recovered embeddings indicates better performance. A clear definition of what "initial embeddings" are is also needed—if they refer to the embeddings input to the denoise module (i.e., the "output noised results" in Figure 1), the current description suggests that fewer differences between the initial and recovered embeddings are preferred, which might imply that less noise is added. Furthermore, the evaluation focuses only on accuracy and does not include results on privacy protection. Since the primary motivation of the study is to address the "unaddressed risk of privacy leakage," the privacy performance of the proposed approach should be included in the evaluation.

Author response:
**Q4. Privacy performance of the proposed method**

Refering to general response to Q1, we added experiment considering the inference attacks in Section 4.3 and 4.4.3.

**Q5. Interpretation of Cosine Similarity**

The cosine similarity measures the similarity between the clean and denoised embeddings in the case of SnD (or privatized embedding in the case of text2text and tokEmbPriv). Clean embedding refers to the output embedding computed from the plain token embeddings, and denoised embedding refers to the output embedding derived from the privatized token embeddings and transformed by the denoise model. A higher cosine similarity means that the denoised embedding is more similar to the original clean embeddings and thus is prefered. We make the expression clearer in Section A.8.

Reviewer comment (Evaluation):
The featured classification tasks do well to test the performance of BERT and T5. However, they do not relevantly test the performance of GPT-2; in practice, GPT-2 does not perform well on classification tasks, which would ideally be handled by more appropriate encoder (e.g., BERT) or encoder-decoder (e.g., T5) architectures. More relevant generative metrics, such as sentence completion (e.g., via HellaSwag), next-word-completion (e.g., via the lambada dataset), or even perplexity are required to show that GPT-2 performance is maintained.

Author response:
**Q1. Testing of GPT2 on classification task**

The autoregressive GPT model has been evaluated on classification tasks in recent research [1]. Furthermore, OpenAI has released the Embedding-as-a-Service (EaaS) for downstream tasks including classification [2], where GPT2 and 3 are provided as the underlying models. Therefore, it’s of pratical concerns to test the performance of GPT2 on the downstream tasks given by the paper. Meanwhile, we are conducting experiment of GPT2 on HellaSwag and lambada dataset, but we haven't completed it due to computer resource and time limit. We will add the results to the paper once the experiments are done.

Reviewer comment (Experiments):
For homomorphic encryption approaches, the authors claim that these techniques are constrained by significant computation overhead. To assert this claim, the proposed method and reference HE methods should be compared in terms of both accuracy, privacy-preserving ability, and wall-clock time.

Author response:
In Section A.9, we evaluated the computation and communication overhead for two encryption-based method [3] [4], and found that our method results in thousands of speedup for the SOTA HE algorithms. The privacy comparison between PCFT and SnD can be found in Section A.1, where PCFT adopts the security definition in multiparty computation while SnD protect the data with LDP.

Reviewer comment (Novelty):
The description of RAPT (which itself offers LDP for privacy-preserving LLM inference) is inaccurate and misleading. RAPT performs LDP to user prompts (thus establishing certifiable privacy via DP), and to recover performance on the noisy data, employs prompt tuning to efficiently fine-tune a server-side model capable of performing inference on the LDP data. This approach is extremely similar to the proposed split-and-denoise framework. The authors should clearly describe RAPT, contrast it to SnP, and benchmark against it as a relevant competitor.

Author response:
**Q3. Difference between RAPT and SnD**

In Section A.1, we illustrated the distinction between RAPT and our framework. Though RAPT employs a reconstruction head to improve the robustness against noises during prompt-tuning stage, *the module only reconstruct the random tokens and doesn’t perform any direct denoise either on the privatized input or the corresponding representations/activations for the private token.* Furthermore, *the reconstruction head is discarded during the inference stage, rendering no denoise mechanism for LLM inference.* In constrast, our work performs explicit local denoise for the output embedding on the user-side, which demonstrates better performance.

Reviewer comment (Experiments):
The paper lacks demonstrated efficacy of the proposed approach against attacks. It is necessary to prove the privacy-preserving capabilities, e.g., by simulating eavesdropping attacks which collect intercepted embedding vectors (during transmission to the server) for malicious actions such as embedding inversion and attribute inference attacks. Li et al. (2023) can be referenced for examples and more details.

Author response:
**Q4. Privacy-preserving capabilities**

Refer to the general response of Q1, we added experiment considering the inference attacks in Section 4.3 and 4.4.3. Such attack could be carried out by an eavesdropper or the curious server.

Reviewer comment (Novelty):
The statement "Split learning is a novel privacy-preserving approach in distributed learning" should be revised to remove "novel", as it is not being introduced in the presented work.

Author response:
**Q6. Inaccurate claims**

We rewrited the claims and highlighted them in blue.

Reviewer comment (Novelty):
The benchmarked method, "TokEmbPriv," requires significantly more discussion during the background and previous work sections.

Author response:
**Q4. Discussion on the limitations of TokEmbPriv**

Thanks for the suggestion. We included the discussions on the limitation of TokEmbPriv and Text2Text in Section 4.4.2. The lack of  denoise mechanism and unbounded noise support in TokenEmbPriv might lead to its poorer performance in downstream task.

Reviewer comment (Experiments):
No evidence to support the practicality of the proposed method. This paper instantiates the denoising model as an L layer transformer-based model that receives the privatised token representations, noise matrix and noisy output computed by the server. Although this denoise model is pre-trained on the server side using public datasets and synthetic noises, each client needs to do the inference of this denoising model locally. Unfortunately, this paper does not empirically evaluate the memory and computational cost of this inference for clients. Therefore, it is not clear if this overhead is any smaller than running the whole LLM on the client side. This evaluation is necessary as one main claim of this paper is that the proposed method introduces only affordable local computations for clients.

Author response:
**Q1. Evidence to support the practicality of the proposed method.** 

We added experiment in Appendix A.9 to compare the computation and memory cost of user-side inference of SnD with the case where the whole model is downloaded. The computation time and memory cost saves by SnD ranges from, respectively, 78% to 96% and 65% to 94%, across the 10 models. The saving is more significant for models of large size, such as GPT2 Xlarge.

Reviewer comment (Evaluation):
No evidence demonstrating the privacy benefits of the proposed method. This paper lacks an empirical evaluation of the privacy leakage of the proposed method. This is particularly important as analytical privacy guarantees chosen by this paper are very loose for example see privacy budgets of 100, 500 and 1000 in the Tables provided in the experiment section.

Author response:
**Q2. Evidence for the privacy benefits & consistence of privacy budget.**
 
We added embedding inversion attack and attribute inference attacks to verify the privacy protection cabilities under different $\eta$s in Section 4.4.3 and 4.3. We found that the three classes of models show different robustness exhibit distinct robustness against inference attacks, and thus the privacy budget varies across the models. The $\eta$ is selected that meets the following two criteria simultaneously: (a) the accuracy for embedding inversion attack is below 0.01, and (b) the accuracy for attribute inference is below 0.53 (the accuracy for random guessing is around 0.5). Noted that we have updated the privacy budget for GPT2 to be 1, 50, 100. We also added evaluation for the performance of extremely low privacy level in Appendix A.11.

Reviewer comment (Experiments):
Shallow discussion of results and not considering SOTA models.

Author response:
**Q3.  Shallow discussion of results.** 

We enhance the discussion in the Section 4. Compared with the vanilla model, Bert, T5, and GPT models yield average model losses of 4.31\%, 4.48\%, and 5.25\%, respectively. In addition, SnD outperforms the baselines by an average of over 10\% benefiting from the denoise model. More details are presented in our paper.

Reviewer comment (Experiments):
Experimental choices including hyperparameters and privacy budgets are not justified/studied and they are not consistent across models. The only statement regarding the choice of the privacy budget that I can see in the paper is the following: "For the three model families, we selected three distinct eta levels for experimentation, given the varying noise tolerance of each model. Specifically, for the Bert models, we set eta to 50, 100, and 500; for the GPT models, the values were 1, 100, and 1000; and for the T5 models, we chose 0.1, 1, and 10."

Author response:
**Q2. Evidence for the privacy benefits & consistence of privacy budget.**
 
We added embedding inversion attack and attribute inference attacks to verify the privacy protection cabilities under different $\eta$s in Section 4.4.3 and 4.3. We found that the three classes of models show different robustness exhibit distinct robustness against inference attacks, and thus the privacy budget varies across the models. The $\eta$ is selected that meets the following two criteria simultaneously: (a) the accuracy for embedding inversion attack is below 0.01, and (b) the accuracy for attribute inference is below 0.53 (the accuracy for random guessing is around 0.5). Noted that we have updated the privacy budget for GPT2 to be 1, 50, 100. We also added evaluation for the performance of extremely low privacy level in Appendix A.11.

Reviewer comment (Writing):
There are many typos:
  1) Missing "." at the end of captions
  2) Consequently, As a result,: As --> as
  3) known as ”embedding as a service”: ”embedding --> ``embedding
  4) this paper represents the pioneering effort in protect user’s: protect --> protecting
  5) Fact Kotonya & Toni (2020),Daily Dialogue: ,Daily --> , Daily

Author response:
**Q4. Typos**

We have corrected the typos and conducted grammer check on the paper.

Reviewer comment (Evaluation):
The reviewer recommends reporting the computational costs and privacy benefits of the proposed method, referring to their first and second concerns in the weakness box.

Author response:
**Q1. Evidence to support the practicality of the proposed method.** 

We added experiment in Appendix A.9 to compare the computation and memory cost of user-side inference of SnD with the case where the whole model is downloaded. The computation time and memory cost saves by SnD ranges from, respectively, 78% to 96% and 65% to 94%, across the 10 models. The saving is more significant for models of large size, such as GPT2 Xlarge.

**Q2. Evidence for the privacy benefits & consistence of privacy budget.**
 
We added embedding inversion attack and attribute inference attacks to verify the privacy protection cabilities under different $\eta$s in Section 4.4.3 and 4.3. We found that the three classes of models show different robustness exhibit distinct robustness against inference attacks, and thus the privacy budget varies across the models. The $\eta$ is selected that meets the following two criteria simultaneously: (a) the accuracy for embedding inversion attack is below 0.01, and (b) the accuracy for attribute inference is below 0.53 (the accuracy for random guessing is around 0.5). Noted that we have updated the privacy budget for GPT2 to be 1, 50, 100. We also added evaluation for the performance of extremely low privacy level in Appendix A.11.