Paper: Privacy Preserving API Fine-tuning for LLMs (ICLR-2024)

Reviewer comment (Theory):
Regarding the privacy-preserving backpropagation. Although $\partial l/\partial h$ is protected, the server still has clean $\partial h/\partial \theta$. This still leaks information about the label. The 2-dimensional TSNE of $\partial h/\partial \theta$ should be similar to $\partial l/\partial \theta$, because multiplying $\partial l/\partial h$ is only a linear operator.

Author response:
For this concern, we agree that $\partial h/\partial \theta$ is similar to $\partial l/\partial\theta$. For this reason, we report $\partial l/\partial\theta$ in Figure 3 ( in addition to $\partial l / \partial h$ before that). There, we observe that $\partial l/\partial\theta$ is also obfuscated through private backpropagation. Please also note that while $\partial l/\partial\theta$ and $\partial h/\partial\theta$ are indeed linearly dependent, there is a  distinct linear dependency **for each sample and at each training step**: as a result, simply trying to recover them with TSNE is unlikely to be fruitful.

If there is any specific experiment we can run to check your hypothesis, we are happy to discuss that.

Reviewer comment (Theory):
Regarding the privacy-preserving forward. If the label can be predicted via only linear transformations of $h_i$, then it means the unaggregated $\{h_i\}$ leaks a lot of information about the label. E.g., the server can simply run some clustering algorithms, which will achieve good performance because $\{h_i\}$ are linearly separable w.r.t. the labels.

Author response:
We split this concern into two statements:

1. If the label can be predicted via only linear transformation, then $h_i$ leaks information about the label

2. If 1. is true, the server can run clustering that will achieve good performance for separating labels

The first statement is true: **if** the label can be predicted via linear transformation, then $h_i$ leaks labels **in the information-theoretic sense**. However, in our case **the label cannot be predicted with a linear transformation**. The algorithm described in Section 3.3 explicitly trains $h_i$ so that no fixed linear transformation can recover labels from them. This idea is formulated and discussed in the first half of page 7.

Furthermore, even the label could be predicted linearly, this would not guarantee that a clustering algorithm would be able to recognize it. To illustrate this point, consider a uniform ball of data points with a radius of 1 that contains points with two labels. All points with label 1 are above a certain (e.g., random) separating hyperplane, and all points with label 0 are below that hyperplane. Therefore, the data is linearly separable. However, no clustering algorithm can distinguish between the correct hyperplane and any other hyperplane without knowing labels. Without labels, the data is a uniform ball that can be split equally well in any direction. 

In preliminary experiments, we found that 2-cluster k-means has near-random performance for both P$^3$EFT and baseline methods. We will add these results in the revised version of the paper.

Reviewer comment (Theory):
No clear security model.

Author response:
We firmly believe that we have provided a detailed portrayal of the attacker model in section 3.1 (i.e., “honest, but curious” model with details); however, we are open to improving it further. If there are any specific aspects that require clarification, kindly provide further details so that we can address them in the revised version.

Reviewer comment (Writing):
In Section 3, the authors claimed that "formulate a protocol for two-party" in the 1st sentence. In the abstract, "the client ..., and the server ...." A client and a server constitute "two-party" already. However, Equation 2 in Section 3.2 contains "two identical independent servers that offer backprop API." The number of parties is not corresponding.

Author response:
Thank you for pointing out this oversight! We will update the submission to clarify that we consider two setups: one with a single server and the other with multiple servers. We will also specify which setup we are addressing in each specific part of the article.

Reviewer comment (Novelty):
As for the formulation, it looks like an application of the $n$-out-of-$n$ secret-sharing scheme. In particular, Equation 2 is essentially similar to Part 2 in [REF1]. Additionally, secret-shared backpropagation has already been solved in the early work [REF2].

Author response:
We thank the reviewer for drawing our attention to the technique of secret sharing, and we agree that it has similarities with Equation 2. However, we believe that there is still a significant difference between these methods. Please see the upcoming General Response for details.

Reviewer comment (Theory):
Could the authors explicitly formulate the security model?

Author response:
We firmly believe that we have provided a detailed portrayal of the attacker model in section 3.1 (i.e., “honest, but curious” model with details); however, we are open to improving it further. If there are any specific aspects that require clarification, kindly provide further details so that we can address them in the revised version.

Reviewer comment (Novelty):
Could the authors explain the difference between the proposed formulation and secret sharing?

Author response:
We thank the reviewer for drawing our attention to the technique of secret sharing, and we agree that it has similarities with Equation 2. However, we believe that there is still a significant difference between these methods. Please see the upcoming General Response for details.

Reviewer comment (Novelty):
The first major concern is the practical significance of the problem setting and its connection to some related topics, such as vertical federation learning. There is a need to examine whether the connection is contrived for predictive tasks and to discuss alternative approaches like local differential privacy on labels or noisy label learning, highlighting the advantages of the proposed method over these existing strategies.

Author response:
Thank you for pointing out this oversight. We fully agree that the paper will benefit from including a discussion and comparison of our method with techniques based on noisy labels and differential privacy.

Previous work has shown that gradients with respect to activations are a very vulnerable aspect of split learning in terms of privacy. Techniques such as noisy labels and differential privacy are used to overcome this very problem [1,2]. However, these techniques rely on using noisy gradients instead of the true gradients, which can lead to a decrease in final quality (see e.g. non-private baselines in [3], [4]).

In contrast to previous work, parameter-efficient fine-tuning does not require computation of gradients with respect to **all parameters**. We only need to update a very small **subset of parameters**, therefore it is needed for the client to know true gradients just for that subset. This allows us to compute true gradients in a completely private manner without decreasing final quality.

Reviewer comment (Novelty):
The second major concern is the novelty of the methodology. With access to the full features of the target domain, the work is related to source-free domain adaptation. It would be beneficial to clarify why the necessity of using labels justifies the approach, given that the applied loss involving labels should be more informative than unsupervised domain adaptation.

Author response:
The majority of language models have been pretrained either on a masked word prediction or causal language modeling objective. As a result, their pretrained activations are not suitable for classification tasks such as sentiment classification or predicting logical relations between statements. We can provide one of our experiments as evidence for this claim: as a baseline, we trained only the classification head without adapters, and the results were significantly worse (Figure 4). Thus, we believe that unsupervised learning may not be very effective for this task. If you know of any results that show otherwise, we would greatly appreciate it if you could point us to them so that we can compare our method with them.

Reviewer comment (Novelty):
The discussion of fine-tuning APIs in paragraphs 3–4 omits recent relevant works, especially in the privacy-preserving context. Suggested references include:  
[1] Earning Extra Performance from Restrictive Feedbacks, 2023  
[2] Offsite-Tuning: Transfer Learning without Full Model, 2023

Author response:
We appreciate your bringing these articles to our attention. We acknowledge that both of these methods are potentially much less vulnerable to potential attacks than our method, but they do have some drawbacks which we will discuss below.

From our perspective, the first article presents a promising approach for our problem setting, but there may be issues with its application due to the large number of forward and backward passes required to obtain gradient estimates, which increases compute cost.

The second article also proposes an elegant method that is fully private for the client by design. This method still requires running a distilled model on the client side. However, the authors explicitly state in their limitations that they used a distilled model size that was not smaller than one-third of the size of the original model. This is still incredibly challenging for clients due to the large memory and computational resources required for training. In contrast, our method only requires the client to perform forward and backward passes through the head layers, which poses no problem. Therefore, we believe that our method is potentially more scalable.

Reviewer comment (Novelty):
The gradient privacy-preserving technique appears to be based on zero-order optimization with random weights for activation functioning like a locally maintained code book. There is a need to explain the differences or novelties compared to previous work regarding these two techniques.

Author response:
To clarify, we do not use zero-order optimization methods. In our approach, we use gradient-based methods (specifically, we optimize the weights using Adam). The procedure outlined in Section 3.2 is designed to give the client opportunity to compute true gradients of the loss with respect to the adapter weights in a way that prevents the server from reconstructing these true gradients (server sees only noisy gradients). We apologize if our explanation was unclear and we will strive to improve it in a revised version.

Regarding the reference to "the random weights for activation", as we understand it, you are referring to the random weights that are multiplied with the activations. These weights remain frozen throughout the entire procedure. We use these weights to ensure that different activation components from different copies of the model (which only differ in adapters) contribute differently to the overall sum, which is used as the input for the head. For example, the first activation component from the first copy contributes the most to the first component of the sum, the second activation component from the second copy contributes the most to the second component of the sum, and the third component of the sum receives an equal contribution from the third components of both the first and second copies, and so on. Intuitively, this allows the distribution of activations for each individual model copy to be more entangled, making it harder to predict labels based on activations alone. If this idea has been used in prior works, please let us know.

Reviewer comment (Evaluation):
If an adversary knows how \( z \) is sampled, \( g_h \) could still be exposed via sum even if the norm of \( z \) is large. Although \( n \) parallel calls have been suggested as a workaround, it would be better if the trade-off between cost and benefits were provided.

Author response:
This line of reasoning would work for the simplified two-party equation (2), if we decide to apply that formula to a single server instead. Please note that this formula is meant as a simple demonstration for two independent servers, each of which only knows one noisy gradient. We further extend this formula to multiple servers in the two paragraphs following (2). “N parallel API calls” are also not necessary for our method and were mentioned more for the purpose of demonstrating that we simply need a set of vectors such that the true gradients belong to their linear span. In the future, we will strive to clearly separate the overall narrative and the final result.

Reviewer comment (Presentation):
There are presentation issues — specifically, in the last paragraph of page 3, \( h \) is not well presented. The figure (Fig. 1) uses \( h’ \) while the main text uses \( h^* \), leading to inconsistency.

Author response:
Thank you very much for your feedback, we will definitely address this discrepancy.

Reviewer comment (Writing):
The description of the proposed method is confusing and missing lots of information. Figure 2, which is supposed to be a visualisation of the proposed framework, makes it even more confusing by introducing new variables that were never used in the description, leading to difficulties in understanding and guessing the missing information.

Author response:
The reviewer asks how LoRA adapters are initialized and how they are used in computations. Overall, we define the multi-party fine-tuning procedure in Section 3.1, which follows standard LoRA fine-tuning. Our algorithm is an extension of this procedure.

Before the training, adapter weights and head weights are randomly initialized on the client side. 
 
During the training on each step:
1. adapters first transferred along with the inputs from client to server
2. server runs forward pass with given adapters
3. server computes activations from the given inputs and adapters and returns them to the client 
4. client computes loss and **real** gradients w.r.t. to the received activations 
5. client transfers **noisy** gradients (produced with method described in S3.2) to the server 
6. server runs backward passes and obtains **noisy** gradients w.r.t to adapter weights
7. server transfers **noisy** gradients w.r.t. to adapter weights to the client
8. client computes **real** gradients w.r.t. adapter weights
9. client executes a step of an arbitrary optimization algorithm.

Overall, we agree that the paper can be improved by adding a dedicated description of the algorithm, and we will improve it in the nearest revision.

Reviewer comment (Reproducibility):
In the proposed method, a client has local adapters and initializes them locally, but it is unclear how this initialization is done. Two possible scenarios are: 1) initialization is done randomly; or 2) initialization is done by copying the weights of adapters owned by the server. Scenario 2 does not make sense because the paper discusses that servers do not want to send their model to clients. Scenario 1 does not make sense either, as in step 5 clients use the gradients with respect to the server adapter parameters.

Author response:
The reviewer asks how LoRA adapters are initialized and how they are used in computations. Overall, we define the multi-party fine-tuning procedure in Section 3.1, which follows standard LoRA fine-tuning. Our algorithm is an extension of this procedure.

Before the training, adapter weights and head weights are randomly initialized on the client side. 
 
During the training on each step:
1. adapters first transferred along with the inputs from client to server
2. server runs forward pass with given adapters
3. server computes activations from the given inputs and adapters and returns them to the client 
4. client computes loss and **real** gradients w.r.t. to the received activations 
5. client transfers **noisy** gradients (produced with method described in S3.2) to the server 
6. server runs backward passes and obtains **noisy** gradients w.r.t to adapter weights
7. server transfers **noisy** gradients w.r.t. to adapter weights to the client
8. client computes **real** gradients w.r.t. adapter weights
9. client executes a step of an arbitrary optimization algorithm.

Overall, we agree that the paper can be improved by adding a dedicated description of the algorithm, and we will improve it in the nearest revision.

Reviewer comment (Reproducibility):
In the proposed method, when a client passes computed features to the local “head” to compute a task-specific loss function, it is unclear what this task-specific loss function is.

Author response:
For SST2 and MRPC tasks use categorical cross-entropy. In general, this is the loss function used in the standard fine-tuning procedure for downstream tasks, as defined in the supplementary code of the GLUE benchmark paper.

Reviewer comment (Reproducibility):
In the proposed method, when a client updates both local adapter parameters and local head parameters, it is unclear how and which adapter parameters are updated, relating back to the concerns raised in point 2.

Author response:
Client updates all adapter parameters after each backpropagation step, which is the standard way of training LoRA adapters defined in Hu et al., (2021) [1]

Reviewer comment (Theory):
The paper claims "privacy guarantees" by stating "designing a two-party fine-tuning protocol that performs standard parameter-efficient fine-tuning with privacy guarantees". However, no privacy guarantees are actually provided; the privacy claim is ad-hoc and based solely on increasing the number of servers under the assumption they do not collude and that they have the same model.

Author response:
While we discuss the privacy of two-party fine-tuning scenario at the end of Section 3.2, we agree that the paper would benefit from making the claims more specific. We will rewrite them in the updated version.

We would like to clarify that the second assumption (same model) is relatively common.
In NLP, there are popular repositories of public models that are used for fine-tuning, e.g. [2] [3]. Most fine-tuning runs reported in NLP research, including most of the runs in the LoRA papers (see [1, 4]) use open-access models. When fine-tuning in this setup, every server can download openly available model weights prior to training, and the client benefits from the shared compute power of the servers. Thus, if the first assumption holds (there are non-colluding servers), then these servers can get the same model.

Reviewer comment (Writing):
The paper is not self-contained; for example, a clear description of LoRA, which is the main building block of the proposed framework, is missing.

Author response:
We agree and will include that description in the nearest revision.

Reviewer comment (Novelty):
It is unclear what the novelty of the proposed privacy-preserving backpropagation is compared to secret sharing in two-party computation, which has been heavily studied in the literature.

Author response:
We appreciate the reviewer's input regarding the technique of secret sharing and agree that it bears some similarities to our method of privacy-preserving backpropagation. However, we maintain that there are important distinctions between us and secret sharing from multi-party  computation. Since another reviewer asked for the same comparison, we will discuss this in more detail in the upcoming General Response.