Paper: Differentially Private Bias-Term Fine-tuning of Foundation Models (ICLR-2024)

Reviewer comment (Writing):
Why is $C_i$ given as an input to Algorithm 1? I wonder if you mean the clipping function $C$.

Author response:
2. Why is Ci given as an input to Algorithm 1? I wonder if you mean the clipping function C.

**Response** We meant Ci is initialized as 1, so that if one ignores all red lines in Algorithms 1, one can recover the non-DP SGD (which still uses Ci in line 8 because we want to write the DP/non-DP SGD in a unified way).

Reviewer comment (Novelty):
I think this work is useful, and should be shared at a workshop maybe, but unless my concerns about novelty are satisfied I'm not sure it merits publishing at a top-tier conference.

Author response:
We thank the reviewer for the comment. Here is a point-to-point response.

1. Contrary to the claims in the paragraph "novelty", it seems to me there is not a huge creative leap in putting together BitFit and DP-SGD as is done here. What is the "substantial algorithmic innovation"? What is the naive way of combining these ideas that your algorithm improves upon?

**Response** We agree that BiTFiT and DP-SGD are both existing methods. However, our main contributions also include the complexity analysis and engineering implementation. 1) Complexity analysis of DP parameter-efficient fine-tuning (PEFT): DP-BiTFiT in main text, DP-LoRA and so on in appendix C. This is a missing piece in previous DP PEFT literature and significantly helpful in determining the benefit of applying different PEFT methods. E.g. we rigorously show the complexity saving is 50%, and the benefit of BiTFiT on long-context language tasks or high-resolution image tasks, i.e. the computation overhead of DP-BiTFiT in the last column of Table 2 is independent of T. This analysis is also missing in the non-DP BiTFiT. Without this analysis, our understanding of efficient fine-tuning methods will be at most empirical, and certain special properties of DP-BiTFiT (like activation-free optimization and feature-dimension-free overhead) will not be revealed. 2) Engineering effort: at the time of writing this paper, none of existing codebases including GhostClip and Opacus remove the forward hooks, because no analysis has established that only BiTFiT, not LoRA/Adapter/Compactor or full fine-tuning, can be activation-free. This is a simple yet effective trick to save memory in large model training. Our code will be open source to benefit the whole DP community.

In summary, the algorithmic innovation that improves the naive way of combining is visualized in Figure 2: we significantly simplify the computation graph from left and upper to the DP-BiTFiT (bottom right), with complexity guarantee.

We hope the reviewer can reconsider the score given that the complexity analysis is an important component of this work. Even though the methods are not new, a theoretical analysis could be insightful and novel, like those convergence analysis to the SGD algorithm.

Reviewer comment (Novelty):
See question about novelty above.

Author response:
We thank the reviewer for the comment. Here is a point-to-point response.

1. Contrary to the claims in the paragraph "novelty", it seems to me there is not a huge creative leap in putting together BitFit and DP-SGD as is done here. What is the "substantial algorithmic innovation"? What is the naive way of combining these ideas that your algorithm improves upon?

**Response** We agree that BiTFiT and DP-SGD are both existing methods. However, our main contributions also include the complexity analysis and engineering implementation. 1) Complexity analysis of DP parameter-efficient fine-tuning (PEFT): DP-BiTFiT in main text, DP-LoRA and so on in appendix C. This is a missing piece in previous DP PEFT literature and significantly helpful in determining the benefit of applying different PEFT methods. E.g. we rigorously show the complexity saving is 50%, and the benefit of BiTFiT on long-context language tasks or high-resolution image tasks, i.e. the computation overhead of DP-BiTFiT in the last column of Table 2 is independent of T. This analysis is also missing in the non-DP BiTFiT. Without this analysis, our understanding of efficient fine-tuning methods will be at most empirical, and certain special properties of DP-BiTFiT (like activation-free optimization and feature-dimension-free overhead) will not be revealed. 2) Engineering effort: at the time of writing this paper, none of existing codebases including GhostClip and Opacus remove the forward hooks, because no analysis has established that only BiTFiT, not LoRA/Adapter/Compactor or full fine-tuning, can be activation-free. This is a simple yet effective trick to save memory in large model training. Our code will be open source to benefit the whole DP community.

In summary, the algorithmic innovation that improves the naive way of combining is visualized in Figure 2: we significantly simplify the computation graph from left and upper to the DP-BiTFiT (bottom right), with complexity guarantee.

Reviewer comment (Theory):
The paper can provide more intuitions about why fine-tuning the bias term can be effective enough with pre-trained models. As a comparison, LoRA is supported by a strong intuition that fine-tuning updates can be considered low-ranks.

Author response:
We agree more understanding of why BiTFiT works is desirable. This has been difficult even for the original paper in the non-DP regime. Looking backwards from the effectiveness of BiTFiT, we may have a intuition that the fine-tuning updates are generally in low-dimension, as is also the case for LoRA (despite the rank, the number of trainable parameters is small). We may test this hypothesis by randomly selecting a small number of parameters to update at each layer. However, even if any small number of parameters would work, BiTFiT has its unique advantage of being activation-free.

Reviewer comment (Experiments):
The limitation of the proposed algorithm may deserve some more discussion. While remark 4.2 mentioned that the method may be less effective for models with convolutional layers without bias terms, it is not clear whether all modules' bias terms (attention, fully connected, convolutional, etc.) have the same effect or some of them are more important than others. Besides, as some recent LLM architecture may not include biases in some layers (PaLM [0]) to increase training stability, it is unclear how general the proposed method can be in practice.

Author response:
We haven't looked into a finer-grained analysis of bias terms. This could be an interesting future work had more people become interested in our method. We agree that not all models are suitable for DP-BiTFiT, but a long list of models in our Table 1 should be. Additionally, we study DP-BiTFiT as a standalone method for fair comparison to other standalone methods. In practice, one can always combine two parameter-efficient methods, such as DP-BiTFiT and last-layer training in Section 4.3. Specifically, turning on DP-BiTFiT only needs one line of code: [param.requires_grad_(False) for name, param in model.named_parameters() if 'bias' in name].

Reviewer comment (Experiments):
Is there any potential limitations of the proposed algorithm in terms of the fine-tuning tasks? Can we expect similar improvement if we fine-tune for more complicated tasks, for example fine-tune LLaMA for GSM8k or for MMLU tasks?

Author response:
I am experimenting with LLaMA and do observe similar behavior.

Reviewer comment (Theory):
Is there any requirement or conclusion about how well-pre-trained a model should be or how complex a model should be so that the proposed method can be effective? An inappropriate extreme example might be that one should not expect tuning the bias in a linear model to fit arbitrarily shifted distribution.

Author response:
Our experiments in Table 3,4,5 show that DP-BiTFiT tends to be more comparable to full fine-tuning when model sizes are larger. This also implies that a better pre-trained model can benefit from DP-BiTFiT more: tuning only bias terms from scratch is not going to work.

Reviewer comment (Theory):
Maybe related to the above question, is there intuition to support the proposed method?

Author response:
We agree more understanding of why BiTFiT works is desirable. This has been difficult even for the original paper in the non-DP regime. Looking backwards from the effectiveness of BiTFiT, we may have a intuition that the fine-tuning updates are generally in low-dimension, as is also the case for LoRA (despite the rank, the number of trainable parameters is small). We may test this hypothesis by randomly selecting a small number of parameters to update at each layer. However, even if any small number of parameters would work, BiTFiT has its unique advantage of being activation-free.

Reviewer comment (Theory):
Are the bias terms in different modules (attention, fully connected, convolutional, etc.) have the same effects?

Author response:
We haven't looked into a finer-grained analysis of bias terms. This could be an interesting future work had more people become interested in our method. We agree that not all models are suitable for DP-BiTFiT, but a long list of models in our Table 1 should be. Additionally, we study DP-BiTFiT as a standalone method for fair comparison to other standalone methods. In practice, one can always combine two parameter-efficient methods, such as DP-BiTFiT and last-layer training in Section 4.3. Specifically, turning on DP-BiTFiT only needs one line of code: [param.requires_grad_(False) for name, param in model.named_parameters() if 'bias' in name].

Reviewer comment (Novelty):
My main concern with this paper is the novelty of this method. It appears that this method directly adapts the existing method BitFit to the DP-SGD.

Author response:
1. Regarding the novelty, we claim at least 2 types of new contributions: 1) Complexity analysis of DP parameter-efficient fine-tuning (PEFT): DP-BiTFiT in main text, DP-LoRA and so on in appendix C. This is a missing piece in previous DP PEFT literature and significantly helpful in determining the benefit of applying different PEFT methods. E.g. we rigorously show the complexity saving is 50%, and the benefit of BiTFiT on long-context language tasks or high-resolution image tasks. This analysis is also missing in the non-DP BiTFiT. 2) Engineering effort: at the time of writing this paper, none of existing codebases including GhostClip and Opacus remove the forward hooks, because no analysis has encouraged to do so. Note our code will be open source to benefit the whole DP community.

Reviewer comment (Evaluation):
The performance of DP-BitFit is limited in some scenarios and requires the additional design for two phases, which makes the results of DP-BitFit less significant.

Author response:
We thank the reviewer for the comment. We would like to clarify a misunderstanding in the "Weakness". **We do not require "the additional design for two phases."** This was an alternative method presented in older version of this paper, hence it should not be treated as a main part of this work on which the review is based.

Reviewer comment (Presentation):
Presentation issues: The table number and title should appear before the table (Table 7-16). It would be better to be more careful with the use \citep and \citet.

Author response:
Dear reviewer,

We have addressed your comments point-by-point, trying our best to clarify any confusion. We have changed all \cite to \citep in this revision and corrected the table captions.

Specifically, **we do not need two-phase training in this paper** (it's merely an alternative in the appendix, hence should not be counted towards *Weakness*). We sincerely hope you would reconsider your score if the score is based on the mis-understanding. 

We would love to elaborate the significance of our complexity analysis and in the engineering efforts: removing the forward hooks is **never implemented in previous codebases** because only DP-BiTFiT is activation-free, not DP-LoRA or any other methods. Being activation-free is an important characteristics that make DP-BiTFiT uniquely suitable for high dimension (e.g. long context or high resolution image) tasks, which was not discovered in existing DP parameter-efficient fine-tuning literature which lacks the analysis on the computation graph and complexity.

Your evaluation is important to us and we look forward to further discussion if needed.

Reviewer comment (Experiments):
In Table 4, it is somewhat weird to me that the perplexity of non-private results is worse than private results in several cases. Also, Table 4 and Table 13 show that DP-BitFiT on GPT2-large is better than DP-BiTFiT is better than DP-SGD (full). I wonder if this is due to the dimensionality issue in DP-SGD, or sub-optimal hyperparameters for DP-SGD (full), or if it is because the comparison is not averaged across several runs for statistical significance.

Author response:
2. Regarding Table 4, we believe the situation is natural: the perplexity is only one of many performance measures other than BLEU. In fact, it is not unusual sometimes the cross-entropy loss (i.e. perplexity) is at odds with the performance measure, e.g. in \url{https://arxiv.org/pdf/2106.07830.pdf} Figure 8. We are not sure why DP-BiTFiT seems to outperform DP-full on GPT2-large, though we do consistently observe that the gap is smaller as we scale up the model sizes. We have tuned the hyperparameters of DP-full carefully so our hypothesis is the dimensional issue.

Reviewer comment (Experiments):
Comparison with DP parameter efficient methods for efficiency. The authors provide the theoretical comparison of DP-BitFit with other DP parameter-efficient methods. Figure 3 and Figure 4 compared the memory, speed, maximum throughput and batch size for DP-BitFit and full parameter updating methods. It would be better to also include the comparison to other DP parameter-efficient methods such as DP-LoRA.

Author response:
3. We haven't compared to DP-LoRA in all experiments. We are happy to do so in the camera-ready revision if the reviewer can consider raising the score.

Reviewer comment (Evaluation):
The empirical evaluation shows performance drop of DP-BiTFiT compared with other parameter-efficient fine-tuning techniques.

Author response:
We agree that there is sometimes a minor drop of DP-BiTFiT, but overall DP-BiTFiT is comparable when applied alone. For fair comparison, we study DP-BiTFiT as a standalone method to compare with other PEFT. A promising application is actually to apply DP-BiTFiT together with LoRA or last-layer tuning, as we have shown in Section 4.3. Also, a small performance drop is acceptable in the cases when we emphasize the memory efficiency (since DP-BiTFiT is activation-free) and implementability (only need one line of code $$[param.requires\_grad\_(False) for name, param in model.named\_parameters() if 'bias' in name]}$$).

Reviewer comment (Theory):
In the Contribution 4, "DP-BiTFiT is a unique algorithm in that the computation overhead is independent of the feature dimension T" where T is the sequence length. The author should be more specific for this claim.

Author response:
We have explained the computation overhead of DP-BiTFiT in the last column of Table 2, where both the time and the space complexity is O(Bp), independent of T. This is also empirically verified in Section 3.3: we quote "DP-BiTFiT is amazingly scalable since its *computational overhead* is negligible and independent of T (though the *total complexity*, mainly due to forward and output gradient, is still linear in T)."

Reviewer comment (Experiments):
This work lacks a clear empirical exploration of the difference between the activation and the model memory. Right now, it is clear that the algorithm uses much less memory and compute. However, it is unclear how the total memory in figure 1 is split between storing the model, materializing activations, etc.

Author response:
We thank the reviewer for the comment and appreciating our work. Although in Figure 1 we didn't precisely split the memory between storing the model, materializing activations, etc., this information can be inferred (approximately in the middle sub-plot): given that DP compactor/LoRA/Adapter have similar (though larger) number of trainable parameters to DP-BiTFiT, we claim most of the memory difference is indeed in the activations.

Reviewer comment (Presentation):
Figure 4 is extremely confusing. How can DP-BITFIT be both on the x-axis and in the legend? What does maximum through of algorithm mean?

Author response:
We would like to clarify Figure 4. The x-label and legend are both correct. The DP-BiTFiT is serving as a reference line diagonally. Any method such as "non-DP full" can be comparable to DP-BiTFiT by looking at the slope (less steeper means less scalable). Each dot represent the actual throughput or memory (from left to right is large to small models). The throughput means number of examples processed per second by maximizing the batch size, as a measure of training speed. We will clarify it in the camera-ready revision.

Reviewer comment (Evaluation):
In table 5, how were the results of Bu et al. obtained? I could not find any 98.9% performance (Cifar-10 result) in their work.

Author response:
We are grateful that the reviewer spot this typo in Table 5: the vit-large result is Ours, obtained through full fine-tuning. The horizontal line should be one row higher.