Paper: LMO-DP: Accurately Fine-Tuning Language Models with Stronger Differential Privacy (ICLR-2024)

Reviewer comment (Novelty):
There are many recent works on tight privacy accounting for differential privacy such as [1] and [2]. The authors should discuss whether they can be applied to the proposed algorithm.

Author response:
Thanks for the suggestions (we will add a discussion and cite them). Their theories can also be applied to our proposed algorithm. We designed our LMO-DP method based on randomization which tries to find a smaller noise while meeting the same level of privacy guarantee. The contributions are orthogonal to these two works. If we integrate LMO-DP with these two works, we need to revise the constraints in Equation (2) and we can replace the RDP by those new accountants/methods with tighter bounds. New efficient and accurate solvers for finding the (sub)optimal noise would be desirable as well.

Reviewer comment (Theory):
In section 3.3, the global optimization seems to be formulated for a fixed pair of neighboring datasets $D$ and $D'$. However, differential privacy requires all pairs of $D$ and $D'$, so to me, there should be a max over all possible $D, D'$ somewhere in the formulation. I hope the authors can explain where the worst case over all neighboring datasets is considered in this formulation.

Author response:
Thanks a lot for this good catch. Yes, our design involves all pairs of D and D’ (worst case protection). It was a typo in the constraints in Equation (2). We will add a supremum over choice of D and D’ in Equation (2) to represent all the neighboring datasets (and worst case). Our proof (Theorem 4.1 in Appendix B.1) and implementation (in Algorithm 3 - LMO-DP in the original submission and the code) are indeed over the worst case, i.e., $\sup_{\forall D,D'\in\mathcal{D}}$.

Reviewer comment (Writing):
Many in-line citations do not have parenthesis and break the flow of the paper. For example: Section 2, DP-SGD Abadi et al. (2016). The reference should have a parenthesis around it.

Author response:
Thanks. We will fix them.

Reviewer comment (Experiments):
In equation (2), is there a particular reason for the range of $\alpha$ to be 2 to 129? Is it possible that the best parameter is achieved outside of this range for some datasets and problems?

Author response:
The range of $\alpha$ was selected from 2 to 32 in most existing works/codes. However, when computing the R\'enyi privacy of the LMO mechanism within this $\alpha$ range, null values frequently arise. Expanding the upper bound of $\alpha$ can facilitate the computation of R\'enyi privacy for the LMO-DP mechanism. Nevertheless, opting for larger $\alpha$ values may result in moments too tiny for effective processing of the floating point numbers. Consequently, there exists a tradeoff between the choice of $\alpha$ and potential privacy leakage. For our specific tasks, we have empirically settled on 128 or 129 as a compromise. For other tasks, the selection of the $\alpha$ range should be based on privacy requirements and practical constraints (e.g., processing the floating point numbers).

Reviewer comment (Evaluation):
It is surprising to me that the noise level for $\varepsilon =2,3$ is slightly larger than $\varepsilon=0.2$. Could the authors explain this phenomenon?

Author response:
Thanks for this question. The noise levels in our Figure 2 are different for $\epsilon=0.2$ and $\epsilon=2$. They look close to each other since different scales for y axes were applied in the subfigures. To show the dynamic of the LMO-DP mechanism, we replace Figure 2 with the statistical figure of sampled noise in the revised version (please see the supplementary document and the submission will be updated asap).

Reviewer comment (Reproducibility):
Algorithm 3 seems to be important, adding it to the paper would be nice.

Author response:
We will move Algorithm 3 to the main text in the new version. Thanks for the suggestion.

Reviewer comment (Experiments):
How can the proposed method be extended to vision tasks? or vision generative models?

Author response:
Thanks for this question and the ideas for extension to other domains. It is very interesting to investigate new DP solutions (with better privacy/utility trade offs) for those tasks. For the traditional vision tasks, there may exist a trivial way to extend LMO-DP with the support of DP-SGD (though the LM Geometry should be revised since it is mainly designed for NLP tasks). More specifically, our proposed method can be extended to these vision tasks since the key component of our method is to search the less noise under the privacy constraints. Then, we still can have a similar framework to search less noise. However, for different domains (e.g., vision task), we may need to change the combination of mixture distributions and the loss functions and so on.

For the vision generative models, if DP works for their training/fine-tuning, the accuracy can be further improved by adapting this method with proper extensions. To our best knowledge, in the case of NAR generative models, we may also need to design a new DP framework (and then optimize the noise) for the training as it solves a BERT like optimization problem during training.

Reviewer comment (Novelty):
For instance NAR generative models solve a bert like optimization problem during training, is there a trivial way to extend this method?

Author response:
Thanks for this question and the ideas for extension to other domains. It is very interesting to investigate new DP solutions (with better privacy/utility trade offs) for those tasks. For the traditional vision tasks, there may exist a trivial way to extend LMO-DP with the support of DP-SGD (though the LM Geometry should be revised since it is mainly designed for NLP tasks). More specifically, our proposed method can be extended to these vision tasks since the key component of our method is to search the less noise under the privacy constraints. Then, we still can have a similar framework to search less noise. However, for different domains (e.g., vision task), we may need to change the combination of mixture distributions and the loss functions and so on.

For the vision generative models, if DP works for their training/fine-tuning, the accuracy can be further improved by adapting this method with proper extensions. To our best knowledge, in the case of NAR generative models, we may also need to design a new DP framework (and then optimize the noise) for the training as it solves a BERT like optimization problem during training.

Reviewer comment (Writing):
The main weakness of this paper is the presentation. The manuscript reads like a rushed submission that requires more careful proofreading.

Author response:
Thanks for these comments. We have fixed the typos/inconsistencies and added more clarifications. The new submission/manuscript will be updated soon.

Reviewer comment (Writing):
Can you explain what you mean by LM geometry in more detail? I found it hard to follow the LM Geometry section. What are the specific characteristics? Is $\mathbb{P}_{i/o}$ simply the conditional probability $\mathbb{P}[o|i]$? This seems to be a conventional language model. I'm failing to understand what aspect of this definition is geometric.

Author response:
Thanks for this question. LM Geometry refers to the geometric structure underlying the Language Model (LM). Let's delve into its components:

- Parameter Space $\Omega$: this represents the configuration space of the LM. For instance, in a word embedding function, $\Omega$ could include parameters defining word vectors.

- Mapping Functions $\mathcal{F}$: these functions map model configurations within $\Omega$ to specific characteristics. For example, in a word embedding function $v(w)$, $\mathcal{F}$ will map words $w$ to vectors in $\mathbb{R}^d$. Thus, $\mathcal{F}$ captures the transformation from model configurations to observable features.

- Probability Distribution $\mathbb{P}_{i/o}$: this probability distribution characterizes the probabilistic transition from an input token $i$ to an output word $o$. In a standard language model, it indeed aligns with the conditional probability $P[o|i]$. However, the emphasis on "geometry" lies in the broader interpretation. It encapsulates the entire distribution of transitions, offering a geometric view of how the model navigates from one token to another probabilistically.

Specific Characteristics:

The specific characteristics captured by $\mathcal{F}$ might include the spatial arrangement of word vectors or the distribution of model weights. These characteristics contribute to the overall geometry, allowing us to analyze and quantify how the LM operates probabilistically.

We will add more clarifications to the new version and update it asap.

Reviewer comment (Presentation):
Can you explain figure 1 in more detail and how it shows that the mixture of distributions covers the whole space?

Author response:
In probability theory, to our best knowledge, there is no universally accepted measure for quantifying the comprehensiveness of a subset of probability functions. Therefore, Algorithm 2 (Quantification of LMO Search Space) is employed to assess the comprehensiveness of this subset in comparison to a universally simulated space, introducing a novel quantification test. We utilize three distance metrics (both probabilistic and deterministic): KL divergence, $\ell_2$ and EMD metrics to measure the distance between these two spaces. Our results demonstrate that, for any given noise 'n', there exists an LMO noise that remains close to 'n'. Please note for the EMD metric, we experience a small divergence (scaled by $10^{-3}$) when the domain of noise increases which necessitates a smaller quantization rate to achieve even better results.

Reviewer comment (Novelty):
It would be interesting to learn more about the mixture of noise distributions. What type of noise is the main contribution?

Author response:
Thanks for the question. It is a Laplace-based two-fold noise (the first-fold is the Laplace distribution while the second-fold is a mixture distribution). Specifically, the "inverse" of "scale parameter" of the Laplace distribution (1/b) is subject to the linear combination of Gamma, exponential and uniform distributions. In addition, we have added an ablation study to exhibit the contribution of noises (please see the supplementary document).

Reviewer comment (Writing):
The authors often refer to the privacy curve $\delta(\varepsilon)$ as the PLRV; while they are equivalent, they're not the same.

Author response:
We will fix it. Thanks.

Reviewer comment (Writing):
As stated above, the main weakness of the paper is the overall presentation. I believe if the authors could improve the write-up, the value to the community would be significantly higher.

Author response:
Thanks for the suggestion and encouraging feedback. We are working on improving the presentation/clarity of the work. The new version will be updated soon.

Reviewer comment (Theory):
The results rely heavily on the supplemental material, with no proofs or high-level descriptions of proofs provided in the main-text. This makes it difficult to follow the line of thought and verify if the high-level approach is sound.

Author response:
Thanks for the suggestion. We have restructured the core methodology section (to be updated soon).

Reviewer comment (Presentation):
One of the key algorithms is Algorithm 3 which is required to understand section 4.3 but does not appear in the main-text.

Author response:
We will move it to the main text. Thanks.

Reviewer comment (Experiments):
This work compares several orthogonal methods to DP on different (often unrelated) axes in a rather confusing way. At the heart of this work is minimizing the amount of noise needed by DP. Yet, this work lists and compares with largely orthogonal work that explores memory reduction (ghost clipping) or parameter efficiency techniques (which improve memory/noise scale through the number of parameters) but can be used in conjunction with these methods. Why does Table 1 give low memory to this work? This seems like a false claim considering this work does not optimize memory but instead the noise standard deviation. Also, this work claims faster convergence but does not provide any theoretical guarantee to convergence, only some empirical exploration.

Author response:
Thanks for this comment.  Yes, our LMO-DP framework is orthogonal to several existing works (we focus on minimizing the noise). For this reason, in most of our experiments, we primarily compare it with the Gaussian mechanism for fine-tuning language models on accuracy. Finally, we added a small group of experiments to compare LMO-DP with SOTA methods. Notice that, although those memory reduction (e.g., Ghost Clipping) and parameter efficiency methods improve the performance from different aspects, they are also commonly considered as SOTA methods. We can also consider such comparisons as showing the improvements on accuracy for such orthogonal methods. 

> Low memory in Table 1

Also, as clarified in Section 1, given such orthogonal contributions, our LMO-DP framework can boost the performance of those methods (e.g., Ghost Clipping) and inherit the benefits of other works (e.g., memory reduction). Thus, in our LMO-DP implementation (shown in the source code), we have two modes, including the one integrated with the Ghost Clipping with low memory and the same accuracy (please see line 1 in lmo-dp/experiments/private-transformers/commands_task1.sh, we can choose ghost or default for “--clipping_mode”). That is why we also give low memory to this work. 

> Faster Convergence

Yes, our faster convergence is based on empirical observation (we will add a note for empirical observation or remove that column in Table 1). The possible reason for such empirically faster convergence lies in the dynamics of LMO-DP noise. By introducing a significantly smaller perturbation, as illustrated in Fig 2, the incorporation of LMO-DP noise in DPSGD-based technologies intriguingly leads to faster convergence.
 
Specifically, as verified by our implementation results, LMO-DP noise for strong privacy guarantees (e.g., privacy budget $\epsilon < 0.2$) yields outcomes equivalent to those obtained with Gaussian and Laplacian noise when applied with larger budgets ($\epsilon > 2$), thanks to its noise function. The state-of-the-art in DP versus Convergence, exemplified by JIT [ICLR 21], has demonstrated that DP negatively impacts the convergence rate, especially for smaller epsilon values, leading to considerably longer convergence times of the SOTA methods.

Hence, LMO noise effectively addresses two challenges (accuracy and convergence) simultaneously, making it a solution that hits two birds with one stone.

Reviewer comment (Evaluation):
The comparison of noise of LMO-DP to standard Gaussian noise for DP is unclear. The work claims "less noise" and relies on Figure 2 to compare this for several choices of epsilon. However, due to the scale, it is very vague what the actual difference is. The figure should use a log scale and also report useful statistics like the average reduction. Interpreting this figure, it looks as though the chosen noise is often 0 (or, vanishingly small). This would be an extremely significant improvement over DP that requires much more analysis to both understand how/where this is coming from and to ensure correctness. One key analysis would be to show what the noise looks like under the extremes (i.e., any single component), and, for example, to show how this noise changes as a function of the components.

Author response:
Thanks very much for this suggestion. We changed it to the log scale and reported the average reduction with more data samples (please see Figure A.1 in the supplementary file). The results for noise comparison look more clear (**the average reduction rate is 95.13% for $\epsilon=0.3$, 92.19% for $\epsilon=0.7$, 87.71% for $\epsilon=2$, and 87.31% for $\epsilon=3$**).

Reviewer comment (Reproducibility):
The empirical details are lacking for both reproducibility and understanding of results. How large are the dataset sizes? How many classes are there? These details significantly influence DP learning.

Author response:
Thanks for this comment. The optimal weights  $\alpha_1$, $\alpha_2$ and $\alpha_3$ for mixture distributions to generate the noise are in the range [0.1, 0.3]. After that, we use the full datasets and all the classes in the original datasets for fine-tuning (same as SOTA). SST-2 has more than 60k+ samples in the training set; QNLI has more than 100k+ samples; MNLI and QQP contain more than 350k but less than 400k samples for each dataset. SST-2, QNLI, and QQP include two classes each; MNLI includes three classes. The hyperparameters setting: batch size 2048, 6 epochs and sampling rate 2048/|D|. We will clarify it in the paper and the readme file for the code.

Reviewer comment (Presentation):
Figure 5 is extremely difficult to interpret. Looking at it, it looks like this model can be trained in less than 10 steps. This is quite surprising. How many steps does non-private training take?

Author response:
This shows the steps that each method needs to achieve the same accuracy. Specifically, we evaluate the steps that need to reach 48.51% accuracy and 41.83% accuracy for the BERT-base and RoBERTa-base on the MNLI-m dataset; the steps that need to reach 70.49% accuracy and 71.09% accuracy for the BERT-base and RoBERTa-base on the QQP dataset; the steps that need to reach 48.98% accuracy and 45.67% accuracy for the BERT-large and RoBERTa-large on the MNLI-m dataset; the steps that need to reach 63.18% accuracy and 67.84% accuracy for the BERT-large and RoBERTa-large on QQP dataset.

**We cannot compare the steps to achieve the highest/converged accuracy since the baseline(s) cannot achieve such high accuracy (given the same small privacy budget $\epsilon$).** Thus, we found the minimum of the highest accuracy for all the tasks on a specific dataset using similar-parameter models as the reference; then we count the steps to reach this reference.

For instance, LMO-DP needs 1146 steps on the MNLI-matched dataset and 192 steps on the SST-2 dataset (6 epochs) using the roberta-base model. The non-private training needs relatively less (but comparable) steps.

Reviewer comment (Evaluation):
Figure 4 and several tables indicate that LMO-DP with a full order magnitude less privacy cost can outperform DP-SGD. This is an outstanding feat that is likely due to Point 5 (the unclear noise comparison). Understanding how this manifests is crucial and is currently underexplored.

Author response:
Thanks for the suggestion. We will clarify this relationship between noise value and the language model performance in the revised version. Specifically, we propose a suboptimal solution for the cross-entropy optimization based solely on minimizing DP accountant, which helps us reduce the noise for gradients. In our revised version, we redraw Figure 2 to better visualize the comparison between the LMO-DP noise and Gaussian noise. With significantly reduced noise (e.g., high reduction rate), LMO-DP only needs a very small privacy cost to achieve high accuracy compared to DP-SGD.

Reviewer comment (Theory):
Though the reviewer is not very familiar with the work of Mohammady et al. (2020), their work requires specifying both the utility metric and the query (to be protected by DP). Theorem 4.1 currently only shows the loss/utility metric (cross-entropy) but does not define the query. This makes it seem as though the query is actually releasing the loss and not the gradients, which would lead to privacy leakage. This requires clarification.

Author response:
This is a good question. Yes, Mohammady et al. (2020), requires specifying the utility metric and the query. We define the cross-entropy loss as the utility metric, and the optimization for noise/randomization is formulated at the level of training/fine-tuning (by considering all the gradients with accounted DP leakage as the query). Note that the LMO space is defined based on the overall DP guarantee for the entire training/fine-tuning, which ensures the same DP guarantee as the DP-SGD on noisy gradients with the RDP accountant.  In other words, noise (injected to the gradients) is optimally derived for the training/fine-tuning while ensuring the same overall DP guarantee as DP-SGD. 

Different from Mohammady et al. (2020), we theoretically prove that optimizing the noise based on the cross-entropy loss/utility metric can be approximated to be equivalent to search over the LMO space (Theorem 4.1). Then, the optimization problem can be more efficiently solved. Note that optimizing the noise at the gradient level may also be feasible, but based on a different metric and different space for DP guarantee of the gradient, which may not be efficiently solved.