Paper: DPFormer: Learning Differentially Private Transformer on Long-Tailed Data (ICLR-2024)

Reviewer comment (Evaluation):
__[W1]__ If data is long-tailed, the hand-wavy definition of this means that some parts of the data are over-represented and some parts are under-represented. As such, if the objective is to get high average accuracy (or its variant for recommendation problem), then ignoring the low frequency elements shouldn't be a problem. This paper seems to be looking at average metrics but it mentions long-tailed data in the title.

Author response:
Regarding long-tailed data [W1].

**Reply:** Thanks for pointing out this perspective. We would like to make it clear that we made no attempt to deal with the fairness issue. Our motivation is long-tailedness in private learning will obstruct the training process (i.e., the attention distraction) and lead to (average) performance drop (i.e., low accuracy on both over-represented and under-represented data). Therefore, our goal in the task of 'improving private learning on long-tailed data' is to mitigate this additional issue and the ideal result is to recover the case of 'non-private (vanilla) learning on long-tailed data' (i.e., potentially low accuracy on under-represented data, but with a high average performance). We make no attempt to improve the model performance on under-represented data. Such goal can be achieved by combining existing method with our method.

Reviewer comment (Theory):
__[W2]__ The paper proposes a phenomenon called `attention distraction'. The suggestion is that tokens with high variance will be unfairly selected by the attention mechanism. But it is not very clear to me what this means formally. In addition, the theory seems to assume that every token's attention value is an independent gaussian, which is also unrealistic as all tokens are dependent on each other due to previous layers and I don't see why adding gaussian noise to gradients results in gaussian distribution for token activations. The whole section 4 assumes these two things but there isn't any empirical or theoretical justification for this.

Author response:
[W2] In addition, the theory seems to assume that every token's attention value is an independent gaussian, which is also unrealistic as all tokens are dependent on each other due to previous layers and I don't see why adding gaussian noise to gradients results in gaussian distribution for token activations. The whole section 4 assumes these two things but there isn't any empirical or theoretical justification for this.

**Reply:** Thanks for pointing out. We would like to stress that these assumptions are commonly used and well-understood in the literature of variational inference and more broadly, bayesian deep learning. We directly adopt this methodolody from that area. The unfortunate fact is, (at this stage of history) we can only assume them being independent Gaussian and approximate it via moment matching. The development of more advanced techniques to help track the evolution of distribution through neural layers is beyond the scope of this work. In addition, we add a more detailed version of analysis of the attention distraction in Appendix A.3 in the revised paper.

Reviewer comment (Experiments):
__[W3]__ The experimental comparison is not thorough at all. Please see question 3 below for a more detailed explanation of why I think that is the case.

Author response:
why the experiments are limited to recommendation settings.

**Reply:** We believe that Recommendation task is the most prevailing sequential task in the real-world that features the property of long-tailedness and lack of pre-trained model. Therefore, we target recommendation settings in the work.

> Is there a comparison with Li et. al?

**Reply:** We compare the efficiency of our phantom clipping with ghost clipping in Li et.al in Figure 3. For the accuracy aspect, the baseline transformer can be viewed as an instantiation of Li et.al since Li et.al did not propose algorithmically different methods.

> Can the authors also try using a general pre-trained transformer for this recommendation tasks?

**Reply:** To our knowledge, how to use a general pre-trained transformer for recommendation task is still a largely open problem. One cannot naively adopt a pre-trained transformer model because the input space is different (in recommendation task, token $i$ is associatedd with the item with ID $i$ while in NLP token $i$ is associated with some (sub)word defined by the tokenizer).

Reviewer comment (Theory):
This paper uses Phantom clipping and Re-attention mechanism to overcome the difficulties of heavy computation and attention distraction. The most important thing is to make the whole mechanism be differentially private and have high utility. However, the authors didn't analyze the privacy of this mechanism, especially after the re-attention. It seems this process will leak privacy.

Author response:
**Reply:** Thanks for pointing out. We have explicitly made the privacy statement in the revised version (Claim A.5). In particular, the privacy guarantee is inherited from that of DP-SGD. Details follow.  

Phantom clipping is an efficient implementation of gradient clipping, having the same functionality as vanilla per-sample clipping (i.e., the same input-output behavior). Therefore, we can think of it as the vanilla gradient clipping when we analyze the privacy.  

For Re-Attention mechanism, observe that the whole computation process only depends on the single data sample (i.e., we do not create dependency among samples between a mini-batch, thus violating DP like Batch Normalization). Another way to understand this is we can do whatever we like in the forward pass as long as the computation associated with one input sample does not rely on other input samples of the mini-batch. (A caveat is discussed in Remark A.2 of Appendix.A.4) Therefore, the private analysis reduces to DP-SGD: By clipping the gradient we can successfully bound the per-sample sensitivity (and add the calibrated noise to guarantee DP).

Reviewer comment (Theory):
What is the privacy guarantee for this method?

Author response:
**Reply:** Thanks for pointing out. We have explicitly made the privacy statement in the revised version (Claim A.5). In particular, the privacy guarantee is inherited from that of DP-SGD. Details follow.  

Phantom clipping is an efficient implementation of gradient clipping, having the same functionality as vanilla per-sample clipping (i.e., the same input-output behavior). Therefore, we can think of it as the vanilla gradient clipping when we analyze the privacy.  

For Re-Attention mechanism, observe that the whole computation process only depends on the single data sample (i.e., we do not create dependency among samples between a mini-batch, thus violating DP like Batch Normalization). Another way to understand this is we can do whatever we like in the forward pass as long as the computation associated with one input sample does not rely on other input samples of the mini-batch. (A caveat is discussed in Remark A.2 of Appendix.A.4) Therefore, the private analysis reduces to DP-SGD: By clipping the gradient we can successfully bound the per-sample sensitivity (and add the calibrated noise to guarantee DP).

Reviewer comment (Evaluation):
What do NDCG@10 and HIT@10 mean?

Author response:
**Reply:** Thanks for pointing out. They are metrics commonly used in recommendation systems to measure the prediction accuracy. They can be simply understood as Top-10 accruacy and weighted Top-10 accuracy as in the classification task. We have included the definitions in Appendix A.9.4 in the revised paper.

Reviewer comment (Experiments):
Some baselines for training with DP are missing where these methods also improve the performance of models for long-tailed data: [1] proposed a way to add DP noise adaptively according to the geometry of the loss space, which can effectively enable smaller noise on long-tailed data. [2] similarly uses auxiliary information such as the frequency of the vocabulary to apply DP noise.

Author response:
**Reply:** Thanks for pointing these great works. We discuss them in the related work in Appendix A.1 in  the revised paper. From our understanding, these works are operating on the optimizer level, while our method does not assume any specific optimizer and uses the underlying optimizer in a black-box manner. Therefore our method is orthogonal to them and can be potentially combined together.

Reviewer comment (Presentation):
Describing section 4 as an algorithm would be easier to read.

Author response:
**Reply:** Thanks for your suggestion. We add an algorithm description in the Appendix A.7.

Reviewer comment (Reproducibility):
Is Figure 3 based on Ghost Clipping or Book-Keeping [1]? BK is supposed to be more efficient than Ghost Clipping.

Author response:
**Reply:** Yes, exactly. We use implementation in BK for the baseline.

Reviewer comment (Writing):
In Equation 7, are the subscripts on the right-hand side supposed to be $X^{(l-1)}$?

Author response:
**Reply:** Yes. Sorry for the confusion.

Reviewer comment (Theory):
It is unclear why phantom clipping has such great advantages over ghost clipping. Observing Claim 3.1, it has exactly the same idea (calculating the norm and avoiding gradient instantiation) as ghost clipping. Specifically, it is not understood why the BM^2 complexity is inherent to ghost clipping under a careful implementation. This raises concern that the paper may over-claim the advantages.

Author response:
$BM^2$ complexity is provided in the ghost clipping paper. Our contribution is to support parameter sharing along with a 'careful implementation'.

Reviewer comment (Evaluation):
Results from other papers (e.g., https://arxiv.org/pdf/2205.10683.pdf) applying ghost clipping to transformers show much better memory efficiency. The gap between Figure 3 and those results is not understood.

Author response:
Our results are not contradictory to existing results. Previous results are obtained when using a large model while we use a relatively small model.  
Note that the advantage of ghost clipping vanishes as model size becomes smaller.

Reviewer comment (Theory):
In section 4.1, the paper mentions that attention distraction mainly happens when different parameters have different variance. However, in DP-SGD, noise with the same standard deviation is added to each parameter. An explanation of this discrepancy is needed.

Author response:
A key observation is, for embedding-based models, not all embedding parameters are equal. As a thought experiment, let the embedding layer be parameterized by $E$ (of shape $m \times d$). Suppose we add a dummy embedding $e_{m+1}$ into $E$ to obtain $E'$ (of shape $(m + 1) \times d$) and start DP training, but no data is associated to that dummy token m+1. During each iteration t, we add Gassian noise $n_t$ to the whole $E'$. After $T$ iterations, it is obvious that e_m+1 will become very large (i.e., $e_{m+1} = n_1 + n_2 + ....+ n_T$, distributed as Gaussian with large variance $T \sigma$). Our definition of effetive error (section 4.2.1) is intended to capture this.

Reviewer comment (Writing):
Section 4.2.2 is very hard to follow, reading like a mixture of several techniques without a clear explanation. There are several confusions, such as whether the assumption of i.i.d. Gaussian for each parameter dimension is realistic, and how to estimate the mean and standard deviation of the start point.

Author response:
Note that Gaussian for each dimension in model parameter is not an assumption. It is induced by the DP Gaussian noise (fixing all other randomness). We presume the reviewer's concern is regarding 'use an approximating distribution $q$ to approximate the  distribution $p$, where $q$ follows a indepent Gaussian distribution'. This is a commonly used and well understood assumption in the literature of variational inference and more broadly, Beyesian deep learning, since $p$ is computationally intractable. We can only approximate it using a Gaussian via moment matching. The development of more advanced techniques to help track the evolution of distribution through neural layers is beyond the scope of this work.  

For example, if the training sequence is [1, 2, 3], then the starting point is $[E_1, E_2, E_3]$ (where $E_i$ is the embedding of token i). The std is estimated by error instantiation (see section 4.2.1). The mean is estimated by itself, $E_i$ (note that access to this noisy parameter can be  
interpreted as a single sampling opportunity from its underlying Gaussian distribution, which can then be viewed as a one-time Markov Chain sampling. Therefore, the noisy parameter can serve as an estimate of its mean).