Paper: RGLA: Reverse Gradient Leakage Attack using Inverted Cross-Entropy Loss Function (ICLR-2024)

Reviewer comment (Novelty):
Few of the authors' claims are supported by evidence. For instance, the claim in the abstract that this is the "first" successful disaggregation of the feature map in a "generic FL setting" is incorrect. It is not the first article to disaggregate feature maps; see for example [1], an important reference missing from the paper. Also, while less important, the relevant work of [4] is missing as well. If the authors claim they target large batches of large resolutions, [4] would have been the reviewer's first choice as a baseline, although [4] uses batch-norm statistics.

Author response:
Thank you for bringing the works [1, 4] to our attention. We would like to emphasize that there are differences between them and our paper in terms of methods and results, and we have removed this statement in the revision. For [1], the Cocktail Party Attack (CPA) treats the problem of gradient inversion as a Blind Source Separation (BSS) issue, based on the premise that the gradients of a Fully Connected (FC) layer can be seen as a linear combination of its inputs. CPA then uses Independent Component Analysis (ICA) to address the BSS problem. In contrast, our method initially leaks the model's output and then further leaks the feature map. This strategy allows all methods that invert model inputs based on model outputs to be applicable in the context of gradient leakage attacks, thereby significantly increasing the vulnerability of gradients. We discuss this paper in the related works section of our rebuttal version.

Since there is no official source code for the previous work [4] (STG), we perform the STG using the STG source code published by FGLA [8], which we also included in our open-source code. We compared our method with STG (with or without BN statistics) using the ImageNet dataset (224$\times$224) and batch size of 8, and the results are shown in Table 1. It is clear from Table 1 that our method outperforms prior work [4] in terms of time efficiency and reconstruction results.

| Method                    | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | Time$\downarrow$(s) |
| ------------------------- | -------------- | -------------- | ----------------- | ------------------- |
| Our RGLA                  | 19.24153       | 0.50947        | 0.55870           | 14.35               |
| STG with BN statistics    | 11.47697       | 0.35237        | 0.66932           | 9484.70             |
| STG without BN statistics | 9.90506        | 0.32183        | 0.89051           | 7497.48             |

Table 1

Reviewer comment (Experiments):
The authors' use of the term "generic FL setting" is misleading. The attack setting described in the threat model is not generic because:  
1. The attack requires access to the ground truth labels, which is contrary to most of its competition. Additionally, the wording "𝒜 can retrieve ground-truth labels from the gradient" is unclear — it is not evident how this is possible, especially since there is no state-of-the-art method that works with duplicate labels.  
2. Authors do not tackle the multiple updates case, which is characteristic of FL.  
3. The attack requires access to an external dataset.  
4. The attack can only work on classification tasks with more classes than the batch size.  
Although 3 and 4 are standard assumptions in the literature, points 1 and 2 make the setting extremely contrived. This simplified setting has some interest, but the claims do not match reality. Moreover, in the code repository, in the attack's code, the signature of the function does not include the labels variable, so executing the function would result in "labels is undefined" errors, misrepresenting the function as written in the article. The "defense" of increasing batch size is more a limitation of the attack than a true defense deserving its own section. There is also no experiment in the main paper showing the effect of image resolution on attack performance, despite the claim that the attack works on high-resolution images.

Author response:
1. We would like to clarify that the assumption of ground-truth labels is reasonable for the reason. This label assumption shared in recent studies [7-10], which either assume that the adversary knows the labels [7, 8], or first extract the labels [9, 10, 11] using existing label inference techniques.

   To improve rigor, we combine our method with a stat-of-the-art label inference technique, iLRG [6]. Table 2 shows the average results of data reconstruction on 100 batches. Notably, iLRG achieved a 100% accuracy rate in our experiments, though the order of the reconstructed labels differs from that of the actual ground-truth labels. As shown in Table 2, the reconstructed results obtained using ground-truth labels and those reconstructed results using the iLRG [6] are similar. Fig. 11 in our rebuttal version provides visual reconstruction results, illustrating that both ground-truth labels and inferred labels can successfully reconstruct visual outcomes, except that using inferred labels causes the reconstructed data to be in a different order from the original data. We also add this experiment and corresponding discussion in the Appendix of our rebuttal version.

   |                                 | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ |
   | ------------------------------- | -------------- | -------------- | ----------------- |
   | Ground-truth label + RGLA(Ours) | 19.24153       | 0.50947        | 0.55870           |
   | iLRG[6]+ RGLA(Ours)             | 19.22677       | 0.49973        | 0.56795           |

   Table 2

Reviewer comment (Writing):
The paper's writing is subpar. Most sentences are either too strong (as in the above examples) or colloquial (e.g., use of "very" eight times, phrases such as "we establish the second equation" followed by a citation), or not understandable due to typos or grammar mistakes. Notations and acronyms should be defined when first introduced. For example, acronyms like DLG, FGLA, HCGLA, GGL, IG, etc., are not introduced. Likewise, notations such as \( y(i) \), \( \nabla \theta^{W}_{L, y(i)} \), \( \mathcal{L}_{1} \), and similar math expressions are undefined; as a result, phrases like "having lower \( \mathcal{L}_{1} \) values" are meaningless to the reviewer, preventing comprehension of Figure 3. While undefined notations are roughly understandable for readers experienced in the literature, they make the paper painful to read.

Author response:
Thank you very much for your suggestion!

1. We improved our expressions in the rebuttal version. 

2. In the rebuttal version, we introduce acronyms such as DLG (Deep leakage attack), etc. $y(i)$ denotes the label of the $i^{th}$ sample, and $\nabla {\theta_ {L}^{b}}_ {y_ {(i)}}$ denotes the $y(i)^{th}$ row of the gradient of the bias layer, and the definition of $\mathcal{L}_ 1$ is in Equation 4 on the fifth page of the main text of the manuscript.

Reviewer comment (Writing):
Minor – Replace all mentions of the "last" layer of FC with "first" layer of FC, as the usual convention is to count layers from the input. Thus, the feature map is the input of the first layer of the FC, not the last. If the authors use a different convention, they must define it.

Author response:
2. In our rebuttal version, we have defined our feature map to be the input to the last fully connected layer

Reviewer comment (Writing):
Minor – Typos: "an analysis method" (page 3?); "almost non-loss" (page 5?); "will resolve" (page 4); The sentence "When B < C, we cannot recover unique y with the established system of equations is overdetermined, thus there may be multiple solutions, of which only one is a right solution, and the remaining solutions are referred to in prior work R-GAP (Zhu & Blaschko, 2020) as the twin solutions." (page 6) is incorrect.

Author response:
3. The typos mentioned have been corrected in the latest version, which you can download and check out!

Reviewer comment (Theory):
The proposed method is built on several existing techniques that made disparate adversarial assumptions. As a result, the proposed RGLA combining these techniques requires a quite restrictive threat model, e.g., access to auxiliary datasets and ground-truth labels. In particular, RGLA requires the target network to have no pooling layer, which is hard to justify in practice. On the other hand, RGLA does not need batch statistics for reconstructing batched input but does seem to require B to not exceed C+1.

Author response:
>**Q1 1. Assumption of auxiliary datasets 2. Assumption of ground-truth labels. 3. Requirement on the target network to have no pooling layer. 4. Requirement on B to not exceed C+1.**

1. Accessing an auxiliary dataset: many of the more recent works [1, 2, 3, 4, 5, 6] in our research field require access to an external dataset. Moreover, we believe that in a real federated learning scenario, an adversary can use some open-source datasets as auxiliary data to facilitate the attack, which is not a difficult thing.

2. Access to ground-truth labels. We would like to clarify that the assumption of ground-truth labels is reasonable for this reason. This label assumption shared in recent studies [1, 2, 4, 9, 10], which either assume that the adversary knows the labels [1, 2], or first extract the labels [4, 9, 10] using existing label inference techniques.

   To improve rigor, we combine our method with a state-of-the-art label inference technique, iLRG [8]. Table 1 shows the average results of data reconstruction on 100 batches. Notably, iLRG achieved a 100% accuracy rate in our experiments, though the order of the reconstructed labels differs from that of the actual ground-truth labels. As shown in Table 1, the reconstructed results obtained using ground-truth labels and those reconstructed results using the iLRG [8] are similar. Fig. 11 in our rebuttal version provides visual reconstruction results, illustrating that both ground-truth labels and inferred labels can successfully reconstruct visual outcomes, except that using inferred labels causes the reconstructed data to be in a different order from the original data. We also add this experiment and corresponding discussion in the Appendix of our rebuttal version.

   |                                 | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ |
   | ------------------------------- | -------------- | -------------- | ----------------- |
   | Ground-truth label + RGLA(Ours) | 19.24153       | 0.50947        | 0.55870           |
   | iLRG [8] + RGLA(Ours)           | 19.22677       | 0.49973        | 0.56795           |

   Table 1

3. The target network to have no pooling layer: Indeed, we removed the average pooling layer over the fully-connected layer (FC layer) in our experiments in order to make the feature map larger so that it contains more information about the original data, which is consistent with what was done in [2]. In order to explore how much information is lost by the average pooling layer, we conducted experiments in two scenarios: one in which the pooling layer is removed and the other in which the pooling layer is retained. And keep the rest of the experimental settings the same, including a batch size of 8, a dataset of ImageNet, and attacking the same 100 data batches. According to the experimental results in Table 2, whether the pooling layer is removed or not, our proposed method is able to reconstruct model outputs that are very close to the actual model outputs, which further reconstructs feature maps that are similar to the original feature maps. However, in the setting with the pool layer, the generator can not generate the original data even with the precise feature map. This phenomenon can be attributed to the fact that the addition of the pooling layer reduces the information about the original data contained in the feature maps, making it difficult for the generator to reconstruct the original image from feature maps that contain less information about the original data. The contribution of our manuscript does not lie in the training of the generator, which is the main contribution of [1]. In the future, we also expect more powerful generator models to be able to reconstruct the original images from corresponding feature maps even with limited information contained.

|                               | $\left\|\hat{y}^{\prime} - \hat{y} \right\|^2$ $\downarrow$ | $\left\|f^{\prime} - f \right\|^2$ $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ |
   | ----------------------------- | ----------------------------------------------------------- | ----------------------------------------------- | --------------- | --------------- | ------------------ |
   | Without average pooling layer | 2.00E-04                                                    | 1.01E-04                                        | 18.72215        | 0.48819         | 0.57847            |
   | With average pooling layer    | 9.57E-05                                                    | 1.76E-07                                        | 10.97777        | 0.29773         | 0.511608           |

   Table 2: The $\hat{y}$ is the actual model output, the $\hat{y}^{\prime}$ is the optimized model output, the $f$ is the actual feature map, and the $f^{\prime}$ is the obtained feature map.

4. Requirement on B to not exceed C+1: We have written the B<=C as a limitation of our method in the conclusion section in the rebuttal version.

Reviewer comment (Novelty):
The technical contributions of this work are not particularly clear. The core techniques adopted by RGLA for enabling large batch recovery (e.g., disaggregating gradients and training a generator model) were discovered in prior work by Xue et al. and the inversion on the model output was discussed by Zhu & Blaschko et al. The only distinction seems to be the relaxation on duplicate labels. Besides, some closely related works were not compared/discussed. For instance, [1] also trains a model to learn inverse mapping, and the idea of disaggregating and inverting feature vectors is very similar to the cocktail party attack [2].

Author response:
>**Q2  1. The technical contributions of this work are not particularly clear. The core techniques adopted by RGLA for enabling large batch recovery (e.g., disaggregating gradients and training a generator model) were discovered in prior work by Xue et al. and the inversion on the model output was discussed by Zhu & Blaschko et al. The only distinction seems to be the relaxation on duplicate labels. 2. Besides, some closely related works were not compared/discussed. For instance, [11] also trains a model to learn inverse mapping, and the idea of disaggregating and inverting feature vectors is very similar to the cocktail party attack [12].**

1. While it is true that Zhu & Blaschko et al. [9] have discussed the inversion of model outputs, their method is applicable only under very restricted conditions (B=1, C=2) and suffers from the challenge of twin data. In contrast, our RGLA approach formulates three innovative relations that enable the optimization of model outputs without the limitation of C. Our method accurately reconstructs model outputs as long as B<C, and it cleverly avoids the presence of twin data. We believe this is a significant advancement because once the model's output is leaked from the gradient, all methods in the model inversion attack domain (inverting the model's output to input) can be applied to gradient leakage attacks. The process of leaking the output and then the feature map to generate the original input, as described in our manuscript, is just one example. In reality, once the model's output is leaked, there are numerous methods in the model inversion attack domain that can be used to reconstruct the model's input. We believe this further emphasizes the uniqueness and importance of our contribution to this field.

2. We have included a discussion of related works [11, 12] in the related works of our rebuttal version.

Reviewer comment (Theory):
The proof of Proposition 2 made assumptions about the expressivity of the model, which should be made explicit in the text.

Author response:
>**Q3 The proof of Proposition 2 made assumptions about the expressivity of the model, which should be made explicit in the text.**

Thank you for your meticulous review of our proof. We explicitly state the assumptions about the expressibility of the model in Section 3 in the rebuttal version, which is the concern of you and the external comment from Dimitar Iliev Dimitrov.

Reviewer comment (Evaluation):
The image similarity-based privacy metrics have some inherent limitations. For instance, the reconstructed images on CelebA have high measured PSNR but barely reveal any practically identifiable information. It would be better to add corresponding discussions.

Author response:
>**Q4 The image similarity-based privacy metrics have some inherent limitations. For instance, the reconstructed images on CelebA have high measured PSNR but barely reveal any practically identifiable information. It would be better to add corresponding discussions.**

Thank you for pointing out this. We have discussed the limitations of the widely-used metrics in our rebuttal version.

Reviewer comment (Experiments):
From what the reviewer understands, the performance degradation of exiting attacks in the duplicated label scenario is an artifact of the label ambiguity. If that’s the case, existing optimization-based attacks should perform as well as if there were no duplicated labels in the extreme case where all images come from the same class, but that’s not what’s observed in Table 8 - existing attacks still perform poorly even if there is no label ambiguity. What causes the performance of existing attacks to drop? How would these methods perform if the true labels are assumed to be known?

Author response:
>**Q5 1. From what the reviewer understands, the performance degradation of exiting attacks in the duplicated label scenario is an artifact of the label ambiguity. If that’s the case, existing optimization-based attacks should perform as well as if there were no duplicated labels in the extreme case where all images come from the same class. 2. That’s not what’s observed in Table 8 - existing attacks still perform poorly even if there is no label ambiguity. What causes the performance of existing attacks to drop? 3. How would these methods perform if the true labels are assumed to be known?**

1. Thank you for your insightful observations. The performance degradation of attacks in scenarios with duplicated labels is due to the entanglement of gradients corresponding to data with the same labels in the fully connected layer's gradient matrix, a phenomenon also described in previous work [13]. Since gradients are calculated through backpropagation, the gradients of the convolutional layers affected by this part also become entangled. As a result, the more duplicated labels there are, the more complex the entanglement of gradients becomes, making the optimization increasingly challenging. Analytical-based methods cannot reconstruct duplicated labels; thus, as the proportion of duplicated label data in a batch increases, the overall reconstruction metrics of the batch deteriorate.

2. The subpar performance of these methods in non-duplicated label scenarios is due to large batch size of 8 and high resolution of 224$\times$224. This results in a search space of 8$\times$224$\times$224 variables, far exceeding the maximum variable size these methods claim to reconstruct in their respective papers (as shown in Table 1 of our manuscript). Therefore, even in the absence of duplicated labels, their methods perform poorly.
3. In our comparative experiments, DLG, IG, and FGLA all used known true labels, while the GGL method used labels inferred by iDLG. For fairness in comparison, as also mentioned by reviewer iafz, we re-performed the GGL method with known labels and updated our experimental results in the rebuttal version. These results demonstrate that our proposed RGLA method still outperforms others in known ground-truth labels scenario.

Reviewer comment (Theory):
The elimination of the twin solution seems to rely on its smaller loss value (as empirically verified in Fig. 3). How are these twin data eliminated on a well-trained model where all training data have relatively small loss values?

Author response:
>**Q7 The elimination of the twin solution seems to rely on its smaller loss value (as empirically verified in Fig. 3). How are these twin data eliminated on a well-trained model where all training data have relatively small loss values?**

Thank you for your question regarding the elimination of twin solutions. In fact, the root cause of the emergence of twin solutions lies in the non-monotonicity of $\frac{\partial l(\hat{y}, y)}{\partial y}$. The loss value is just an experimental manifestation of the difference between the twin solution and the positive solution, when the model is well-trained, both the twin solution and the positive solution may have small loss values. In such cases, it becomes necessary to find other method to distinguish between twin solutions and correct solutions. We take this as our future work.

In the rebuttal version of our manuscript, we have clarified in the threat model section that our attack primarily occurs during the model's initialization or pre-training phase. Regardless of the stage at which the attack takes place, any leakage of data is a significant security concern that warrants attention.

Reviewer comment (Experiments):
As RGLA relies on inverting the model output to feature maps, how would defense methods that perturb/modify the FCL part of the model (e.g., [1][2]) affect the attack performance?

Author response:
>**Q8 As RGLA relies on inverting the model output to feature maps, how would defense methods that perturb/modify the FCL part of the model (e.g., [13] [14]) affect the attack performance?**

Thank you for your question. In response to the defense method in [13], it defends by perturbing the feature map. The training of the global model is affected when the feature map is largely perturbed, while our method is still effective in recovering the feature map when the perturbation is small. In order to verify the effectiveness of our method against the defense proposed in [13], we conducted experiments with the maximum defense parameter value (80%) set in the [1, 13] paper, and the results in Table 3 shows that our method is still effective.

|                            | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ |
| -------------------------- | -------------- | -------------- | ----------------- |
| RGLA without defense       | 18.79270       | 0.50706        | 0.55866           |
| RGLA with 80% Soteria [13] | 16.34467       | 0.44887        | 0.58806           |

Table 3

For the defense method mentioned in [14], it defends by embedding a variational module consisting of an encoder and a decoder in the network. Since this variational module contains randomly sampled vectors, it becomes very difficult to train a generator that is able to reverse from the feature map back to the original input, even if the feature map is accurate. Therefore, this defense method mainly defends our attack by affecting the training of the generator.

Reviewer comment (Experiments):
It is interesting to see that gradient clipping and additive noise have little effect on FGLA as combining these two essentially provides some notion of differential privacy. What is the largest epsilon value that is able to defend FGLA and the proposed RGLA attack?

Author response:
>**Q9 It is interesting to see that gradient clipping and additive noise have little effect on FGLA as combining these two essentially provides some notion of differential privacy. What is the largest epsilon value that is able to defend FGLA and the proposed RGLA attack?**

Thank you for your interesting question. Regarding gradient clipping defense, in our experiments, we utilized our auxiliary dataset to estimate the second norm of the gradients and used this information to restore clipped gradients, effectively mitigating the clipping defense. The implementation details are available in the source code provided in our supplementary materials. Simultaneously, FGLA extracts feature maps using the following equation, thereby directly bypassing the Clip defense. Therefore, we did not identify an effective largest epsilon value that could defend against both RGLA and FGLA.

$f^\prime\approx\frac{{\nabla\theta^W_L}^\prime}{{\nabla\theta^b_L}^\prime}=\frac{\nabla\theta^W_L\cdot\frac{\mathcal{C}}{\|\nabla\theta\|}}{\nabla\theta^b_L\cdot\frac{\mathcal{C}}{\|\nabla\theta\|}}$

As for the noisy gradient defense, we evaluate RGLA and FGLA under increasing noise levels. Table 4 shows the PSNR values of reconstructed results of RGLA and FGLA under different noise level. The experimental results are shown in the table below. If we consider a PSNR of 14 as the threshold, then the largest epsilon value that RGLA can reconstruct is 0.1, while for FGLA it is 0.06. We hope these experimental results answer your question and demonstrate the effectiveness of our methods.

|            | 0.01  | 0.02  | 0.03  | 0.04  | 0.05  | 0.06  | 0.07  | 0.08  | 0.09  | 0.1   | 0.11  |
| ---------- | :---: | :---: | :---: | :---: | :---: | :---: | ----- | :---: | :---: | ----- | ----- |
| RGLA(Ours) | 18.90 | 18.97 | 18.41 | 17.76 | 17.08 | 16.43 | 15.81 | 15.23 | 14.72 | 14.25 | 13.83 |
| FGLA       | 18.83 | 18.79 | 18.08 | 17.23 | 15.87 | 14.19 | 12.92 | 11.43 | 10.24 | 9.78  | 9.39  |

Table 4

Reviewer comment (Reproducibility):
The paper lacks sufficient details regarding the training of the generator responsible for mapping final feature maps back into the original input samples. As readers, we are left with questions about the level of effort required to train such a generator and the upper bounds of its capabilities. Additionally, it would be valuable to understand how the performance of the feature inverter generator is affected by the increasing accuracy of the victim model. Could you provide more information into these aspects to enhance our understanding of your work?

Author response:
Thank you very much for your praise on our manuscripts, such as "innovative attacks", "insightful design", "solid theoretical analyses", "extensive and thorough experiments", "well-written", and so on. We are honored to receive such praise and we do our best to address your concerns and questions.

>**Q1  1. There is a lack of detailed information on the training of the generator used for mapping final feature maps to original input samples and clarity on the effort required to train this generator. 2. How the generator's performance is influenced by the increasing accuracy of the victim model.**

Thank you very much for your question.

1. The structure of the generator and its training details have been thoroughly described in the prior work [1]. Here, I would like to add some details that were not mentioned in [1]: the generator has a total of 100,590,979 parameters, and consumes about 23GB of video memory when the batch size is 32 during training. When experimenting on ImageNet, it takes about 2 hours to complete an epoch, and typically after about 4 epochs of training, the generator is able to effectively invert the feature map back to the original model input. This generator training does not affect the efficiency of our attacks. This is because it is becoming increasingly common to use a pre-trained convolutional layer plus an untrained fully-connected layer and to fix the convolutional layer to train only the fully-connected layer during the training process. In this case, we only need to train one generator to attack the private data of multiple clients in multiple rounds of federated learning.

2. As the accuracy of the attacked model increases, the feature map will contain more information about the original data, potentially reducing the time required to train an efficient generator.

Reviewer comment (Theory):
My primary question is about the information available to the attacker through gradients, which is essentially an aggregated representation from a batch of inputs. I'm curious about how your method manages to distinguish features of different images within the same class. I couldn't identify explicit constraints in the optimization objective that encourage disentanglement between samples from the same class. Intuitively, I would expect that reconstructed samples from the same class might exhibit entangled features, resulting in averaged reconstruction or mixed semantics. Could you provide additional insights, either theoretical or empirical, to clarify how your approach achieves this distinction? This would be my biggest concern for understanding how your approach works.

Author response:
This is a good question. Our reconstructed images are generated from feature maps by a generator. Therefore, the problem of distinguishing different images within the same class becomes a problem of distinguishing their feature maps. According to Equation (5) in our paper, the computation of feature maps is as follows: $\boldsymbol{f}^{\prime}=\left(\frac{\partial l\left(\hat{\boldsymbol{y}}^{\prime},\boldsymbol{y}\right)}{\partial\hat{\boldsymbol{y}}^{\prime}}\right)^{-1}\cdot\nabla\boldsymbol{\theta}_L^W$

With known labels and gradients, the feature maps are determined by the optimized model outputs. Our research actually addresses how to distinguish model outputs of different images within the same class. The results of the model output depend on our optimization objective:

$\mathcal{L}_w = \left\|\nabla\boldsymbol{\theta}_L^W\cdot\boldsymbol{\theta}_L^W-\frac{\partial l\left(\hat{\boldsymbol{y}}^{\prime},\boldsymbol{y}\right)}{\partial\boldsymbol{\hat{\boldsymbol{y}}}^{\prime}}\cdot\left(\hat{\boldsymbol{y}}^{\prime},-\boldsymbol{\theta}_L^b\right)\right\|^2$,

$\mathcal{L}_ b = \left\|\nabla\boldsymbol{\theta}_{L}^{b}-\frac{\partial l(\hat{\boldsymbol{y}}^{\prime},\boldsymbol{y})}{\partial\hat{\boldsymbol{y}}^{\prime}}\cdot\mathbb{I}\right\|^{2}$

$\mathcal{L}_{loss} = \left\|l(\hat{\boldsymbol{y}}^{\prime},\boldsymbol{y})-[-\frac1B\sum _{i=1}^B\ln(\nabla\boldsymbol{\theta} _{L\boldsymbol{y}(i)}^b+\frac{\lambda _{\boldsymbol{y}(i)}}B)]\right\|^2$

In our loss objective, the model output considers all samples in a batch, with a size of B$\times$C, rather than an average output. $\mathcal{L}_ w$ establishes equations for every element in the model output, $\mathcal{L}_ {b}$ establishes equations for model outputs within the same class, and $\mathcal{L}_ {loss}$ establishes the overall equation for the entire model output. Therefore, an averaged reconstructed model output might satisfy $\mathcal{L}_ b$ and $\mathcal{L}_ {loss}$, but not satisfy $\mathcal{L}_ w$, as $\mathcal{L}_ w$ requires every element in the model output to meet the equation without any summing or averaging. Thus, our method does not result in similar model outputs for data in the same class, thereby not reconstructing averaged images for the original images within the same class.

The key difference between our method and previous ones is that the latter mainly match averaged gradients, while our method establishes equations for every element in the model output, providing a more granular approach. Only when every element of the model output satisfies the optimization objective will the optimization loss reach its minimum value. Therefore, our method can distinguish images within the same class.

Reviewer comment (Evaluation):
It appears that the reconstructed samples exhibit a degree of over-smoothing, lacking sharp edges and finer details. This effect seems reminiscent of applying a large Total Variation loss or L2 loss. Could you please explain the reasons behind this observation? Is it related to the characteristics of the feature inversion generator? Are there potential room for improvement? If the level of smoothness can be adjusted, what will the results look like if they do not appear so smooth?

Author response:
Thank you for your question! Regarding the excessive smoothing that occurs in the reconstructed samples, as well as the lack of sharp edges and fine details, this may indeed be related to the Total Variation loss or L2 loss that we used. In addition to this, we believe it is also related to the architectural characteristics of our generator. Our generator employs a multi-layer convolutional and deconvolutional network, complemented by LeakyReLU activation functions and batch normalization (BatchNorm). This design helps produce smoother images and mitigating overfitting to some extent. However, it might also lead to the loss of high-frequency details in images. During the training of the generator, we used Mean Squared Error Loss (MSELoss) along with image quality metrics such as PSNR, SSIM, and LPIPS. The combination of these loss functions might further enhance the smoothing effect, particularly as PSNR and SSIM tend to assess overall structural similarity rather than precise detail reproduction. Regarding improvements to reduce image over-smoothing, we believe that adjustments to the loss function, optimization of the generator’s network structure, or the incorporation of techniques specifically aimed at high-frequency detail restoration could yield enhancements. For instance, exploring more complex loss functions like Perceptual Loss or losses used in Generative Adversarial Networks (GANs) might encourage the generation of more detailed and textured images. However, due to time constraints during the rebuttal phase, we were unable to explore more optimization functions and more robust generator models. We hope the reviewers can understand us.

Reviewer comment (Experiments):
See weakness 2, I am curious about the performance of proposed approach when using average pooling layer before the final FC layer.

Author response:
See our answer to Q2.

Reference:

[1] Dongyun Xue, Haomiao Yang, Mengyu Ge, Jingwei Li, Guowen Xu, and Hongwei Li. Fast generation-based gradient leakage attacks against highly compressed gradients. IEEE INFOCOM 2023 - IEEE Conference on Computer Communications, 2023.