Paper: GREAT Score: Global Robustness Evaluation of Adversarial Perturbation using Generative Models (ICLR-2024)

Reviewer comment (Theory):
The definition of robustness in Eq. (1) and Eq.(4) seems to be confusing and possibly wrong. In Eq. (1), $\Delta_{\min}$ is defined as the minimal perturbation of a sample-label pair causing the change of the top-1 class prediction. Then, I understand $g(x)$ is a lower bound of $\Delta_{\min}$. However, in Eq. (3), $g(x)$ is defined as the gap between two probabilities. Therefore, I am confused about how the gap between two probabilities measures the minimal perturbation.

Author response:
## Question 1: The definition of robustness in Eq. (1) and Eq.(4) seems to be confusing and possibly wrong, how does the gap between two probabilities measure the minimal perturbation?

The reviewer's question is exactly one of the major contributions of our work - establishing the connection between the defined metric in Eq. (1) and the derived certified lower bound in Appendix A.2.3. The novelty lies in the use of a generative model taking random Gaussian noise as input and adapting the Stein’s Lemma to derive a global certified lower bound on the minimal perturbation defined in Eq. (1). Please see the proof of Theorem 1 in Appendix 6.4 for the detailed derivation. If the reviewer still feels uncertain about this connection, we are willing to engage in further discussion to address the concern.

Reviewer comment (Presentation):
The table and figure should be resized to be larger, which makes it easier to read.

Author response:
## Question 4: The table and figure should be resized to be larger, which makes it easier to read.

In the updated version, we have included new changes requested by reviewers and shortened some parts to remain in 9 pages.

Reviewer comment (Presentation):
In the background and related works section, it is suggested to simplify the introduction of generative models and add the description of global robustness evaluation rather than Appendix 6.3.

Author response:
In the updated version, we have included new changes requested by reviewers and shortened some parts to remain in 9 pages.

Reviewer comment (Theory):
It is recommended to give a more detailed time complexity calculation.

Author response:
Here we analyze the time complexity of our algorithm. The time complexity of the GREAT Score algorithm is determined by the number of iterations (generated samples) in the loop, which is denoted as $N_S$. Within each iteration, the algorithm performs operations such as random selection, sampling from a Gaussian distribution, generating samples, and predicting class labels using the classifier. We assume these operations have constant time complexity $I$ and absorb them in the big $O$ notation.

Additionally, the algorithm computes the sample mean of the recorded statistics, which involves summing and dividing the values. As there are $N_S$ values to sum and divide, this step has a time complexity of $O(N_S)$.

Therefore, the overall time complexity of the algorithm can be approximated as $O(N_S \cdot I)$.

For CW attack, the optimization process iteratively uses gradients to find the adversarial perturbation. The number of iterations required depends on factors such as the desired level of attack success and the convergence criteria. Each iteration involves computing gradients, updating variables, and evaluating the objective function. It also involves a hyperparameter search stage to adjust the weighted loss function.

Let $B$ be the complexity of backpropagation, $T_g$ be the number of iterative optimizations, and $T_b$ be the number of binary search steps for the hyperparaneter. The dominant computation complexity of CW attack for $N_S$ samples is in the order of $O(N_S \cdot T_g \cdot T_b \cdot B)$. Normally, $T_g$ is set to 1000, and $T_b$ is set to 9. Therefore, the CW attack algorithm is much more time-consuming than ours. In the table below, we include a table of run-time comparison of GREAT Score and CW Attack on the same 500 examples for Rebuffi\_extra and Gowal\_extra models.





| Model Name     | GREAT Score (seconds per sample) | CW Attack (seconds per sample) |
| -------------- | -------------------------- | ------------------------ |
| Rebuffi\_extra |       0.038                     |                    11.376       | 
| Gowal\_extra   |        0.034             | 11.544                   |

Reviewer comment (Presentation):
In the experimental results section, the layout of the experimental results graphs and tables is confusing, and it is recommended to rearrange them.

Author response:
We understand the reviewer's concern, but the space of our revision is constrained by the ICLR submission guideline that does not allow for an extra page in the revised version. Nonetheless, we have made adjustments to our revised version to include the requested changes by all reviewers and optimize the graph and table presentations.

Reviewer comment (Presentation):
Appendix chapters need to be arranged according to the order in which they appear in the main text.

Author response:
We have updated our paper to rearrange the order.

Reviewer comment (Experiments):
For other competitive methods, it would be better to use all the data in the dataset to measure robustness rather than the strategy proposed by the author?

Author response:
For a fair comparison, we indeed compared our metric with other methods on the same set of generated samples. For baseline methods, we also reported their robust accuracy on the entire test set. For example, as demonstrated in Table 2, the utilization of AutoAttack on both synthetic and original data resulted in a low correlation coefficient of 0.7296, suggesting the instability of using AutoAttack for model ranking. Moreover, scalability is the unique strength of our method, as our method only requires forward inferences with the predictions made on the generated data samples, whereas attack algorithms are computationally intense, as discussed in the run-time analysis in Sec. 4.4.

Reviewer comment (Evaluation):
Does it still make sense to use a lower bound if a true global robustness evaluation can be obtained with high computational efficiency?

Author response:
Based on the findings presented in following research, it has been established that the computation of the local minimum perturbation is an NP-hard/NP-complete problem, which means the true global robustness is computationally inefficient to obtain. Therefore, computationally efficient lower bounds such as GREAT Score become a viable alternative for global robustness evaluation.


Daskalakis, Constantinos, Stratis Skoulakis, and Manolis Zampetakis. "The complexity of constrained min-max optimization." Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing. 2021.

Katz, Guy, et al. "Reluplex: An efficient SMT solver for verifying deep neural networks." Computer Aided Verification: 29th International Conference, CAV 2017, Heidelberg, Germany, July 24-28, 2017, Proceedings, Part I 30. Springer International Publishing, 2017.

Reviewer comment (Writing):
The paper's description of some relevant work is either terse or lacking. For instance, "using the Rebuffi_extra model (Rebuffi et al., 2021)" <- please give a short inline description of this model.

Author response:
## Question 2: The paper's description of some relevant work is either terse or lacking.

We understand the reviewer's concern, but our revision is constrained by the ICLR submission guideline that does not allow for an extra page in the revised version. Nonetheless, we have made adjustments to our revised version to expand the descriptions of related work in the main paper. We also note that we have provided extended discussion in the supplementary material.

Reviewer comment (Writing):
The paper's description of some relevant work is either terse or lacking. For instance, "successful adversarial perturbations (an upper bound on the minimal perturbation of each sample) returned by any norm-minimization adversarial attack method such as the CW attack (Carlini & Wagner, 2017)" <- Please give a brief description of the CW attack.

Author response:
### Question 3.1: Explain CW (L2) attack solves an optimization objective that results in low L2 distortion.

Using our nation, consider a $K$-way classifier $f$. Let $x$ be a data sample and $y$ be its top-1 classification label. Denote $\delta$ as the adversarial perturbation. The untargeted CW Attack ($L_2$ norm)  solves the following optimization objective:

$$\delta^* = \underset{\delta}{\arg \min }  ( \Vert \delta \Vert_2^2  + \alpha \cdot \max \{f_y(x + \delta) - \max_{k \in \{1,\ldots,K\},k\neq y} f_k(x + \delta),0 \}  ),$$ where $f_k(\cdot)$ is the prediction of the $k$-th class, and $\alpha > 0$ is a hyperparameter. 




If the second term of the objective function is zero, then by definition $x+\delta$ is a perturbed sample of $x$ causing label flipping. In CW attack implementation, given $\alpha$, it uses a gradient-based optimization solver and runs it for $T_g$ iterations to solve for the smallest $\delta$ that makes the second term zero. Then, it performs a binary search on $\alpha$ for a $T_b$ number of trials (in each trial, another round of multiple iterations is executed) to ensure the second term remains zero while aiming to minimize the first term (the square of the $L_2$ performance level). Finally, the attack algorithm returns the smallest perturbation in the entire process such that $f_y(x + \delta) < \max_{k \in \{1,\ldots,K\},k\neq y} f_k(x + \delta)$ (equivalently, when the second term is 0). If no solution is found, one should increase either $T_b$ and $T_g$. 

The dominant computation complexity of CW attack for $N_S$ samples is in the order of $O(N_S \cdot T_g \cdot T_b \cdot B)$, where $B$ is the complexity of backpropagation. On the other hand, the complexity of GREAT Score is $O(N_S \cdot I)$, where $N_S$ is the number of generated samples, and $I$ is the forward inference cost of the generative model and the classifier. A run-time comparison is shown in Sec. 4.4. In the table below, we include a table of run-time comparisons of GREAT Score and CW Attack on the same 500 examples for Rebuffi\_extra and Gowal\_extra models.





| Model Name     | GREAT Score (seconds per sample) | CW Attack(seconds per sample) |
| -------------- | -------------------------- | ------------------------ |
| Rebuffi\_extra |       0.038                     |                    11.376       | 
| Gowal\_extra   |        0.034             | 11.544                   |

Reviewer comment (Evaluation):
"Moreover, as a byproduct of using generative models, our adversarial robustness evaluation procedure is executed with only synthetically generated data instead of real data, which is particularly appealing to privacy-aware robustness assessment schemes, e.g., remote robustness evaluation or auditing by a third party with restricted access to data and model." <- Please note that this is not a panacea, i.e., see:  
- Carlini, Nicolas, et al. "Extracting training data from diffusion models." 32nd USENIX Security Symposium (USENIX Security 23). 2023.  
- Duan, Jinhao, et al. "Are diffusion models vulnerable to membership inference attacks?." arXiv preprint arXiv:2302.01316 (2023).

Author response:
## Question 4: Privacy-aware question for training data. 

Thank you for your advice. To our understanding, the mentioned paper discloses the risk of leaking private *training* data through model queries at inference time. However, our statement is about preventing the use of private and sensitive data to evaluate/audit a model. The scope is different from the privacy of training data. For example, when evaluating online facial recognition APIs (as discussed in Sec. 4.5), using our generated samples is more privacy-preserving than using real private and sensitive facial images. On the other hand, in this context, the mentioned paper is interested in finding private facial images used to train those APIs, which is beyond our scope.

Reviewer comment (Writing):
"In Figure 2, we compare the cumulative robust accuracy (RA)" <- RA is used before its definition on page 7; please define on the first use of acronym RA.

Author response:
## Question 5: "In Figure 2, we compare the cumulative robust accuracy (RA)" <- RA is used before its definition on page 7; please define on the first use of the acronym RA. 

We have defined it in the summary of Classifiers on the RobustBench part.

Reviewer comment (Writing):
"using the Rebuffi_extra model (Rebuffi et al., 2021)" <- please give a short inline description of this model.

Author response:
## Question 6: "using the Rebuffi_extra model (Rebuffi et al., 2021)" <- please give a short inline description of this model.

We have added it to the updated version.

Reviewer comment (Theory):
The authors utilise the generative models to estimate models' global robustness and try to provide statistical guarantees which is claimed as one of the main theorems. My biggest concern is from this point. Since generative models are not perfect, there exists a notable gap between the generative and true data distribution. We are still worrying about the error bound induced by the difference between the underlying data distribution and generative data distribution. The paper would greatly benefit from including probabilistic guarantees or error bounds on the estimated global robustness concerning the true data distribution, as opposed to solely with respect to the generative data distribution. Such bounds would significantly enhance the appeal and credibility of the results.

Author response:
The reviewer's intuition is correct. In our ablation study of generative models (GMs), we do find that better GMs give higher ranking. In Figure 3, we show that increasing the Inception Score of GMs can significantly increase the Spearman's rank correlation. Intuitively, a higher inception score means better learning of GMs to the underlying data distribution, resulting in improved ranking efficiency in our case. We also had a discussion on the approximation guarantee of some GMs to the true data distribution in Sec. 6.2.

Reviewer comment (Evaluation):
Table 1 illustrates the comparison between (Calibrated) GREAT Score v.s. minimal distortion found by CW attack on CIFAR-10. However, the table does not explicitly show the advantages of GREAT score compared to other metrics. For example, the CW distortion score between Rebuffi_extra and Gowal_extra is pretty large, but the GREAT score only differs 0.003.

Author response:
The main message we wanted to deliver for Table 1 is to show the complete statistics of different methods on all CIFAR-10 models. The results also verify our main theoretical contribution (Theorem 1) that GREAT Scores are indeed certified lower bounds of minimal perturbation, as the perturbations found by CW Attack (an upper bound) are all numerically larger than that of GREAT Score.

The "Advantages" of GREAT Scores are demonstrated in other figures and tables in the same section. For example, high model ranking correlation (Sec. 4.3, Table 2 & Table 3), fast run time (Sec. 4.4 & Fig. 4), and its utility in the evaluation of online black-box APIs (Sec. 4.5, Table 4 & Table 5).

Reviewer comment (Novelty):
Is there any other options besides generative models or GANs, which can approximate the unknown data distribution?

Author response:
This is a great question! Beyond our discussion in Sec. 6.2, in addition to generative models such as GANs or Diffusion models, another approach commonly used to approximate unknown data distributions is Kernel Density Estimation (KDE). KDE is a non-parametric statistical tool that can provide an estimation of the underlying data distribution based on observed data samples.

Several research papers have explored the application of KDE for approximating hidden data distributions, such as

S. A. Bigdeli, G. Lin, L. A. Dunbar, T. Portenier and M. Zwicker, "Learning Generative Models Using Denoising Density Estimators," in IEEE Transactions on Neural Networks and Learning Systems, doi: 10.1109/TNNLS.2023.3308191.




M. Rosenblatt, Remarks on some nonparametric estimates of a density function.Ann.Math. Stat.27, 832–837 (1956).

E. Parzen, On estimation of a probability density function and mode. Ann. Math. Stat.33, 1065–1076 (1962).

However, it is important to note that KDEs typically perform well in low-dimensional settings and they are known to suffer from the curse of dimensionality. For common datasets like CIFAR-10 and ImageNet, KDE may not be the most effective approach to accurately approximate the underlying data distribution given the limited size of their training data.