Paper: Towards Adversarially Robust Condensed Dataset by Curvature Regularization (ICLR-2024)

Reviewer comment (Evaluation):
Previous methodologies were conducted without considerations related to adversarial robustness. Achieving performance that doesn't significantly surpass these methods might not hold substantial meaning. Notably, the performance against strong attacks does not seem to indicate significant robustness. When proposing a methodology considering adversarial robustness, I believe that the proposed method should exhibit a performance better than random chance (10% for 10-class tasks) when subjected to strong attacks. The setting for strong attacks reflects the benchmark for invisible perturbations in the adversarial robustness research area. Approaches that fail to demonstrate significant performance improvements in this setting are arguably limited.

Author response:
**The proposed method should exhibit a performance better than random chance.**

We don’t believe it is fair to hold adversarial defense methods for condensed datasets to such standard. Many of the adversarial defense methods that work well under the real dataset setting do not apply in the condensed dataset setting. This is because one of the fundamental premises of dataset condensation is that no real data will be given to the downstream model at training time, which is important to preserve many of the properties of condensed datasets, including size, training efficiency, and privacy. Without real data, the data distribution that the model learns is already largely different compared to the test images, making it much easier for the adversary to find adversarial examples. Therefore, we don’t think it is realistic to expect the same degree of robustness from condensed datasets and real datasets. 

In addition, one reason for the low level of robustness is the inherent low level of clean accuracy. For example, many adversarial defense methods can retain a robustness of around 50%, on a model that can normally achieve 90% clean accuracy on CIFAR10. However, DC can only achieve 50% clean accuracy on CIFAR10, so it is not surprising that when applied the same level of attack, the robustness accuracy deteriorates severely. 

We are aware of the lack of previous research in this area. This is why in addition to comparison to vanilla dataset condensation methods, in the appendix we also showed a comparison between our results and those obtained from adversarial training, which is recognized as one of the more effective adversarial defense methods for standard datasets. This comparison demonstrates that our method yields superior results.

Reviewer comment (Evaluation):
Naturally, training methodologies considering adversarial robustness demand higher computational costs compared to standard training. Given that the proposed method appears to simultaneously utilize both the conventional method (gradient matching) and curvature regularization, it seems to require a relatively high computational cost (please correct me if I'm mistaken). However, the enhanced robustness performance doesn't seem to justify this computational overhead.

Author response:
**The enhanced robustness performance doesn’t seem to justify this computational overhead. Has there been an analysis regarding computational cost?**

Although we don’t have a formal analysis of the computational cost, it is very negligible. In fact, although the problem of robust dataset condensation is a tri-level optimization problem, we formulated our method in this exact way to reduce the potential computational cost by embedding the procedure in the bi-level optimization of dataset condensation.

You can look at our algorithm in the appendix. Our perturbation $z$ is calculated from the gradients from the real images, where the gradient is already calculated in the original DC algorithm. Then, we simply do another forward pass (without a backward pass) with real images plus $z$, and calculate the final regularization through the difference of the losses. This means the overall computational overhead is only a single forward pass per iteration.

Reviewer comment (Evaluation):
The proposed method shows a decline in performance on regular data compared to traditional methods. While a trade-off between robustness and accuracy can be expected, the drop in clean accuracy doesn't seem to be adequately compensated by an increase in robust accuracy.

Author response:
**The proposed method shows decline in performance on regular data.**

In fact, we show improvements to clean accuracy after applying our method in the appendix. (Our base condensation method is DC). While this increase in accuracy may be due to differences in experimental procedures, and is not an intended consequence of GUARD, it does show that there is no clear decrease in clean accuracy.

Reviewer comment (Evaluation):
Has there been an analysis or experiment regarding the computational cost, specifically on the overhead introduced compared to traditional methodologies?

Author response:
**The enhanced robustness performance doesn’t seem to justify this computational overhead. Has there been an analysis regarding computational cost?**

Although we don’t have a formal analysis of the computational cost, it is very negligible. In fact, although the problem of robust dataset condensation is a tri-level optimization problem, we formulated our method in this exact way to reduce the potential computational cost by embedding the procedure in the bi-level optimization of dataset condensation.

You can look at our algorithm in the appendix. Our perturbation $z$ is calculated from the gradients from the real images, where the gradient is already calculated in the original DC algorithm. Then, we simply do another forward pass (without a backward pass) with real images plus $z$, and calculate the final regularization through the difference of the losses. This means the overall computational overhead is only a single forward pass per iteration.

Reviewer comment (Evaluation):
Are there performance results under the general AutoAttack setting, which has the same epsilon upper bound as the strong attack?

Author response:
**Are there performance results under AutoAttack setting with the same epsilon as strong attack?**

Unfortunately we haven’t conducted experiments in that particular setting.

Reviewer comment (Experiments):
In the weaknesses section, drawbacks are highlighted using the standard strong attack as a reference. What are the author's thoughts on this? My knowledge in the dataset condensation field is limited; hence, I'm unaware if there have been studies focusing on adversarial robustness in this domain. Considering the methodologies compared in the paper, it appears there might not be. Therefore, it's challenging to judge the appropriateness of the paper's settings. Given that the model has been trained on limited data, I'd like to inquire whether it's suitable to compare the performances of methodologies under a weaker setting (epsilon=2), rather than the conventional adversarial settings employed in the vision domain.

Author response:
**The proposed method should exhibit a performance better than random chance.**

We don’t believe it is fair to hold adversarial defense methods for condensed datasets to such standard. Many of the adversarial defense methods that work well under the real dataset setting do not apply in the condensed dataset setting. This is because one of the fundamental premises of dataset condensation is that no real data will be given to the downstream model at training time, which is important to preserve many of the properties of condensed datasets, including size, training efficiency, and privacy. Without real data, the data distribution that the model learns is already largely different compared to the test images, making it much easier for the adversary to find adversarial examples. Therefore, we don’t think it is realistic to expect the same degree of robustness from condensed datasets and real datasets. 

In addition, one reason for the low level of robustness is the inherent low level of clean accuracy. For example, many adversarial defense methods can retain a robustness of around 50%, on a model that can normally achieve 90% clean accuracy on CIFAR10. However, DC can only achieve 50% clean accuracy on CIFAR10, so it is not surprising that when applied the same level of attack, the robustness accuracy deteriorates severely. 

We are aware of the lack of previous research in this area. This is why in addition to comparison to vanilla dataset condensation methods, in the appendix we also showed a comparison between our results and those obtained from adversarial training, which is recognized as one of the more effective adversarial defense methods for standard datasets. This comparison demonstrates that our method yields superior results.

Reviewer comment (Experiments):
Are there any other cases where adding curvature regularization to methods other than gradient matching leads to an improvement (not marginal) in robustness performance?

Author response:
**Are there any other cases where adding curvature regularization to methods other than gradient matching leads to an improvement?**

Yes! In our appendix, we presented results where we utilized SRe2L [1], a condensation method that performs better on high resolution datasets, and tested our method on ImageNette where it showed an improvement.

[1] Zeyuan Yin, Eric Xing, and Zhiqiang Shen. Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspective. Neurips 2023.

Reviewer comment (Evaluation):
In the experimental section, it is unclear what the improvement in adversarial robustness at the scale shown in Table 2 indicates, especially when considering the standard deviation. It seems to only weakly suggest that the proposed method may exhibit better consistency in terms of adversarial robustness, making it difficult to consider this experimental result as strong support for the author's conclusions.

Author response:
**The result seems to only weakly suggest that the proposed method may exhibit better consistency in terms of adversarial robustness.**

We do agree the results may seem weak, but improving adversarial robustness on condensed datasets is not a trivial task. Many of the adversarial defense methods that work well under the real dataset setting do not apply in the condensed dataset setting. This is because one of the fundamental premises of dataset condensation is that no real data will be given to the downstream model at training time, which is important to preserve many of the properties of condensed datasets, including size, training efficiency, and privacy. Without real data, the data distribution that the model learns is already largely different compared to the test images, making it much easier for the adversary to find adversarial examples. Therefore, we don’t think it is realistic to expect the same degree of robustness from condensed datasets and real datasets. 

In addition, one reason for the low level of robustness is the inherent low level of clean accuracy. For example, many adversarial defense methods can retain a robustness of around 50%, on a model that can normally achieve 90% clean accuracy on CIFAR10. However, DC can only achieve 50% clean accuracy on CIFAR10, so it is not surprising that when applied the same level of attack, the robustness accuracy deteriorates severely. 

In the appendix, we conducted a comparison between our results and those obtained from adversarial training, which is recognized as one of the more effective adversarial defense methods for standard datasets. This comparison demonstrates that our method yields superior results.

Reviewer comment (Theory):
Regarding the method, the author uses Taylor expansion to find the relationship between adversarial samples and the first and second-order information of samples under the model. However, they only briefly mention that "previous work has shown that regularizing gradients gives a false sense of security about the robustness of neural networks" and then solely rely on the curvature term as a regularization term—without further theoretical or experimental justification. This is considered unacceptable.

Author response:
**The authors only briefly mention that “previous work has shown that regularizing gradients give a false sense of security”**

Our proposition highlights that both gradients and curvature impact the upper bound of the robust loss function. We are aware of existing research indicating that gradient-based defenses can create a false sense of security and are vulnerable to certain attacks, so we conducted preliminary experiments that suggested gradient regularization generally underperforms compared to curvature regularization. Based on these findings, we chose to focus on curvature regularization in our paper.

Reviewer comment (Evaluation):
In the experimental section, there is a significant decrease in performance on clean samples after applying this method (as shown in the appendix). It is uncertain if this contradicts the statement in the paper that "in practice, it is important that the condensed data enable the model to perform well both in i.i.d test setting and under adversarial attacks," as it currently seems contradictory. Furthermore, these experimental results should be presented in the main body of the paper.

Author response:
**I am unsure if this contracts the statement that “it is important that the condensed data enable the model to perform well both in i.i.d test setting and under adversarial attacks**

The condensation method employed in our study is DC, and in the appendix we actually demonstrated improvements in clean accuracy after applying our method. Although this increase in accuracy might partially result from differences in experimental procedures, it importantly indicates that our approach does not lead to a clear decrease in clean accuracy.

Reviewer comment (Novelty):
Dataset Condensation (DC) was a recent technique that has not been recognized by many people. Whether it is promising to be used in the big data era is not clear. The authors proposed to study whether DC methods are adversarially robust. The motivation is very weak. I cannot see any significance in studying this topic.

Author response:
**DC was a recent technique that has not been recognized by many people.**

Dataset condensation has been proven of interest to the community by its popularity and the large body of related works. Our work is extremely relevant in this area, as several papers [1, 2] have discussed the potential application of dataset condensation in trustworthy machine learning (and hence why we study the topic of adversarial robustness).

[1] Zongxiong Chen, Jiahui Geng, Derui Zhu, Herbert Woisetschlaeger, Qing Li, Sonja Schimmler, Ruben Mayer, and Chunming Rong. A Comprehensive Study on Dataset Distillation: Performance, Privacy, Robustness and Fairness. arXiv preprint arXiv:2305.03355, 2023.

[2] Jiahui Geng, Zongxiong Chen, Yuandou Wang, Herbert Woisetschlaeger, Qing Li, Sonja Schimmler, Ruben Mayer, Zhiming Zhao, and Chunming Rong. A Survey on Dataset Distillation: Approaches, Applications, and Future Directions. arXiv preprint arXiv:2305.01975, 2023.

Reviewer comment (Theory):
Although overturned by the authors, the hypothesis "adversarial robustness might be inherently induced by the dataset condensation process" does not make much sense. It is not clear why DC can be linked with the previous work that suggested the adversarial vulnerability of models is often tied to these non-observable features since the authors did not study what type of features are discarded by DC methods. Also, the goal of DC seems to reduce training samples other than discarding non-observable features. The connection between these two types of work seems to be hallucinated without any evidence.

Author response:
**The hypothesis does not make much sense**

The hypothesis is based on the premise that synthesizing a smaller training dataset from a larger one inherently involves discarding some information and features. We believe this selective reduction is a necessary part of the data condensation process.

Reviewer comment (Novelty):
The novelty of the proposed method is not clear. Curvature Regularization has been considered by (Moosavi-Dezfooli et al. 2019). The technical novelty is limited.

Author response:
**The novelty of the proposed method is not clear.** 

GUARD is not just a simple combination of the Dataset Condensation with Gradient Matching (DC) and the CURE method. Unlike the CURE regularizer, our method embeds the regularization into the condensed dataset, allowing for no decrease in actual training time efficiency.

That being said, our work serves as an important foundation for robust dataset condensation and is much more than just the proposed GUARD framework. Not only did we provide a formulation of robust dataset condensation and a comprehensive robustness benchmark of SOTA methods, but we are also the first to theoretically prove that minimizing the adversarial loss of the condensed dataset and minimizing the adversarial loss of the real dataset is only different by a constant factor, which is very useful for future works in this area.

Reviewer comment (Experiments):
The experiment is weak. Only conduct on MNIST and CIFAR10. Only evaluate ConvNet with 3 layers.

Author response:
**The Experiment is Weak.**

Our experiments in the main paper focused on MNIST and CIFAR10 because we attached the regularizer to DC, which is notably less effective with high-resolution datasets. However, in the appendix, we extended our experimentation using SRe2L [3], a more suitable condensation method for high-resolution datasets. This allowed us to test our method on ImageNette and present these results for a more comprehensive evaluation

[3] Zeyuan Yin, Eric Xing, and Zhiqiang Shen. Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspective. Neurips 2023.

Reviewer comment (Theory):
The theoretical part (section 5) is not sound enough.

Author response:
Regrettably, due to space limitations in the main paper, we had to place the bulk of proof in the appendix. Please view the appendix for a more formal proof of the proposition.

Reviewer comment (Novelty):
The reference is not comprehensive. The main idea is to match the gradient which tends to have smaller curvature (sharpness). There are many existing literatures on how to reach flatter minima, e.g., sharpness aware minimization, implicit regularization. Though these works are not about condensaton, some of them might have relations to adversarial robustness (like arXiv:2305.05392). Also, there are other works also trying to give such adversarially robust condensation like arXiv:2211.10752, but they are neither mentioned and nor compared.

Author response:
Thank you for suggesting papers regarding sharpness aware minimization and other related topics. We agree that it relates to our topic of study, and have included them in our paper.

Reviewer comment (Writing):
The definition of $h$ is missing in Proposition 1.

Author response:
h is a feature extractor, which was defined in the formal proof in our appendix. We have added the definition to the proposition for clarity.

Reviewer comment (Theory):
Why is $\tilde \ell(x,v)$ convex in terms of $x$ in the appendix A given that $\ell(x)$ is convex? It's obvious that $\tilde\ell(x,\cdot)$ is convex but $\tilde\ell(\cdot,v)$ might not be convex. Say $\ell(x)=-x^3+3x^2$ and hence $\ell(x)$ is convex in $[0,1]$. The corresponding quadratic approximation $\tilde\ell(x,v)$ has second order derivative on $\frac{\partial^2\tilde\ell(x,v)}{\partial x^2}=-6x$ is non-positive in $[0,1]$, which implies that it is non-convex in $[0,1].$

Author response:
The first term, $\ell(x)$, is convex by assumption. The second term, $\nabla \ell(x)^{\top} v$, is a linear function in $v$ and inherently convex. The third term, $\frac{1}{2} v^{\top} H v$, is a quadratic form. Given that $H$m the Hessian of a convex function $\ell(x)$, is positive semi-definite, the quadratic form is also convex in $v$. As the sum of convex functions retain convexity, the collective convexity of all three terms ensures that $\tilde{\ell}(x+v)$ is convex.

Reviewer comment (Theory):
I think eq(12) in the appendix A requires $h^{-1}(\cdot)$ to be a contraction map, why is that true? If it is true, then $\left\|h\left(x^{\prime}\right)-\mathbb{E}_{x \sim D_c}[h(x)]\right\| \leq \sigma$ basically implies that the distance between the $x'$ and the mean of training samples with label $c$ is no larger than $\sigma$. Is it a reasonable assumption?

Author response:
The assumption in eq(12) is adopted from a specific dataset condensation method outlined in [1]. This method specifically optimizes the synthetic dataset to minimize the distance between the distribution of the real and synthetic data.  

[1] Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In IEEE/CVF Winter Conference on Applications of Computer Vision, 2023.