Paper: Self-Paced Augmentations (SPAug) for Improving Model Robustness (ICLR-2024)

Reviewer comment (Evaluation):
In Section 4.3, you mentioned that the learnable SPAug has a significant improvement effect on the performance of AugMix in processing corrupted data. However, in the above Table 2, compared to the optimal model, your performance improvement is very limited or there is no improvement at all. The superiority of the model is not sufficiently reflected.

Author response:
Thank you for your comment! After reviewing your feedback, we realized that the performance improvement of SPAug over AugMix may not have been adequately highlighted. The tables below provides a summary of the performance enhancements achieved by incorporating SPAug with the AugMix policy.

| Dataset   | JSD? | AugMix          | AugMix+SPAug (Ours) | % Improvement |
|-----------|------|-----------------|----------------------|--------------|
| CIFAR-10  | No   | $14.0\pm{0.4}$  | $13.0\pm{0.2}$       | +1.0         |
| CIFAR-10  | Yes  | $11.2\pm{0.3}$  | $11.0\pm{0.2}$       | +0.2         |

Table 1: Comparison of SPAug+AugMix vs. AugMix in terms of mean corruption error on CIFAR-10-C with WRN-40-2 backbone.

| Dataset     | JSD? | AugMix          | AugMix+SPAug (Ours) | % Improvement |
|-------------|------|-----------------|----------------------|--------------|
| CIFAR-100   | No   | $40.0\pm{0.1}$  | $39.0\pm{0.1}$       | +1.0         |
| CIFAR-100   | Yes  | $36.1\pm{0.1}$  | $35.0\pm{0.1}$       | +1.1         |

Table 2: Comparison of SPAug+AugMix vs. AugMix in terms of mean corruption error on CIFAR-100-C with WRN-40-2 backbone.

| Dataset   | JSD? | AugMix          | AugMix+SPAug (Ours) | % Improvement |
|-----------|------|-----------------|----------------------|--------------|
| CIFAR-10  | No   | $10.7\pm{0.2}$  | $10.0\pm{0.1}$       | +0.7         |
| CIFAR-10  | Yes  | $9.3\pm{0.0}$   | $8.7\pm{0.0}$        | +0.6         |

Table 3: Comparison of SPAug+AugMix vs. AugMix in terms of mean corruption error on CIFAR-10-C with WRN-28-10 backbone.

| Dataset     | JSD? | AugMix          | AugMix+SPAug (Ours) | % Improvement |
|-------------|------|-----------------|----------------------|--------------|
| CIFAR-100   | No   | $33.6\pm{0.3}$  | $33.3\pm{0.0}$       | +0.3         |
| CIFAR-100   | Yes  | $31.9\pm{0.2}$  | $31.3\pm{0.2}$       | +0.6         |

Table 4: Comparison of SPAug+AugMix vs. AugMix in terms of mean corruption error on CIFAR-100-C with WRN-28-10 backbone.


As demonstrated in the tables above, it is evident that coupling SPAug with AugMix consistently results in a significant improvement in performance on both CIFAR-10 and CIFAR-100, whether using the WRN-40-2 or WRN-28-10 backbone. This further underscores the benefit of employing instance-specific augmentation parameters over uniform augmentation parameters.

Reviewer comment (Experiments):
You did not discuss the threshold τ in the subsequent experiments. It is not clear how you optimized the threshold. It feels like you are showing the best experimental results that you got separately with τ=0.1 or τ=0.2.

Author response:
Thank you for the comment. 

For each dataset and the backbone architecture, we performed a hyperparamter tuning with $\tau$ values of 0.0, 0.1, 0.25, 0.5, 0.75, 0.9, and 1.0. The presented results in the paper are for the optimal configuration of $\tau$.

It was observed that, in most cases, a $\tau$ value of 0.75 or 0.9 yielded the best results. Also, SPAug improves over the baseline for a wide range of choices for $\tau$ demonstrating it's robustness to this hyper-parameter as it can be seen in Tables 3, and 4 in the supplementary material.

Reviewer comment (Experiments):
The proposed augmentation, as illustrated in Eq.1, shares a similar formulation as two widely used augmentation methods: CutMix and Mixup. However, there is no comparison between the proposed method and CutMix/Mixup.

Author response:
Thank you for the comment. Although the proposed approach may appear similar to CutMix and MixUp at first glance, they are fundamentally different from our SPAug. The primary distinction lies in the fact that CutMix and MixUp fall under a uniform augmentation policy. In other words, the hyperparameters for CutMix and MixUp are fixed for all samples in the dataset, resulting in uniform augmentation intensity applied to all samples. In contrast, SPAug introduces sample-dependent augmentation intensity during training, tailored to how each sample converges. Samples that converge rapidly (referred to as "easy samples") are encouraged to undergo higher levels of augmentation intensity, while those converging slowly (referred to as "hard samples") are encouraged to train with less synthetic augmentation applied. Moreover, instead of employing computationally expensive optimization processes such as reinforcement learning or bi-level optimization, we draw inspiration from the curriculum learning literature and propose to use each sample's training loss (cross-entropy loss) as a proxy measure to determine their ease or difficulty of convergence.

However, in response to the reviewer's feedback, we have decided to compare our SPAug method with CutMix and MixUp to provide a more comprehensive analysis. Following your comment, we have conducted the comparison as shown in Table 1 and Table 2 (below). 

| Standard | CutOut | MixUp | CutMix | AA | AugMix | AugMix+SPAugLearnable |
|----------|--------|-------|--------|----|--------|-----------------------|
| 27.2     | 26.8   | 22.3  | 27.1   | 23.9 | 11.2   | 11.0                  |

Table 1: Comparison of corrupted Test Error (C-Err.) of SPAug with AugMix policy on CIFAR-10-C with WRN-40-2.



| Standard | CutOut | MixUp | CutMix | AA | AugMix | AugMix+SPAugLearnable |
|----------|--------|-------|--------|----|--------|-----------------------|
| 53.2     | 53.5   | 50.4  | 52.9   | 49.6 | 36.1   | 35.0                  |

Table 2: Comparison of corrupted Test Error (C-Err.) of SPAug with AugMix policy on CIFAR-100-C with WRN-40-2.

As shown in Table 1 and Table 2, the augmentation policies CutOut, MixUp, and CutMix result in significantly higher corrupted test error rates (C-Err.) than Augmix+SPAugLearnable. This indicates that models trained with those augmentation policies have lower generalizability against common corruptions.

In regards to demonstrating results on ImageNet, we have done some preliminary comparisons in Table 3 of the main body. We will extend experiments on this dataset in future revisions.

Reviewer comment (Experiments):
There are also some related works [a, b], which also adjust the augmentation strength according to other training statistics. They are not discussed and compared in experiments.  

[a] Universal Adaptive Data Augmentation  
[b] ADAPTIVE DATA AUGMENTATION FOR IMAGE CLASSIFICATION

Author response:
Thank you for bringing our attention to these related works. However, we would like to clarify the differences between the proposed SPAug and two other related methods, Universal Adaptive Data Augmentation (UADA) [1] and Adaptive Data Augmentation [2].

When comparing our work to UADA, the main distinction lies in the fact that, although UADA adjusts the augmentation parameters adaptively over iterations, it maintains uniform parameters for all data samples in the dataset. In other words, UADA's goal is to change (or adapt) the augmentation parameters based on the model's gradient information during training. In contrast, our objective is to have instance-dependent augmentation parameters that change based on how each sample converges during training. We will discuss UADA in the related work section, highlighting this distinction between SPAug and UADA. However, it's worth noting that UADA does not provide a comparison of their performance on corrupted test sets (such as CIFAR-10-C and CIFAR-100-C), which currently prevents us from making a direct comparison between UADA and SPAug.

The main intuition behind the Adaptive Data Augmentation framework [2] is to find a small transformation that results in the maximum classification loss, leading to training with worst-case data augmentation. However, their proposed worst-case augmentation framework only works with translational augmentations and cannot be directly extended to more complex augmentation operations, such as color transformations or mixing. Furthermore, they have validated their results on very small datasets, such as MNIST-500 and Small-NORB, which prevents us from making a direct comparison with their work. Nevertheless, we will discuss this work in the Related Work section since it represents one of the early attempts to demonstrate the importance of adaptive data augmentations.

Reviewer comment (Experiments):
The method is not compared directly with other reported results and the provided baselines seems to be weak, making results not accurate. For instance, in RA the error reported on CIFAR10/100 for Wide-ResNet 28-10 are respectively 2.7 and 16.7, while in the proposed paper are 3.3 and 19.1.

Author response:
We thank you for raising this point. We should clarify that the difference lies in the fact that the authors of the RA paper, apply cutout after RandAugment in the results they report. In our experiments, since our goal was to directly compare with RandAugment, we did not apply cutout to the baseline (i.e., RA) and to SPAug+RandAug. We will make this clear in the next revision and also include the comparison to RandAugment + Cutout as well.

Reviewer comment (Experiments):
Due to the stochastic nature of the augmentations, the corresponding loss for a given sample can fluctuate and introduce a certain noise on the selection of the easy/hard samples. Did you find any instabilities in the training due to this noise?

Author response:
Thank you for your comment. No, we did not observe any training instabilities during the training. 

For example, **Figure 7 in Supplementary Document** shows CE loss when training WRN-40-2 backbone with AugMix policy w/ and w/o SPAug. As can be seen, there is no training instabilities observed.

Reviewer comment (Experiments):
Why you train with a limited budget instead of training until convergence? This makes results not compareble with the state of the art.

Author response:
Thank you for your comment! The reason we show results for both 100 epochs and 200 epochs (standard training epochs used in most previous works) is to demonstrate that when SPAug is included, the network tends to converge faster than with a uniform augmentation policy. As observed by another reviewer, uniform augmentation policies require more time than standard training to fully converge. For example, when considering the AA policy on CIFAR100 with a WRN-28-10 backbone, training for 200 epochs improves performance by +0.9\% (40.5\% to 39.6\%) compared to training for 100 epochs. However, if we incorporate SPAug into AA, the performance improves by +0.4\% (39.3\% to 38.9\%) because even with 100 epochs, we can see that the network has converged to a better optimum than with the uniform policy. As we emphasized in the introduction, our goal in incorporating instance-dependent augmentation parameters is primarily to enhance convergence, which is why we report results for both the standard training epochs and half of the standard training epochs.

Reviewer comment (Experiments):
What is the reason to propose the toy experiment in section 4.2. The following evaluations are also performed on the same dataset.

Author response:
Thank you for the comment! The reason for the toy experiment in Section 4.2 is to demonstrate how SPAug improves performance with a hand-crafted augmentation policy. Typically, when we begin training a neural network, we have no prior knowledge of the best augmentation policy for a given task. For instance, we might start with very basic augmentations like random resized crop and flip, then gradually add color augmentations such as random intensity, hue, saturation, etc., and even incorporate perspective transformations. Since the optimal hyperparameters for these augmentations are often unknown or only partially determined through hyperparameter tuning, the toy experiment suggests that adding SPAug on top of such suboptimal augmentation parameters still leads to performance improvement. The remaining results presented in the paper aim to demonstrate that SPAug can even enhance performance compared to well-optimized augmentation policies previously established for each dataset.

Reviewer comment (Reproducibility):
What is the value of the thresholds $\tau$ for the experiments in Table 2?

Author response:
Thank you for the comment. The values reported in the paper for the AugMix + SPAug policy with WRN-40-2 and WRN-28-10 backbones use threshold values of 0.75 or 0.9. We conducted hyperparameter tuning for $\tau$ with values ranging from 0.1 to 1.0. It was observed that, in most cases, a $\tau$ value of 0.75 or 0.9 yielded the best results. Also, SPAug improves over the baseline for a wide range of choices for $\tau$ demonstrating it's robustness to this hyper-parameter as it can be seen in Tables 3, and 4 in the supplementary material.

Reviewer comment (Novelty):
The authors did not explain the difference of their approach from AugMax [1] framework, which combines augmented images with adversarially calculated weights. Similarly, Hou et al. [2] adopted the idea of curriculum learning and applied it to decide when to augment data during training, but this is also not discussed by the authors. Other augmentation techniques being sota, PRIME [4] and TrivialAugment [5], are not compared with.

Author response:
We thank the reviewer for pointing out these interesting related works [1-2]. While we recognize that the high-level idea of AugMax, MADAug, and SPAug is to have instance specific augmentation parameters, our approach of addressing this problem fundamentally differs from others.

*Answer to Q1*
> How is the work different from [1,2] and what improvements have been made regarding them? This work is quite similar to AugMax[1] in the sense of combining augmented images with their original version using weights calculated by back-propagation.

When considering AugMax alongside SPAug, we would like to emphasize the following differences:
- In AugMax, these instance-specific parameters are determined in an adversarial manner. Hence, during training, each sample is trained with its \textbf{maximum} augmentation level. In contrast, SPAug determines these parameters based on each sample's Cross-Entropy (CE) loss at the previous epoch.
- Since AugMax determines the instance-specific parameters in an adversarial manner, for each sample, at each training iteration, they need to perform a potentially expensive adversarial attack, which adds computational overhead. In contrast, SPAug uses the sample's previous epoch loss as the proxy value, hence there is little to no computational overhead over standard training.
- In addition to the above two main differences, AugMax introduces changes to the network architecture to boost its performance. Since, in AugMax, samples are trained with their maximum augmentations, it leads to a deviation of distribution between augmented and input data, which makes the training more challenging, as noted in the paper. To address this issue, they use a different normalization technique termed DuBIN, to disentangle the instance-wise feature heterogeneity of AugMax samples. Since SPAug controls augmentation intensity based on its previous loss values, the augmented views won't deviate much from the original training distribution as in AugMax. In contrast, our method (SPAug) does not require additional modifications to network architecture or the training setup and can be easily integrated to existing training pipelines. When we use the same basic WRN40-2 architecture (with regular 2D batchnorm) that we use in our paper, we find that on CIFAR-100 AugMax has 36.6\% corrupted error (which is 1.6\% worst than that of SPAug) . In the next revision, we will include this comparison.

When considering the second work (MADAug), we want to draw the reviewer's attention to the fact that it was posted on arXiv on Sat, 9 Sep 2023, which is very close to the ICLR submission date and can be considered as a concurrent related work. Similar to SPAug, MADAug also proposes having instance-wise augmentation parameters, further demonstrating the recent trend in this direction. However, after carefully reviewing their approach, we have observed the following fundamental differences with our work:
- They also acknowledge that at the beginning of training, samples are not well-fitted; hence applying the same level of augmentations at that phase reduces the final accuracy. Therefore, they propose to use a manually designed probability schedule $p(t) = \tanh(t/\tau)$, where $t$ denotes the current training epoch. In addition, MADAug assigns an augmentation probability $p$ and magnitude $\lambda$ to each sample, determined by the policy network parameterized by $\theta$, which takes the image features extracted from the task model parameterized by $w$. The bi-level optimization process consists of three steps:
   1. One-step gradient descent on $w_t$ to achieve a closed-form surrogate $\hat{w}$ of the lower-level problem solution.
   2. Updating policy network parameters $\theta_t$ by minimizing the validation loss computed by the meta-task model $\hat{w}t$ on a mini-batch of the validation set.
   3. Updating the main model parameters $w$ based on the parameter $\theta_{t+1}$ of the policy model.

Due to this bi-level optimization, MADAug requires 2 forward passes for each training minibatch as well as forward passes on the validation minibatch, hence taking approximately $1.5\times$ longer training time per epoch compared to our method when training Wide-ResNet-28-10 on the CIFAR-10 dataset.

Reviewer comment (Experiments):
The authors only show results of SPAug combined with AugMix on ImageNet, while the results of it combined with AutoAugment and RandomAugment are not provided.

Author response:
Thank you for the comment. Following the previous works, we limited our backbones to ResNets and Wide-ResNet (WRN) architectures in the main body. We also included AllConvNet, DenseNet architectures as well in Table 2 of the **supplementary material**.) Similarly, following some of the literature, we have included ImageNet results in Table 3 on the ResNet-50 architecture.  However, we see the value of your comment, and we will try compare the results of SPAug with ViT architectures on ImageNet in the next revision, as it requires longer training time and some time to set up the experiments.

Reviewer comment (Experiments):
Only a single architecture is tested; it is unclear how this method would perform with transformer training strategies.

Author response:
Thank you for the comment. Following the previous works, we limited our backbones to ResNets and Wide-ResNet (WRN) architectures in the main body. We also included AllConvNet, DenseNet architectures as well in Table 2 of the **supplementary material**.) Similarly, following some of the literature, we have included ImageNet results in Table 3 on the ResNet-50 architecture.  However, we see the value of your comment, and we will try compare the results of SPAug with ViT architectures on ImageNet in the next revision, as it requires longer training time and some time to set up the experiments.