Paper: DualAug: Exploiting Additional Heavy Augmentation with OOD Data Rejection (ICLR-2024)

Reviewer comment (Experiments):
While this paper demonstrates performance improvements, its applicability is primarily limited to classification tasks, including contrastive learning. As previously mentioned in the strengths section, given the extensive history of AutoAugmentation research, the room for improvement in these specific domains appears limited. Therefore, it is essential to explore the impact of this research on other datasets. For instance, leveraging additional datasets like those used in the AutoAugmentation [1] paper (Flowers, Caltech, Pets, Aircraft, Cars) could provide further insights. Moreover, the proposed method, relying on softmax scores for out-of-distribution sample detection, may require refinement when applied to other tasks, such as object detection (as DADA [2] did).

Author response:
Thank you for your constructive and insightful reviews. The following is our response.

**Weakness1: More dataset results of DualAug **

We present more results on Flowers, Caltech, Pets, Aircraft, Cars as below tables. The experiment uses the inception-v4 model. DualAug demonstrates a significant improvement over AutoAugment in almost all FGVC datasets except for Pets.
| Method | Flowers  | Caltech | Pets | Aircraft | Cars | Avg. |
| :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| AA | 87.55% | 81.76% | **84.52%** | 85.63% | 93.51% | 86.59% |
| AA+DualAug | **88.92%** | **84.85%** | 83.70% | **86.74%** | **94.01%** | **87.64%** |

Reviewer comment (Evaluation):
Despite the paper's assertion that the proposed algorithm incurs minimal computational costs by utilizing an additional branch for feed forwarding to obtain heavy and basic augmentation datasets, the authors do not provide an analysis of the algorithm's additional computational expenditure. For instance, it would be valuable to compare this algorithm's cost-effectiveness with that of existing methods like RA, which do not incur extra costs. Such an analysis could strengthen the argument for the proposed algorithm's efficacy.

Author response:
**Weakness 2: Additional computational expenditure.**

We conduct experiments to ensure that the training costs are aligned. We increase RA's training time to ensure a fair comparison, calling it RA*. The results in the table below clearly demonstrate that RA+DualAug outperforms the RA* and RA. The experiment is conducted using WRN-28-2 on CIFAR-100. 

It is important to note that the policy search cost for RA, AA, and other automated augmentation methods is quite expensive, despite not incurring extra costs in training (as reported in the table below and DeepAA's Table 4). Therefore, the increase in computational cost is slight and reasonable when compared to the significant performance improvement that DualAug provides.

| Method | Accuracy(%) | Search Time(h) | Train Time(h) |
| --- | :---: | :---: | :---: |
| RA | 77.94±.15 | 25 | 1 |
| RA* | 78.11±.13 | 25 | 1.5 |
| RA+DualAug | **78.46±.12** | 0 | 1.5 |

Reviewer comment (Novelty):
(Lack of reference to related work) The paper does not reference two relevant works: DADA [2], which focuses on Data Augmentation using Differentiability, and CUDA [3], which provides an analysis of the number of augmentation operations in both class imbalance and balanced tasks.

Author response:
**Weakness 4: Lack of reference to related work**

We apologize for the negligence. We have now added the DADA and CUDA to references in the revision.

Reviewer comment (Evaluation):
Is it possible to explore alternative metrics for the OOD score, such as the Mahalanobis distance [4]? While the Softmax-based approach is effective, there may be even better scoring methods worth considering.

Author response:
**Question 1: Explore alternative metrics for the OOD score.**

From a theoretical perspective, it appears that refining the metrics for the OOD score could improve the performance of DualAug. However, when integrating it into DualAug, the trade-off between computational expenses of OOD score calculation and the gain of performance must be carefully weighed within the context of the specific task.  

Considering Mahalanobis distance in [1] is quite complex. We attempt to use energy-base model (EBS)[2] as an alternative OOD Score. EBM may have a slight edge over softmax. The experiment is conducted using WRN-40-2 on CIFAR-100. 

| Method | Accuracy(%) |
| --- | :---: |
| AA+DualAug(SoftMax) | 79.83±.19 |
| AA+DualAug(EBM) | 79.95±.21 |

[1] A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks, NeurIPS 2018

[2] Energy-based Out-of-distribution Detection, NeurIPS 2020

Reviewer comment (Experiments):
Could the authors extend their analysis to encompass other tasks, such as object detection?

Author response:
**Weakness 1 & Question 2: Other tasks of DualAug**

DualAug can also be extended to other tasks, such as object detection. We can analyze its effectiveness in two phases: the pre-training phase and the finetuning phase.
In the pre-training phase, we have observed that DADA has been used to improve the ImageNet pre-training backbone for object detection. Our pre-training backbone can also be leveraged for object detection and other related tasks. When we expand the results of Table 5 to the object detection task, we find that DualAug leads to improvements in performance.

| Method | Classification(Acc. %) | VOC 07+12 Detection(AP) |
| --- | --- | --- |
| Simsiam | 68.20% | 51.67 |
| Simsiam+DualAug | **68.67%** | **52.02** |

In the finetuning phase, when DualAug is applied to a specific task, its softmax score can also be utilized for OOD detection. Following [1], we conduct an experiment to demonstrate the performance of DualAug in object detection.

| Method | VOC 07 Detection(AP) |
| :---: | :---: |
| AA[1] | 56.30 |
| AA[1]+DualAug | **56.73** |

Expanding DualAug to encompass a wider range of tasks is an exciting aspect of future work. We believe it will lead to further exploration and advancement in the field.

[1] Learning Data Augmentation Strategies for Object Detection ECCV 2020

Reviewer comment (Experiments):
Is there potential for the proposed method to be integrated with mixed sample data augmentation techniques like MixUp [5], CutMix [6], or similar approaches?

Author response:
**Question 3: Integrate DualAug with mixed sample data augmentation techniques**

We use Mixup as an example to analyze the integration of DualAug with mixed-base augmentation. The experiment is conducted using ResNet-18 on CIFAR-10. Upon observation, it appears that mixup tends to select $\alpha<=1$ in $Beta(α, α)$  which may not provide an enough amplitude of augmentation. To address this, we can simply set $\alpha>1$ and utilize DualAug to filter OOD data in mixup data. The table below demonstrates that we achieve some improvement compared to standard mixup. 

| Method | Accuracy(%) |
| --- | :---: |
| Mixup($\alpha=1.0$) | 95.64±.11 |
| Mixup($\alpha=10.0$) | 94.67±.14 |
| Mixup($\alpha=10.0$)+DualAug | **95.78±.09** |

We would like to extend our sincere appreciation once again for your valuable reviews, which have greatly improved the quality of our manuscript. We look forward to your continued discussions.

Reviewer comment (Experiments):
Threshold Selection for OOD Detection: The paper utilizes the 3σ rule of thumb to set the threshold for filtering out OOD data. While this is a straightforward approach, it may not be the most optimal one in practice. The choice of threshold values in OOD detection can be crucial, and it's unclear whether the 3σ rule is universally suitable for different datasets and models. A more robust method for threshold selection would enhance the reliability of the approach.

Author response:
**Weakness 1: Threshold Selection for OOD Detection**

In almost all tasks, datasets, and models, we have observed that the 3-sigma rule **consistently yields improvements**, and we do not deliberately adjust  λ. 

In addition, it is a common practice to set a threshold for identifying ID/OOD samples manually in the field of OOD detection, as noted in [1]. The 3-sigma rule offers an adaptive method to dynamically adjust this threshold.

[1] A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks ICLR 2017

Reviewer comment (Novelty):
Marginal improvement over existing Methods: One aspect of the paper that requires attention is the relatively marginal improvement in performance demonstrated by the DualAug framework, especially when compared to previous works in data augmentation. While the paper presents itself as an effective approach to addressing the challenges of data augmentation, the magnitude of the performance gains achieved is not particularly substantial.

Author response:
** Weakness 2 & Question: Marginal improvement over existing Methods **

The table below illustrates the performance of the main data augmentation methods on CIFAR-100, WRN-28-10 in recent years. The improvement achieved by DualAug is reasonable, compared to the previous methods.
| Method | Accuracy(%) | $\Delta$ |
| :---: | :---: | :---: |
| AutoAugment (2018) | 82.90 | - |
| RandAugment (2019) | 83.30 | 0.40 |
| TrivialAugment (2021) | 83.54 | 0.24 |
| DeepAA (2022) | 84.02 | 0.48 |
| DeepAA+DualAug (2023) | **84.39** | 0.37 |


Moreover, DualAug's **stable performance gain** is quite noticeable as observed by the reviewer. DualAug provides consistent improvement in different datasets, various baseline data augmentations (AA, RA, and DeepAA), and tasks (classification, semi-supervised learning, and self-supervised learning). Our extra experiment on more datasets in Response to Reviewer c5kH (1/7) also proves that DualAug is competitive.

Thanks once more for the valuable feedback and insightful review you provided. I anticipate engaging in further discussions with you.

Reviewer comment (Novelty):
The idea is simple and clear which I like. However, the effectiveness is under question because the gains are very marginal. Hence making me question the value of this work to the research community.

Author response:
** Weakness 2 & Question: Marginal improvement over existing Methods **

The table below illustrates the performance of the main data augmentation methods on CIFAR-100, WRN-28-10 in recent years. The improvement achieved by DualAug is reasonable, compared to the previous methods.
| Method | Accuracy(%) | $\Delta$ |
| :---: | :---: | :---: |
| AutoAugment (2018) | 82.90 | - |
| RandAugment (2019) | 83.30 | 0.40 |
| TrivialAugment (2021) | 83.54 | 0.24 |
| DeepAA (2022) | 84.02 | 0.48 |
| DeepAA+DualAug (2023) | **84.39** | 0.37 |


Moreover, DualAug's **stable performance gain** is quite noticeable as observed by the reviewer. DualAug provides consistent improvement in different datasets, various baseline data augmentations (AA, RA, and DeepAA), and tasks (classification, semi-supervised learning, and self-supervised learning). Our extra experiment on more datasets in Response to Reviewer c5kH (1/7) also proves that DualAug is competitive.

Thanks once more for the valuable feedback and insightful review you provided. I anticipate engaging in further discussions with you.

Reviewer comment (Experiments):
The evaluation seems missing some important evaluations on similar approaches: a) it is missing a comparison with [1], which seems very similar in aim and has also results on the same datasets. From table 2 in [1], results are actually a bit better in [1], which is a paper from 2020.

Author response:
Thank you for your constructive and insightful reviews. The following is our response.

**Weakness 1 & Question 1: Comparison and Discussion with TeachAugment and KDforAA[1].**

**KDforAA[1]**: We present a fair comparison between DualAug and KDforAA[1], as shown in the table below. The experiments are based on AutoAugment and use the WRN-28-10 model. To ensure fairness, the settings and Memory/Time cost are aligned with those of KDforAA[1]. DualAug demonstrates better performance compared to KDforAA[1].
| Dataset  | KDforAA* | AA + DualAug |
| :---: | :---: | :---: |
| CIFAR-10 | 97.6 | **97.63±.09** |
| CIFAR-100 | 83.8 | **83.94±.26** |

\* : Reported in KDforAA[1].

[1] Wei, Longhui, et al. "Circumventing outliers of autoaugment with knowledge distillation." ECCV, 2020.

**TeachAugment**: We provide a fair comparison between TeachAugment and DualAug in the same training pipeline, as shown in the table below. The experiments use the WRN-28-10 model. To ensure fairness, we use the EMA model to filter OOD data in Adv. Aug+DualAug, as done in TeachAugment.

| Dataset | TeachAug | Adv. Aug+DualAug | DeepAA+DualAug |
| :---: | :---: | :---: | :---: |
| CIFAR-10 | 97.25%±.11% | 97.35%±.09% | **97.52%±.18%** |
| CIFAR-100 | 83.08%±.39% | 83.02%±.34% | **84.54%±.23%** |

TeachAug and DualAug present comparable performance in the fair setting, using the Adv. Aug as the baseline. Furthermore, compared to TeachAug, we are able to improve upon a more widely automated augmentation method(DeepAA+DualAug), resulting in better performance in the same training pipeline. It is also worth noting that DualAug exhibits better performance in comparedison to TeachAug in self-supervised learning task, as shown in Table 5 (68.67% vs. 68.20%).

Reviewer comment (Experiments):
The proposed method is quite similar to TeachAugment, as reported by the authors, but the comparison is performed only on ImageNet (table 2) and without the same training. The improvement could be due to a better training pipeline. The authors should train TeachAugment on their pipeline, making sure that there are no differences in the training other than the method.

Author response:
Thank you for your constructive and insightful reviews. The following is our response.

**Weakness 1 & Question 1: Comparison and Discussion with TeachAugment and KDforAA[1].**

**KDforAA[1]**: We present a fair comparison between DualAug and KDforAA[1], as shown in the table below. The experiments are based on AutoAugment and use the WRN-28-10 model. To ensure fairness, the settings and Memory/Time cost are aligned with those of KDforAA[1]. DualAug demonstrates better performance compared to KDforAA[1].
| Dataset  | KDforAA* | AA + DualAug |
| :---: | :---: | :---: |
| CIFAR-10 | 97.6 | **97.63±.09** |
| CIFAR-100 | 83.8 | **83.94±.26** |

\* : Reported in KDforAA[1].

[1] Wei, Longhui, et al. "Circumventing outliers of autoaugment with knowledge distillation." ECCV, 2020.

**TeachAugment**: We provide a fair comparison between TeachAugment and DualAug in the same training pipeline, as shown in the table below. The experiments use the WRN-28-10 model. To ensure fairness, we use the EMA model to filter OOD data in Adv. Aug+DualAug, as done in TeachAugment.

| Dataset | TeachAug | Adv. Aug+DualAug | DeepAA+DualAug |
| :---: | :---: | :---: | :---: |
| CIFAR-10 | 97.25%±.11% | 97.35%±.09% | **97.52%±.18%** |
| CIFAR-100 | 83.08%±.39% | 83.02%±.34% | **84.54%±.23%** |

TeachAug and DualAug present comparable performance in the fair setting, using the Adv. Aug as the baseline. Furthermore, compared to TeachAug, we are able to improve upon a more widely automated augmentation method(DeepAA+DualAug), resulting in better performance in the same training pipeline. It is also worth noting that DualAug exhibits better performance in comparedison to TeachAug in self-supervised learning task, as shown in Table 5 (68.67% vs. 68.20%).

Reviewer comment (Experiments):
The method seems to have quite some hyper-parameters such as \labda, M, warm-up phase, which makes it more complex for a real deployment.

Author response:
**Weakness 2: Hyper-parameters of DualAug** 

The hyper-parameters of DualAug have shown **consistent performance** across a wide range of experiments. This includes λ for the 3σ rule, the upper limit of M, and the warm-up phase. In Sections 4.1 Supervised Learning, 4.2 Semi-Supervised Learning, and 4.3 Contrastive Self-Supervised  Learning, these parameters are **consistently** set at 1.0, 10, and 20%, with the exception of ImageNet classification. Additionally, Figure 5 illustrates their consistent performance across different settings.

Reviewer comment (Experiments):
There is no clear comparison with other methods in terms of computation cost. For instance, in my understanding this approach has a higher computational complexity as TeachAugment, as it requires to evaluate both basic and heavy transformation for each sample.

Author response:
**Weakness 3 & Question 2: Computational cost of the proposed approach and other methods.**

We present a comparison of computation costs in the table below. The experiment is carried out on CIFAR-100 using a single 2080Ti GPU with WRN-28-10 model. As shown, when considering memory cost, DualAug uses the same memory cost as AA, while TeachAug and KDforAA require additional memory. When it comes to time cost, DualAug uses the least additional time cost. All in all, considering all factors, DualAug is a reasonable choice for achieving better performance in comparison to KDforAA and TeachAug.  

| Method | Time Cost | Memory Cost  |
| :---: | :---: | :---: |
| AA | 5.6h | 4453MiB |
| AA+DualAug | 8.6h | 4453MiB$^1$  |
| KDforAA[1] | 12.9h$^2$ | 5804MiB |
| TeachAug | 9.5h | 8714MiB |

1.  In our implementation, DualAug computes the scores of two branches one after another, thus it does not incur any additional memory cost.
2.  Containing the time of training the teacher model

Reviewer comment (Evaluation):
The improvement provided by the approach, although stable on different datasets, seems marginal.

Author response:
**Weakness 4: Improvement of DualAug**

The table below illustrates the performance of the main data augmentation methods on CIFAR-100, WRN-28-10 in recent years. The improvement achieved by DualAug is reasonable, compared to the previous method.

| Method | Accuracy(%) | $\Delta$ |
| :---: | :---: | :---: |
| AutoAugment (2018) | 82.90 | - |
| RandAugment (2019) | 83.30 | 0.40 |
| TrivialAugment (2021) | 83.54 | 0.24 |
| DeepAA (2022) | 84.02 | 0.48 |
| DeepAA+DualAug (2023) | **84.39** | 0.37 |

Moreover, as observed by the reviewer, DualAug's **"stable on different datasets"** is quite noticeable. In addition to different datasets, it is also stable across various baseline data augmentations (AA, RA, and DeepAA) and tasks (classification, semi-supervised learning, and self-supervised learning). Our extra experiment on more dataset in Response to Reviewer c5kH (1/7) also proves that DualAug is competitive.

Reviewer comment (Experiments):
I would like to see a fair comparison with [1] and TeachAugment on the commonly used datasets.

Author response:
Thank you for your constructive and insightful reviews. The following is our response.

**Weakness 1 & Question 1: Comparison and Discussion with TeachAugment and KDforAA[1].**

**KDforAA[1]**: We present a fair comparison between DualAug and KDforAA[1], as shown in the table below. The experiments are based on AutoAugment and use the WRN-28-10 model. To ensure fairness, the settings and Memory/Time cost are aligned with those of KDforAA[1]. DualAug demonstrates better performance compared to KDforAA[1].
| Dataset  | KDforAA* | AA + DualAug |
| :---: | :---: | :---: |
| CIFAR-10 | 97.6 | **97.63±.09** |
| CIFAR-100 | 83.8 | **83.94±.26** |

\* : Reported in KDforAA[1].

[1] Wei, Longhui, et al. "Circumventing outliers of autoaugment with knowledge distillation." ECCV, 2020.

**TeachAugment**: We provide a fair comparison between TeachAugment and DualAug in the same training pipeline, as shown in the table below. The experiments use the WRN-28-10 model. To ensure fairness, we use the EMA model to filter OOD data in Adv. Aug+DualAug, as done in TeachAugment.

| Dataset | TeachAug | Adv. Aug+DualAug | DeepAA+DualAug |
| :---: | :---: | :---: | :---: |
| CIFAR-10 | 97.25%±.11% | 97.35%±.09% | **97.52%±.18%** |
| CIFAR-100 | 83.08%±.39% | 83.02%±.34% | **84.54%±.23%** |

TeachAug and DualAug present comparable performance in the fair setting, using the Adv. Aug as the baseline. Furthermore, compared to TeachAug, we are able to improve upon a more widely automated augmentation method(DeepAA+DualAug), resulting in better performance in the same training pipeline. It is also worth noting that DualAug exhibits better performance in comparedison to TeachAug in self-supervised learning task, as shown in Table 5 (68.67% vs. 68.20%).

Reviewer comment (Evaluation):
Could you compare the computational cost of the proposed approach and competing methods?

Author response:
**Weakness 3 & Question 2: Computational cost of the proposed approach and other methods.**

We present a comparison of computation costs in the table below. The experiment is carried out on CIFAR-100 using a single 2080Ti GPU with WRN-28-10 model. As shown, when considering memory cost, DualAug uses the same memory cost as AA, while TeachAug and KDforAA require additional memory. When it comes to time cost, DualAug uses the least additional time cost. All in all, considering all factors, DualAug is a reasonable choice for achieving better performance in comparison to KDforAA and TeachAug.  

| Method | Time Cost | Memory Cost  |
| :---: | :---: | :---: |
| AA | 5.6h | 4453MiB |
| AA+DualAug | 8.6h | 4453MiB$^1$  |
| KDforAA[1] | 12.9h$^2$ | 5804MiB |
| TeachAug | 9.5h | 8714MiB |

1.  In our implementation, DualAug computes the scores of two branches one after another, thus it does not incur any additional memory cost.
2.  Containing the time of training the teacher model

Reviewer comment (Reproducibility):
How often are mean and std of the basic distribution estimated?

Author response:
**Question 3: How often are mean and std of the basic distribution estimated?**

The mean and std of the basic distribution are estimated in every iteration. In Appendix A, Algorithm 1 details the process for estimating the mean and std of the basic distribution.

We would like to express our gratitude once again for taking the time to review our manuscript. We hope that my response has addressed your concerns and that you will reconsider the value of my manuscript.