Paper: Kill Two Birds with One Stone: Rethinking Data Augmentation for Deep Long-tailed Learning (ICLR-2024)

Reviewer comment (Evaluation):
While the main tables present average values, the standard deviations are missing; it would be better to provide complete results, including standard deviations, in the appendices.

Author response:
Thank you for your comments. Here, we simply provide a demonstrative result. The table shows partial results of DODA on CIFAR-100-LT, including standard deviations. Furthermore, based on your suggestions, we will supplement other results in the appendix to ensure the comprehensiveness of our work.

**Table: Accuracy (%) on CIFAR-100-LT dataset.**
|Method|IR = 10|IR = 50|IR = 100|
|:-|:-:|:-:|:-:|
|CE  + DODA|59.9 $\pm$ 1.2|48.0 $\pm$ 0.8|44.5 $\pm$ 0.5|
|BS  + DODA|61.8 $\pm$ 1.0|54.0 $\pm$ 0.9|48.7 $\pm$ 0.7|
|BCL + DODA|62.7 $\pm$ 1.0|53.6 $\pm$ 0.4|51.0 $\pm$ 0.6|

Reviewer comment (Evaluation):
Could you provide some valuable insights regarding the decline in performance that we see in the RIDE column of Table 1?

Author response:
Thank you for your comments. As you mentioned, we find that the performance of DODA in balanced settings or settings close to balanced (IR = 10) is not as stable as in imbalanced settings, such as fluctuating performance on RIDE. We consider that its multi-expert setting may lead to inconsistencies in the tendency when choosing augmentations, i.e., conflicting tendencies. Nevertheless, by observing the experimental results, we find that DODA still shows superior performance in the medium classes.

Reviewer comment (Experiments):
UniformAugment (LingChen et al., 2020) is a suitable baseline for comparison because of its straightforward application of a set of augmentations at random. It would be nice to see the UniformAugment baseline results.

Author response:
Thank you for your insightful comments. Based on your suggestions, we have conducted comparative experiments on UniformAugment and DODA on CIFAR-100-LT (IR = 50 and 100). The experimental results shown in the table below indicate that under imbalanced distribution, DODA achieves better performance than UniformAugment. This improvement is comprehensive, especially in the tail classes, indicating that allowing each class to choose the augmentation method that is beneficial to itself can avoid significant sacrifices. This is consistent with what you mentioned.

**Table: Accuracy (%) on CIFAR-100-LT dataset (IR = 50 and 100) wtih UniformAugment.**
|Method|Head|Medium|Tail|All (50)|Head|Medium|Tail|All (100)|
|:-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
|CE + UniformAugment|68.5|34.8|10.4|**44.2**|71.5|38.5|5.8 |**40.2**|
|CE + DODA          |71.2|40.3|12.6|**48.0**|74.8|43.8|10.0|**44.5**|
|BS + UniformAugment|62.7|46.7|38.0|**51.5**|62.8|49.1|26.7|**47.2**|
|BS + DODA          |62.2|51.2|41.5|**54.0**|63.1|49.3|31.2|**48.7**|

Reviewer comment (Experiments):
It seems that the proposed methodology could be effective even in scenarios handling balanced datasets. Have you experimentally verified this by any chance?

Author response:
Thank you for your comments. We have conducted exploratory experiments on balanced data distribution, e.g., original CIFAR-100, but it was not mentioned in our paper because our current focus is on the long-tailed setting. We strongly agree with your point of view, and the experimental results also confirm this. From the results in the table below, we can see that DODA still shows some effectiveness under balanced settings, although it is not as good as some methods for balanced distribution. Considering that DODA is aimed at the sacrifice problem under imbalanced distribution, we think this performance is acceptable.

**Table: Accuracy (%) on CIFAR-100 dataset.**
|Method|AutoAugment|UniformAugment|CUDA|DODA|
|:-:|:-:|:-:|:-:|:-:|
|Acc.|71.6|69.2|69.8|69.6|

The original baseline uses Wide-ResNet as the backbone. To maintain fairness, we replace it with a standard ResNet-32.

Reviewer comment (Writing):
’its’ in the caption of Figure 6; quotation mark issues in LaTeX :)

Author response:
Thanks again for your time and thoughtful suggestions, we have corrected this error :)

Reviewer comment (Experiments):
The proposed method for DODA is exclusively used where augmentations are solely for one single class and do not encourage inter-class interaction. Augmentation methods such as mixup and its improved version Remix (Chou et al. 2020) have proven to be efficient in data-imbalance settings. However, comparison of the proposed method with Remix + LTL methods is lacking.

Author response:
Thank you for your comments. Based on your suggestion, we have conducted comparative experiments on Remix, CUDA, and DODA on CIFAR-100-LT (IR = 50 and 100). The experimental results shown in the table below indicate:

- All three augmentation strategies can achieve performance improvements under imbalanced data distribution, demonstrating that reasonable use of data augmentation can alleviate the long-tailed problem.

- DODA achieved better performance than Remix, especially for tail classes, indicating that allowing each class to choose DAs beneficial to itself can avoid significant sacrifices.

**Table: Accuracy (%) on CIFAR-100-LT dataset (IR = 50 and 100) wtih Remix.**
|Method|Head|Medium|Tail|All (50)|Head|Medium|Tail|All (100)|
|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
|CE + Remix|70.3|38.8|11.0|**46.7**|71.7|42.0|5.7 |**41.5**|
|CE + CUDA |68.3|38.4|13.7|**46.2**|70.7|41.4|9.3 |**42.0**|
|CE + DODA |71.2|40.3|12.6|**48.0**|74.8|43.8|10.0|**44.5**|
|BS + Remix|62.8|47.7|41.2|**52.6**|64.0|50.7|27.4|**48.4**|
|BS + CUDA |63.6|48.4|37.3|**52.7**|62.5|49.1|29.4|**47.9**|
|BS + DODA |62.2|51.2|41.5|**54.0**|63.1|49.3|31.2|**48.7**|

Reviewer comment (Experiments):
The algorithm optimizes the augmentation strategy on 10 predefined augmentation functions with fixed strengths. The authors should discuss the optimality of the selection of these functions and provide ablation studies for the same.

Author response:
Thank you for your valuable comments. To verify the effectiveness of DODA, we have chosen ten widely used augmentation methods. Essentially, we could also choose more augmentation methods. In this case, the augmentation preference (or preference list) for each class may change, as they find DAs that are more beneficial to them. That is to say, when we add other augmentation methods, each class still tends to choose the most beneficial augmentation for itself. 

Based on your suggestion, on CIFAR-100-LT (IR = 100) and BS baseline, we have compared different candidate augmentation combinations, including 10, 15, and 20, and the corresponding experimental results are shown in the following table. We can find that DODA is still effective even though the predefined augmentation methods have changed. In addition, with the increase in the number of DAs, DODA may need more training overhead to obtain the optimal performance, i.e., it needs to iterate more on the weight matrix, and 200 epochs may not be enough to find the optimal weight.

**Table: Accuracy (%) on CIFAR-100-LT dataset (IR = 100) wtih different candidate augmentation combinations.**
|Method|Head|Medium|Tail|All (100)|
|:-:|:-:|:-:|:-:|:-:|
|BS|59.6|42.3|23.7|**42.8**|
|BS + DODA w/ 10 DAs|63.1|49.3|31.2|**48.7**|
|BS + DODA w/ 15 DAs|62.7|48.7|28.2|**47.9**|
|BS + DODA w/ 20 DAs|63.1|50.0|28.6|**48.2**|

We also provide specific augmentation methods.

    l = [
        (Flip, 0, 1),
        (Mirror, 0, 1),
        (EdgeEnhance, 0, 1),
        (Detail, 0, 1),
        (Smooth, 0, 1),
        (AutoContrast, 0, 1),
        (Equalize, 0, 1),
        (Invert, 0, 1),
        (GaussianBlur, 0, 2),
        (Rotate, 0, 30), // 10

        (ShearX, 0., 0.3),
        (ShearY, 0., 0.3),
        (Color, 0.1, 1.9),
        (Brightness, 0.1, 1.9),
        (Sharpness, 0.1, 1.9), // 15

        (TranslateXabs, 0., 100),
        (TranslateYabs, 0., 100),
        (Posterize, 0, 4),
        (Solarize, 0, 256),
        (SolarizeAdd, 0, 110), //20
        ]

Reviewer comment (Experiments):
The effectiveness of DODA on state-of-the-art long-tailed methods PaCo [R1], and NCL [R2] is not demonstrated in the paper. It would be great if the results are provided in the rebuttal.

Author response:
Thanks so much for bringing these recent works to us.  Following your suggestions, we have analyzed these two papers to determine the feasibility of combining them with DODA. Due to computational resource constraints, we have replicated PoCa and NCL and conducted tests under two imbalance settings on CIFAR-100-LT. We respectfully argue two points:

- For PoCa, based on the current experimental environment, we regret that we cannot directly run the official code, so we cannot provide quantitative results. However, we further analyzed the combinability of PoCa and DODA, and we found that through automatic augmentation of a single contrastive view, the two methods can achieve a good combination, and we continue to explore the subsequent results.

- For NCL, we conducted comparative experiments on CIFAR-100-LT to verify the superiority of DODA, and the experimental results are shown in the following table. We can find that DODA has achieved better performance under the two settings, indicating that DODA can effectively alleviate this intertwined imbalance problem.

    **Table: Accuracy (%) on CIFAR-100-LT dataset (IR = 50 and 100) wtih NCL.**
    |Method|Head|Medium|Tail|All (50)|Head|Medium|Tail|All (100)|
    |:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
    |NCL + CUDA|68.9|52.5|47.8|**58.4**|69.2|55.2|34.4|**53.9**|
    |NCL + DODA|68.1|53.2|49.2|**58.6**|69.4|55.6|35.7|**54.4**|

Reviewer comment (Reproducibility):
For contrastive methods such as BCL, how is DODA being used to generate multiple views of the same image for different classes?

Author response:
Thank you for your comments. For contrastive learning-based methods, e.g., BCL, we define different views of the current input in the *getitem* function when we obtain the dataset. One view is obtained by sampling DA according to the augmentation weight distribution of each class in DODA, and the other is an augmented view obtained traditionally for all classes, i.e., according to the code provided by each contrastive learning baselines, a consistent augmentation is adopted.

Reviewer comment (Presentation):
Section 4.3 mentions that 'DODA can avoid more classes being sacrificed', but it seems that this cannot be intuitively found based on Figure 4.

Author response:
Sorry for the confusion. By observing Figure 2 and Figure 4 simultaneously, we can observe that the sacrifice rate of DODA is much lower, indicating that it can significantly alleviate the sacrifice problem. We also display the sacrifice rates of different methods in the following table so that you can more intuitively confirm the effectiveness of DODA.

**Table: Sacrifice rate (%) under diffirent DAs on CAFIR-100-LT dataset. $\delta$ is the relative improvement.**
|IR   |CE + Cutout|CE + CUDA|CE + CUDA &Cutout|CE + DODA | $\delta$ v.s. CUDA
|:-:|:-:|:-:|:-:|:-:|:-:|
|$50$    |$31\\%$|$38\\%$|$27\\%$ |$7\\%$|$-31\\% p$|
|$100$ |$21\\%$|$29\\%$|$23\\%$ |$5\\%$ |$-24\\% p$|

Reviewer comment (Presentation):
Regarding Section 4.3, the authors mention that DODA can avoid more classes being sacrificed. However, it may not be intuitive to find this based on Figure 4. Therefore, it would be helpful if the author could provide a more detailed explanation of the model's effect, which would allow others to better understand the improvements brought about by this method.

Author response:
Sorry for the confusion. By observing Figure 2 and Figure 4 simultaneously, we can observe that the sacrifice rate of DODA is much lower, indicating that it can significantly alleviate the sacrifice problem. We also display the sacrifice rates of different methods in the following table so that you can more intuitively confirm the effectiveness of DODA.

**Table: Sacrifice rate (%) under diffirent DAs on CAFIR-100-LT dataset. $\delta$ is the relative improvement.**
|IR   |CE + Cutout|CE + CUDA|CE + CUDA &Cutout|CE + DODA | $\delta$ v.s. CUDA
|:-:|:-:|:-:|:-:|:-:|:-:|
|$50$    |$31\\%$|$38\\%$|$27\\%$ |$7\\%$|$-31\\% p$|
|$100$ |$21\\%$|$29\\%$|$23\\%$ |$5\\%$ |$-24\\% p$|