Paper: OpenMixup: A Comprehensive Mixup Benchmark for Visual Classification (ICLR-2024)

Reviewer comment (Novelty):
**Lack of clear Take-Home Message**: The paper is a tech report that describes the functionalities and the different components in the library. While I do not dismiss the massive work put behind building the library, the paper is missing a clear take-home message from the "interesting observations" claimed by the authors. There is no substantial findings or actionable insights beyond the existence of the associated codebase.

Author response:
Thank you for your valuable comments. Based on your advice, we have summarized and analyzed a great number of benchmarking results to compare and rank all the included mixup methods in terms of **performance, applicability,** and **overall capability**. 

Specifically, regarding performance, we averaged the accuracy rankings of all mixup algorithms for each downstream task, as well as averaging their robustness and calibration (supplementary experiments) rankings separately. Finally, these ranking averages were averaged again to produce the comprehensive range of performance ranking results. As for speed, we adopt a similar ranking computation scheme. With the overall capability ranking, we combined the performance and applicability rankings with a 1:1 weighting to obtain the final take-home rankings.

Finally, we provide the **take-home rankings** as shown in the following tables. Again, we appreciate your constructive suggestion.


|               | Mixup | CutMix | DeiT | SmoothMix | GridMix | ResizeMix | ManifoldMix | FMix | AttentiveMix | SaliencyMix | PuzzleMix | AlignMix | AutoMix | SAMix | TransMix | SMMix |
|---------------|:-----:|:------:|:----:|:---------:|:-------:|:---------:|:-----------:|:----:|:------------:|:-----------:|:---------:|:--------:|:-------:|:-----:|:--------:|:-----:|
| Performance   |   13  |   11   |   5  |     16    |    15   |     8     |      12     |  14  |       7      |      9      |     6     |    10    |    2    |   1   |     4    |   3   |
| Applicability |   1   |    1   |   1  |     1     |    1    |     1     |      1      |   1  |       3      |      1      |     4     |     2    |    7    |   6   |     5    |   5   |
| Overall       |   8   |    6   |   1  |     11    |    10   |     4     |      7      |   9  |       5      |      5      |     5     |     6    |    4    |   2   |     4    |   3   |

Reviewer comment (Evaluation):
**Misalignment of Evaluation Metrics**: The paper introduces several evaluation metrics which are not applied in the experiments, such as calibration or robustness. The primary focus remains on accuracy, which as already been extensively studied in the papers associated with each individual method. This limits the paper's potential to offer a fresh perspective on the methods by not exploring alternative criteria beyond accuracy, especially as other evaluation metrics are already described as available in the benchmark.

Author response:
Thank you for raising the issue we had with evaluation metrics. Based on your suggestion, we have revised the article where the presentation was not rigorous. Meanwhile, for robustness and calibration evaluation, we have added the related experiments, as shown in the table below. 

The results show that although these new evaluation metrics provide a variety of interpretable perspectives, the conclusions they provide are still consistent with those of accuracy. This is also consistent with what we know about mixup as an augmentation method. An augmentation method is good as long as it is effective, fast, and can be applied to multiple scenarios concurrently. Furthermore, we have provided comprehensive radar plots in the paper, which clearly reflect the performance of each algorithm in different scenarios. We suppose this is a comprehensive evaluation of existing mixup algorithms. Nonetheless, we have added the corresponding experiments you mentioned and have come up with consensus results.

Reviewer comment (Presentation):
The color gradient in the plots make it difficult to really differentiate two methods.

Author response:
We appreciate that you pointed out the lack of differentiation of color gradients. Since we cover a lot of algorithms, selecting various color gradients can be more or less difficult to differentiate. We have tried to make changes in our revision to make the color gradients in plots more distinguishable. Again, thank you for your valuable suggestions.

Reviewer comment (Novelty):
What would be the main take-home message from the empirical study presented? Beyond the existence of the open-source library, what should we remember from this paper?

Author response:
Thank you for your valuable comments. Based on your advice, we have summarized and analyzed a great number of benchmarking results to compare and rank all the included mixup methods in terms of **performance, applicability,** and **overall capability**. 

Specifically, regarding performance, we averaged the accuracy rankings of all mixup algorithms for each downstream task, as well as averaging their robustness and calibration (supplementary experiments) rankings separately. Finally, these ranking averages were averaged again to produce the comprehensive range of performance ranking results. As for speed, we adopt a similar ranking computation scheme. With the overall capability ranking, we combined the performance and applicability rankings with a 1:1 weighting to obtain the final take-home rankings.

Finally, we provide the **take-home rankings** as shown in the following tables. Again, we appreciate your constructive suggestion.


|               | Mixup | CutMix | DeiT | SmoothMix | GridMix | ResizeMix | ManifoldMix | FMix | AttentiveMix | SaliencyMix | PuzzleMix | AlignMix | AutoMix | SAMix | TransMix | SMMix |
|---------------|:-----:|:------:|:----:|:---------:|:-------:|:---------:|:-----------:|:----:|:------------:|:-----------:|:---------:|:--------:|:-------:|:-----:|:--------:|:-----:|
| Performance   |   13  |   11   |   5  |     16    |    15   |     8     |      12     |  14  |       7      |      9      |     6     |    10    |    2    |   1   |     4    |   3   |
| Applicability |   1   |    1   |   1  |     1     |    1    |     1     |      1      |   1  |       3      |      1      |     4     |     2    |    7    |   6   |     5    |   5   |
| Overall       |   8   |    6   |   1  |     11    |    10   |     4     |      7      |   9  |       5      |      5      |     5     |     6    |    4    |   2   |     4    |   3   |

Reviewer comment (Evaluation):
The paper would be improved with a comparison of the methods on other evaluation metrics than accuracy, as it might provide a different perspective on the implemented methods. However, it is understandable that this would require a lot of new experiments.

Author response:
As mentioned in the second response, based on your suggestion, we have added the related robustness and calibration evaluation experiments, as shown in Table A8 in the revised version. The results show that although these new evaluation metrics provide a variety of interpretable perspectives, the conclusions they provide are still consistent with those of accuracy. This is also consistent with what we know about mixup as an augmentation method. An augmentation method is good as long as it is effective, fast, and can be applied to multiple scenarios concurrently. Furthermore, we have provided comprehensive radar plots in the paper, which clearly reflect the performance of each algorithm in different scenarios. We suppose this is a comprehensive evaluation of existing mixup algorithms. However, we have added the corresponding experiments you mentioned and have come up with consensus results.

Reviewer comment (Novelty):
This article is a summary of the previous Mixup-based data augmentation approaches and does not propose new tasks and methods.

Author response:
Again, we would like to express our appreciation to you for pointing out the issue of academic advancements (new tasks and methods). Regarding the points you raised for improvement:

### **1. No New Tasks and Methods.**

First, we have elaborated on **why we need a comprehensive mixup benchmark.** and **why we need a standardized open-source platform.** in the abstract and Sec.1. Briefly, mixup is a widely-used data augmentation method in modern Convolutional Neural Networks (CNNs) and Visual Transformers (ViTs). To the best of our knowledge, however, as a fundamental building block of today's computer vision architecture, there has been no such comprehensive benchmarking study (which is both **time-consuming** and **computationally resource-intensive**) for the mixup community to drive progress in a standard manner. As researchers, we believe that we can all agree that an area will **struggle to advance without a comprehensive benchmark**. This is the first reason that we focus on mixup, especially for building a comprehensive mixup visual classification benchmark. **It is quite urgent work for the community**. **This area needs a benchmark.** In this context, our work is indeed **the first** mixup visual classification benchmark. Even though we do not present new algorithms or define new tasks, we are still highly meaningful to the community as the first benchmarking study and a unified codebase platform. 

Second, we have to emphasize that most current mixup methods are crafted with diverse angles of enhancement, applying distinct configurations and even different coding styles **without an all-in-one open-source coding platform** for streamlined pre-processing, method design, training, and evaluation. The lack of standardization **poses obstacles to fair comparison and effortless development**, necessitating **costly trial-and-error for researchers**. This is the second reason that we focus on mixup augmentations, especially for providing a unified model design and training platform.

In summary, we conclude our main contributions and values as follows:

**(a)** Mixup is an augmentation method that is widely employed nowadays. However, there is no comprehensive benchmark for mixup algorithms so far. This will lead to the **absence of a standardized criterion** for numerous mixup classification methods, and researchers have no idea how to choose a proper method. Personally, there are many researchers who reached out to me and stated that the existing mixup classification algorithms are **varied and full of pitfalls**. 

To address this issue, OpenMixup is presented as **the first** comprehensive mixup visual classification benchmark. It first enables researchers to **fairly compare** the performance of their models against a broad set of mixup baselines, providing a **clear and objective measurement** of the method's effectiveness. Such a first-of-its benchmarking study is both **time-consuming** and **resource-intensive**, but we still carried it out **from scratch**. 

I suppose this makes our work truly meaningful and full of value, even though the scope is objectively limited.

**(b)** Apart from the benchmarking study, we provide **a standardized model design and training codebase platform** for mixup-based visual classification. The overall framework is streamlined and modular, which can enhance the **accessibility and customizability of downstream deployment and applications**.

**(c) Interesting observations, insights, and take-home conclusive rankings** on comprehensive performance, robustness, training stability, and convergence are obtained (**In Sec.4**) with diverse analysis toolkits (*e.g.* **accuracy radar plots**, **trade-off plots**, loss landscape, CAM visualization, *etc.*) from our OpenMixup platform, enabling researchers to identify what specific properties contribute to the success of different methods. In the revision, we aim to provide take-home ranking results that offer a clear message, helping researchers select the proper mixup methods. Archers to identify what specific properties contribute to the success of different methods.

Reviewer comment (Experiments):
Lack of latest ViT's backbone results for benchmarks. Such as FastViT: "Vasu P K A, Gabriel J, Zhu J, et al. FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterization[J]. //Proceedings of the IEEE/CVF International Conference on Computer Vision (2023)"

Author response:
### **3. The FastViT Backbone Results.**

We appreciate you pointing out our missing backbone. Due to the limited time and computational resources we have in the rebuttal period, we would add this work to the reference first and conduct related experiments on this work in our subsequent revisions. Thanks again.

Reviewer comment (Novelty):
The main contribution seems to be the implementation of mixup based models. I am not sure ICLR is a good venue for this kind of work.

Author response:
Again, we would like to express our appreciation to you for pointing out the issue of the scope and the value to the community. Here, we will address your concern from the following two aspects:

### **1. On the Value and Contributions of OpenMixup.**

We appreciate your valuable comments. We present a comprehensive overview of our contributions and novelties to the community.

**(1)** Mixup is an augmentation method that is widely employed nowadays. However, there is no comprehensive benchmark for mixup so far. This will lead to the **absence of a standardized criterion** for numerous mixup classification methods, and researchers have no idea how to choose a proper method. Personally, there are many researchers who reached out to me and stated that the existing mixup classification algorithms are **varied and full of pitfalls**. As researchers, we believe that we can all agree that an area will **struggle to advance without a comprehensive benchmark**. It is quite urgent for the community. This area needs a benchmark.

To address this issue, OpenMixup is presented as **the first** comprehensive mixup visual classification benchmark. It first enables researchers to **fairly compare** the performance of their models against a broad set of mixup baselines, providing a **clear and objective measurement** of the method's effectiveness. Such a first-of-its benchmarking study is both **time-consuming** and **resource-intensive**, but we still carried it out **from scratch**. 

I suppose this makes our work truly meaningful and full of value, even though the scope is objectively limited.

**(2)** Apart from the benchmarking study, we provide **a standardized model design and training codebase platform** for mixup-based visual classification. The overall framework is streamlined and modular, which can enhance the **accessibility and customizability of downstream deployment and applications**.

**(3)** **Interesting observations and insights** on comprehensive performance, robustness, training stability, and convergence are obtained (**In Sec.4**) with diverse analysis toolkits (*e.g.* **accuracy radar plots**, **trade-off plots**, loss landscape, CAM visualization, *etc.*) from our OpenMixup platform, enabling researchers to identify what specific properties contribute to the success of different methods. In the revision, we aim to provide take-home ranking results that offer a clear message, helping researchers select the proper mixup methods.

Reviewer comment (Experiments):
Detection and Segmentation are currently implemented as Transfer Learning from classification trained backbones which is quite limited as there are also mixup models for detection.

Author response:
We appreciate your valuable comment. First, as emphasized in our revision (**in Sec.1**), although mixup exerts its power in other downstream tasks like object detection, it is derived from its supervised classification counterpart with **shared attributes**. Meanwhile, mixup augmentations utilized in segmentation tasks need to be analyzed and specially designed on a case-by-case basis since most of the mixup methods for classification tasks cannot be directly used in segmentation tasks. This is less relevant to the study of the properties of the mixup algorithm itself. We suppose that the study of mixup classification is sufficient to provide a comprehensive understanding of mixup. 

Second, while our main focus is indeed on supervised visual classification tasks with mixup augmentations, the platform we provided, OpenMixup, can be extended to other tasks, as you mentioned, such as object detection and semantic segmentation. We have provided **Transfer Learning to Semantic Segmentation** and more detailed benchmarking results in the revised supplementary materials to better evaluate the mixup methods. We believe that the unified streamlined platform we provided could be beneficial for these downstream tasks, and we intend to extend our work in these directions in the future.

Reviewer comment (Novelty):
How this engineering focused and narrowed application benchmark benefit the wide community of ICLR?

Author response:
Again, we would like to express our appreciation to you for pointing out the issue of the scope and the value to the community. Here, we will address your concern from the following two aspects:

### **1. On the Value and Contributions of OpenMixup.**

We appreciate your valuable comments. We present a comprehensive overview of our contributions and novelties to the community.

**(1)** Mixup is an augmentation method that is widely employed nowadays. However, there is no comprehensive benchmark for mixup so far. This will lead to the **absence of a standardized criterion** for numerous mixup classification methods, and researchers have no idea how to choose a proper method. Personally, there are many researchers who reached out to me and stated that the existing mixup classification algorithms are **varied and full of pitfalls**. As researchers, we believe that we can all agree that an area will **struggle to advance without a comprehensive benchmark**. It is quite urgent for the community. This area needs a benchmark.

To address this issue, OpenMixup is presented as **the first** comprehensive mixup visual classification benchmark. It first enables researchers to **fairly compare** the performance of their models against a broad set of mixup baselines, providing a **clear and objective measurement** of the method's effectiveness. Such a first-of-its benchmarking study is both **time-consuming** and **resource-intensive**, but we still carried it out **from scratch**. 

I suppose this makes our work truly meaningful and full of value, even though the scope is objectively limited.

**(2)** Apart from the benchmarking study, we provide **a standardized model design and training codebase platform** for mixup-based visual classification. The overall framework is streamlined and modular, which can enhance the **accessibility and customizability of downstream deployment and applications**.

**(3)** **Interesting observations and insights** on comprehensive performance, robustness, training stability, and convergence are obtained (**In Sec.4**) with diverse analysis toolkits (*e.g.* **accuracy radar plots**, **trade-off plots**, loss landscape, CAM visualization, *etc.*) from our OpenMixup platform, enabling researchers to identify what specific properties contribute to the success of different methods. In the revision, we aim to provide take-home ranking results that offer a clear message, helping researchers select the proper mixup methods.

Thanks for your timely reply and suggestions. We have also provided useful conclusions in Sec. 4 (e.g., the take-home message in Table 5) and believe these will benefit the community in both academic research and practical applications of mixup augmentations. Once again, we express our sincere appreciation for your time and efforts in reviewing our work.

Reviewer comment (Reproducibility):
How do you extend the current framework to add the mixup algorithms for object detection that are not based on transfer learning?

Author response:
We appreciate your valuable comment. First, as emphasized in our revision (**in Sec.1**), although mixup exerts its power in other downstream tasks like object detection, it is derived from its supervised classification counterpart with **shared attributes**. Meanwhile, mixup augmentations utilized in segmentation tasks need to be analyzed and specially designed on a case-by-case basis since most of the mixup methods for classification tasks cannot be directly used in segmentation tasks. This is less relevant to the study of the properties of the mixup algorithm itself. We suppose that the study of mixup classification is sufficient to provide a comprehensive understanding of mixup. 

Second, while our main focus is indeed on supervised visual classification tasks with mixup augmentations, the platform we provided, OpenMixup, can be extended to other tasks, as you mentioned, such as object detection and semantic segmentation. We have provided **Transfer Learning to Semantic Segmentation** and more detailed benchmarking results in the revised supplementary materials to better evaluate the mixup methods. We believe that the unified streamlined platform we provided could be beneficial for these downstream tasks, and we intend to extend our work in these directions in the future.