Paper: Tailoring Mixup to Data using Kernel Warping functions (ICLR-2024)

Reviewer comment (Experiments):
Compared to RegMixup, the proposed method brings marginal improvement in classification task accuracies on different models (Tables 2 and 3). Plus, the benchmark result of the Manifold Mixup in classification is missing. The concern is on the expressivity of the proposed method since general data augmentation aims to enrich the diversity of data and cover more uncertain data points as much as possible instead of encouraging similar data to be mixed more.

Author response:
- W1:
  - 1. **Comparison with RegMixup:** We would like to refer the reviewer to Appendix B in which we show that compared to both RegMixup and MIT, our proposed approach is much more efficient both in terms of computation time and memory usage. Our method relies on a single batch of augmented data, while the augmentations in RegMixup and MIT double the size of the batch, leading to more compute and more required memory, as discussed in section 4.1 in the paper. 
  - 2. **Manifold Mixup on classification (Table 1):** We reproduced results of Manifold Mixup (Verma et al., ICML 2019) on CIFAR 10 and CIFAR100 using a Resnet50. We achieve similar performance improvements, but with better calibration. As for RegMixup, we can consider combining our method with Manifold Mixup in future work.
  - 3. **Data augmentation aims to cover uncertain data points (Table 2):** The assumption underlying this work is that there is a trade-off in data augmentation procedures between adding *diversity* and introducing *uncertainty*. While we agree that it is important to cover more regions of the space, it is important to do it without adding errors and avoid *manifold intrusion* (Guo et al., 2019; Baena et al., 2022), *i.e*, conflicts between the synthetic labels of the mixed-up examples and the labels of original training data. Our intuition is that mixing data that are too far introduces uncertainty on the true underlying label of the new point, for which a linear combination of the labels might not be an accurate target. We confirm this hypothesis by the following experiment that we report in Table 2, in which we compare *accuracy* and *calibration* metrics when mixing *only* pairs of points with distances lower or higher than a given quantile of the overall pairwise distances within the batch, using a Resnet34 on CIFAR10 and CIFAR100 datasets. Note that one should compare results of "Lower $q$" with "Higher $1 - q$" to have equivalent numbers of possible element to mix with (*diversity*). We can see that, in general, *mixing pairs with lower distances leads to better calibration than mixing pairs with higher distances* (*uncertainty*). The introduction of uncertainty when mixing pairs with high distance can be explained by *manifold intrusion* (Guo et al., 2019; Baena et al., 2022). These results confirm that there is a trade-off between adding *diversity* and *uncertainty with* data augmentation.

Reviewer comment (Theory):
The warped Gaussian Kernel into the original Beta distribution for the mixup coefficient $\lambda$ highly depends on batched data similarity. If the batch size is small (for large-scale tasks), the relative distances among the data will collapse, i.e. the same samples will be mixup with different $\lambda$ whether an outlier appears. In Eqn (6), if $n$ is small, the total distance is sensitive to the largest sample distance. It weakens the robustness of the proposed method.

Author response:
- W2. This is an interesting remark for which several points need to be discussed. 
  - 1. **Statistics can be computed before training (Table 3):** First, if one can use a meaningful distance between data points different from the distance in the embedding space of the model to train, statistics such as mean or median of the pairwise similarity matrix can be computed beforehand to normalize the distance during training. In our experiments on regression tasks, we also compared with normalizing using the mean over the whole training set instead of the batch. We found slightly improved results on Airfoil, for which the batch size is 16, but no meaningful improvements for the two other datasets, for which the batch size is 128. These results are provided in Table 3.
  - 2. **Median can be used to be more robust to outliers:** Second, our rationale behind the normalization by the mean is mainly to rescale the distances to similar scales between tasks and models, and same for $\tau_{max}$ and $\tau_{std}$. If the batch size is too small, one could use the median instead of the mean to be more robust to outliers.
  - 3. **Distance impact distribution of $\lambda$:** Third, regarding the remark "the same samples will be mixup with different $\lambda$ whether an outlier appears", we would like to point that since we are sampling different $\lambda$ at each iteration, it is very likely that the same pair of samples will be mixed with different $\lambda$ at each iteration, whatever the mixup approach we are using. In our case, the *underlying distribution* of the $\lambda$ might vary for the same pair of samples between iterations depending on the distances in each batch, but we do not think that it hinders the robustness of the approach. If the distance between two samples is high, it is *more likely* that the $\alpha$ parameter will be small and thus that $\lambda$ will be close to 0 and 1, even if the batch size is small.

Reviewer comment (Evaluation):
The evaluation and explanation of how the proposed method enhances calibration over vanilla Mixup are insufficient. For example, (Thulasidasan et al., 2019) have concluded that label smoothing in Mixup training significantly contributes to improved calibration, supported by comprehensive observations such as "score of winning class along the time" and "overconfidence". C-Mixup (Yao et al., 2022a) provides the theoretical justification for Theorem 3. To enhance the comparison, it would be better to establish a more meaningful connection between the proposed method and calibration, moving beyond only the presentation of ECE and Brier scores.

Author response:
- W3. **See W1.3** We provide in Table 2 an analysis to back up our intuition that mixing data that are too far introduces uncertainty on the true underlying label of the new point, and that a linear combination of the labels might not be an accurate target.

Reviewer comment (Writing):
Missing reference to the initial mention of "MIT" in Section 2.2.

Author response:
- Typos:
  - The MIT method is introduced in Wang et al. (2023), that we cite in the same sentence. We will make it clearer in the revised version.

Reviewer comment (Writing):
In Eqn(6), the LHS should indicate the dependence of $n$ if the measure on RHS depends on all $n$ samples.

Author response:
- Typos:
  - We will add the mention of the dependence on $n$ in the LHS.

Reviewer comment (Theory):
Since general data augmentation aims to enrich the diversity of data and cover more uncertain data points, is adjusting the distribution of coefficients based on data distribution reasonable to enhance the expressivity? Please see weakness 1.

Author response:
- Q1. **See W1.3**

Reviewer comment (Experiments):
Why choose different benchmarks between Tables 3 and 4, since Manifold Mixup can also be applied to classification problems?

Author response:
- Q2. **See W1.2:** We followed the settings of C-Mixup for our benchmark on regression tasks, for which Manifold Mixup was easily implemented. We provide results for Manifold Mixup on classification tasks in Table 1.

Reviewer comment (Evaluation):
As suggested by (Wang et al., 2023), Mixup fails to improve the calibration in terms of other metrics such as Calibrated ECE and Optimal ECE. Can the author explain more or provide more evidence that the method in this paper does help the calibration? Why does altering the underlying distribution of $\lambda$ yield a better calibration over vanilla Mixup, even with the same similarity measure of input and output (Row 2 in Table 1)?

Author response:
- Q3. **Optimal ECE are compared:** Indeed as mentioned by Wang et al. (2023), but also Ashuka et al. (2020), one should compare ECE after calibration through temperature scaling instead of raw ECE, since the ordering of results can change between the two. That's why we report *Optimal ECE* results after temperature scaling.

Reviewer comment (Novelty):
The idea of dynamically controlling the sampling of $\lambda$ in mixup is not novel, and the idea of controlling the pairing of the data examples based on distributional similarities (like k-mixup) is also well-investigated. As a result, combining these two types of ideas to formulate a new algorithm, like the one proposed in this paper, is also intuitively and empirically straightforward, and doesn't seem to be much surprising or interesting. In my opinion it the amount of contributions in this work is not sufficient to be a full paper in ICLR.

Author response:
- (W1) May we ask the reviewer for references linked to the claim on the novelty of our work ? As far as we know, at the time of submission, we did not find related work that also proposed to dynamically change the underlying distribution of $\lambda$ during training. Furthermore, we are not controlling the *pairing of points*, but how strongly the data are mixed by changing the sampling distribution of the $\lambda$, using the **distance between the two points** to mix, and **not distribution similarities unlike k-mixup** (Greenewald et al., 2021).

Reviewer comment (Experiments):
As a experiment-based work, the datasets used in the experiments are not adequate. For example, in classifications the authors only investigated their proposed algorithm on image datasets. In fact, some datasets, especially the ones with lower dimensionalities, tend to benefit less or even negatively from mixup, and also L2 distance between data points in these datasets may be more statistically meaningful. Such datasets are also worth of experiment verifications of the Kernel Warping Mixup. This also applies to the regression tasks.

Author response:
- (W2) Our goal is to study the impact of mixup on calibration in general for both classification and regression tasks. We considered datasets for classification that are widely used in the literature to accurately compare with other baselines, and they are mainly image datasets. However, it is not the case for the *regression tasks*, where the datasets used are *tabular* and *times series* data. Specifically, on *Airfoil* (tabular data), we can see that **both mixup and manifold mixup have negative effects** on performance and calibration, but not with neither C-Mixup nor our approach. In this case, we are using L2 distance between labels since it works best in this case as shown in Yao et al. (2022).

Reviewer comment (Experiments):
The essential hyperparameters $\tau_{max}$ and $\tau_{std}$ used are chosen through cross-validation to find the optimal values. This may cost extra time before the real training is even started.

Author response:
- (W3) While we agree that finding the correct values for $\tau_{max}$ and $\tau_{std}$ through cross validation beforehand takes additional time, we would like to point that **it is the case for any hyperparameter** and that **other mixup methods also have additional hyperparameters**, such as $\alpha$ in the Beta distributions or the strength of the regularization in RegMixup, or the layers selected to apply mixup in Manifold Mixup. In our case, **we do not have to search for a good potential $\alpha$** since it is already reflected in both $\tau_{max}$ and $\tau_{std}$, which makes one less hyperparameter.

Reviewer comment (Writing):
In the last paragraph of Section 1, "... improve both performance and ...", what performance exactly? Is it refering to generalization performance?

Author response:
- (Q1) When stating that our method "improves both performance and calibration", we are referring to generalization performance in-distribution, i.e. accuracy on the test set, *and* that we improve calibration at the same time.

Reviewer comment (Theory):
Section 3.3. Why $\tau$ should be exponentially correlated with the distance?

Author response:
- (Q2) **The exponential correlation is linked to the warping function that we consider**, i.e. in our case, the Beta CDF function. As can be seen in figure 2 of the paper, there is a *logarithmic* correlation on the shape of the Beta CDF and the parameter $\tau$. Based on this, we defined the similarity kernel with an *exponential correlation* with the distance. As mentioned in the paper, the correlation with the distance will depend on the warping function considered in the framework.

Reviewer comment (Reproducibility):
How is the cross-validation used to find the hyperparameters conducted in details? Is it conducted before the real training? Or is it conducted simultaneously during the training? What objective is considered in finding the "optimal" $\tau$'s?

Author response:
- (Q4) The cross-validation is conducted before training, with a stratified sampling on a 90/10 split of the training set, and the results are averaged across 4 different splits. The hyperparameters are selected to have **a good trade-off between calibration and accuracy**. As mentioned in Pinto et al. (2022), *cross-validating hyperparameters based solely on the ECE can prefer models with lower accuracy but better calibration*. However, a method improving calibration should avoid degrading accuracy. We will clarify the process of the cross validation in the paper.

Reviewer comment (Theory):
Algorithm 1. It seems that the values of $\tau$'s for the inputs and the targets are computed in the identical way. Then what is the point of defining them separately? And also, if the $\tau^o$ is computed separately, how would one define the similarity between two one-hot labels in classifications? Is it going to be like a simple equity indicator function?

Author response:
- (Q5) We thank the reviewer for pointing out this typo in the algorithm. We compute $\tau_i^{(i)}$ with $\tau^{(i)}(\mathbf{x}, i, \sigma; \tau^{(i)}\_{max}, \tau^{(i)}\_{std})$, and $\tau_i^{(o)}$ with $\tau^{(o)}(\mathbf{x}, i, \sigma; \tau^{(o)}\_{max}, \tau^{(o)}\_{std})$. If one wants to use different $\tau^{(i)}$ and $\tau^{(o)}$, one has to use different pairs of $(\tau^{(i)}\_{max}, \tau^{(i)}\_{std})$ and $(\tau^{(o)}\_{max}, \tau^{(o)}\_{std})$.  
- Regarding (Q5), **we do not compute similarity between one-hot labels**. As presented in the paper and in the answer to (Q3), we considered three different distance metrics in classification tasks: *L2 distance on inputs*, *L2 distance on embeddings of inputs*, and *L2 distance between class weights* of the classification layer associated to the input data.

Reviewer comment (Theory):
Section 4.2. Why is the input distance or embedding distance not taken into consideration here? For some regression problem, I believe the similarity between two targets doesn't necessarily indicate a comparable similarity between their inputs.

Author response:
- (Q6) While we agree that we could have considered input or embedding distance for regression tasks, it was shown in Yao et al. (2022), that label distance works best in this case. We thank the reviewer for this remark and will make it clear in the revised version.

Reviewer comment (Experiments):
I don't think regression task can be convincing as downstream task for novelty. Regression task, in somehow, can already achieve very good result. I think a challenging downstream task such as object detection can make this paper more attractive.

Author response:
The main goal of the paper is to study and improve the impact of mixup on calibration in general. We present experiments for both classification and regression tasks, since it is **two very different settings**, and not that well studied in regression. Nevertheless, we show the *flexibility* of our framework, which can improve calibration in both tasks. However, as mentioned in the conclusion of our paper, we are interested to apply our method on more difficult regression tasks such as Monocular Depth Estimation, but leave that to future work.

Reviewer comment (Reproducibility):
How do you define/calculate distance?

Author response:
As presented in Section 3.3, we use an L2 distance, on either input data, embedding of inputs, classification weights or labels depending on the task.

Reviewer comment (Experiments):
(W1) Is the studied problem really important to mixup augmentations, i.e., it is practically useful to conduct mixup interpolation with similar samples? I cannot find any empirical analysis demonstrating that previous mixup methods will encounter serious drawbacks (e.g., performances, calibration abilities, generalization abilities to tasks) because of not mixing similar samples well. For example, the authors should provide some visualizations of effects or statistics to demonstrate the problem in addition to Figure 1.

Author response:
- (W1) The assumption underlying this work is that there is a trade-off in data augmentation procedures between adding *diversity* and introducing *uncertainty* through **manifold intrusion** (Guo et al., 2019; Baena et al., 2022), *i.e*, conflicts between the synthetic labels of the mixed-up examples and the labels of original training data. To show that, we conduct in Table 1 an empirical analysis that compares *accuracy* and *calibration* metrics when mixing *only* pairs of points with distances lower or higher than a given quantile of the overall pairwise distances within the batch, using a Resnet34 on CIFAR10 and CIFAR100 datasets. Note that one should compare results of "Lower $q$" with "Higher $1 - q$" to have equivalent numbers of possible element to mix with (*diversity*). We can see that, in general, **mixing pairs with lower distances leads to better calibration than mixing pairs with higher distances** (*uncertainty*). The introduction of uncertainty when mixing pairs with high distance can be explained by *manifold intrusion* (Guo et al., 2019; Baena et al., 2022). These results confirm that there is a trade-off between adding *diversity* and *uncertainty with* data augmentation.

Reviewer comment (Experiments):
(W2a) The compared baselines are restricted to classical methods in classification tasks, and the performance gains are limited in all comparison results. These results make me doubt the importance of the studied problem in this paper, i.e., mixing more similar mixup samples than less similar ones.

Author response:
- (W2) Along with the empirical analysis detailed above, we also provide experiments on the Tiny-Imagenet benchmark in Table 2. We can see that our approach improves both performance and calibration over Mixup baselines on this dataset as well, and reach similar calibration than RegMixup. We are working on adding the other baselines for this dataset as well. As mentioned in the conclusion of the paper, we are working on combining our approach with other mixup methods. Regarding the references mentioned by the reviewer:
    - The OpenMixup paper [1] and library mentioned is still under construction, but **we plan to contribute to this library** once our paper is accepted.
    - **We are already comparing against C-Mixup [2]** on the same datasets in the experiments on regression tasks.
    - CutMix [3,5], PuzzleMix [4] and TransMix [6] are **methods focused on image data**. As our main goal is to study and improve the impact of mixup on *calibration in general*, we do not want to restrict our evaluation to image datasets and image methods. We consider regression tasks on tabular and time series data, to evaluate calibration of Mixup methods in this setting, which have rarely been investigated in the literature, but all of the above methods are not adapted to these types of data.
    - As both MixupE [8] and Decouple Mixup [7] have been published **very recently**, implementing these methods will require further work.

Reviewer comment (Experiments):
(W2b) The experiments are small-scale (e.g., CIFAR-10/100) with classical network architectures (ResNet variants) and old-fashioned baselines. There are many open-source mixup methods and benchmarks for classification and regression tasks [1, 2], and I suggest the authors consider more practical and modern experiment settings (e.g., large-scale experiments on ImageNet, modern Transformer backbones on CIFAR-100).

Author response:
*As mentioned in Section 4.1*, we search for the best hyperparameters through **cross-validation**. The cross-validation is conducted before training, with a stratified sampling on a 90/10 split of the training set, and the results are averaged across 4 different splits. The optimal values will vary between datasets, since they depend on the statistics of the pairwise distances. We assume that the behavior of these parameters is also impacted by the number of class and their separability in each dataset. We provide the resulting heatmaps of the cross validation in Figure 5 and Figure 6 for a better visualization, but we will also include the exact values obtained for all the cross validations in the Appendix.

Reviewer comment (Experiments):
(W3) Hyper-parameter sensitivity. As shown in Appendix D and E (Table 6), the hyper-parameters of the proposed method vary significantly on different datasets and tasks. The ablation and sensitivity analysis of these hyper-parameters should be added. Meanwhile, I wonder how the authors determine the two hyper-parameters, which should be detailed in the appendix.

Author response:
*As mentioned in Section 4.1*, we search for the best hyperparameters through **cross-validation**. The cross-validation is conducted before training, with a stratified sampling on a 90/10 split of the training set, and the results are averaged across 4 different splits. The optimal values will vary between datasets, since they depend on the statistics of the pairwise distances. We assume that the behavior of these parameters is also impacted by the number of class and their separability in each dataset. We provide the resulting heatmaps of the cross validation in Figure 5 and Figure 6 for a better visualization, but we will also include the exact values obtained for all the cross validations in the Appendix.

Reviewer comment (Presentation):
(W4) The related work section is not well presented. I suggest the author combine Sec. 2.2 and Sec. 3.1 as a new section called Preliminary. Meanwhile, the authors may include more recently published mixup algorithms in different categories.

Author response:
We will make sure to add the latest relevant papers that we might have missed since the moment of submission. Regarding the two subsections mentioned by the reviewer, we will try to combine them, but it might be difficult since they cover different topics. While Sec. 2.2 presents related work on calibration, Sec. 3.1 introduces notations used throughout the paper.