Paper: Selective Mixup Helps with Distribution Shifts, But Not (Only) because of Mixup (ICLR-2024)

Reviewer comment (Experiments):
The authors claim that the selective mixup yields a training distribution over those classes that is closer to the uniform distribution. The authors do not provide results for the imbalanced classification on standard datasets such as CIFAR-10LT, CIFAR-100 LT, and imagenet1k-LT.

Author response:
- The 5 datasets we use come directly from the most directly-relevant prior work (selective mixup for distribution shifts).

- Our main claim is that this prior work missed important ablations, hence our priority is to perform the missing experiments on the same datasets.

- The point of this paper is to improve our scientific understanding of selective mixup with distribution shifts. Experiments on *-LT could be interesting but would add little value to the main claims.

Reviewer comment (Novelty):
I find a lack of novelty in the finding that mixup yields a training distribution with higher entropy, i.e., closer to the uniform distribution, Could you provide any strong theoretical justification that the actual mixup is not contributing to an improvement in the performance since existing works such as MiSLAS do show superior performance to vanilla classifier retraining on training with class balanced samplers.

Author response:
Not sure what "*lack of novelty*" means (??). Prior work missed this as an explanation for the effects of selective mixup. **This is exactly why this paper is important: it highlighting an effect that has been in plain sight but missed in highly-cited prior work.**

--------------
> Could you provide any strong theoretical justification that the actual mixup is not contributing to an improvement

The paper never makes this claim, which is factually incorrect from our experiments. The paper states the exact opposite multiple times:

- yearbook dataset: "*(...) confirms the complementarity of the effects of resampling and within-class selective mixup*"
- arxiv dataset: "*the performance of selective mixup is explained by cumulative effects of vanilla mixup and resampling effects*"
- civilComments dataset: "*the performance of selective mixup is the result of the independent effects of vanilla mixup and resampling*"

Different datasets benefit differently from mixup and/or resampling. This is why there is no single simple story, and why the results are analyzed on each dataset separately.

Reviewer comment (Evaluation):
In Figure 2, the authors mention that ''The ranking of various criteria for selective sampling is similar whether with or without Mixup''. However, claiming that the performance between selective sampling is similar to that of selective Mixup seems somewhat strained. For instance, in the case of "Diff. domain+ Same class," selective Mixup demonstrates a $6\%$ higher accuracy than selective sampling. What accounts for the superiority of selective Mixup over selective sampling in this scenario?

Author response:
There are indeed small differences (<5%), which we propose to mention in the caption. We will clarify that the key observation **is correct for the best-performing version** (same domain/diff. class, and resampling for uniform combinations), which is >25% above the baseline and the version that most will care about.

Reviewer comment (Evaluation):
In Figure 2, for ''Same domain'', ''Diff. class'' and ''Diff. domain + Diff. class'', selective sampling is much better than selective Mixup. Does this indicate that vanilla Mixup is harmful in this case? Such observations are more evident in Figure 8. The authors have not discussed the reasons behind the occasional superiority of selective Mixup over selective sampling.

Author response:
Yes indeed. One can see it just by observing that vanilla mixup is worse than the baseline on this dataset. Prior work has indeed shown that mixup is not always beneficial.

We already acknowledge that some cases benefit from selective mixup in the way proposed by Yao et al. For example see Fig.4 (yearbook): "*it indicates a genuine benefit from mixup restricted to pairs of the same class*". This is also exactly what the title of the paper says ("not always"). We propose to highlight this takeaway in the caption of Fig.4.

Reviewer comment (Evaluation):
In Figure 6, given the effective performance of vanilla Mixup, it appears that vanilla Mixup is the main driver for the improvement in selective Mixup, even with the optimal criteria. This observation contradicts the third point outlined in the summarized contributions in the Introduction.

Author response:
The reviewer is absolutely correct. On this dataset, 4/5 of the improvement over the baseline can be attributed to vanilla mixup, and 1/5 to resampling. We propose to tone down the third summary point ("*resampling is SOMETIMES the main driver*").

Reviewer comment (Writing):
Some of the notations are not so clear. Like in Table 1, what's the definition of Resampling (uniform cl.) + concatenated pairs, and why it has different proportion of majority class compared with "Resampling (uniform classes)"?

Author response:
This can indeed use a formal explanation:

- **Resampling (uniform cl.) + mixup** uses paired instances such as:
{$ \textrm{mix}(x_1, \tilde{x}_1), \textrm{mix}(x_2, \tilde{x}_2), \textrm{mix}(x_3, \tilde{x}_3), ... $}
where $x_i$ are sampled with uniform classes (50% proportion) and $\tilde{x}_i$ are sampled indiscriminatively (hence with the original 78% proportion). Overall we get something inbetween (64%).

- **Resampling (uniform cl.) + concatenated pairs** uses the same instances as above without the mixing:
{$ x_1, \tilde{x}_1, x_2, \tilde{x}_2, x_3, \tilde{x}_3, ... $}
The proportion of labels is thus also ~64%.

Reviewer comment (Experiments):
What makes the difference in the sampling ratio between the selective sampling without mixup and resampling? Is this determined by the hyperparameter of mixup? When changing the lambda for the beta distribution of mixup, are similar results as Fig 15 hold?

Author response:
This can indeed use a formal explanation:

- **Resampling (uniform cl.) + mixup** uses paired instances such as:
{$ \textrm{mix}(x_1, \tilde{x}_1), \textrm{mix}(x_2, \tilde{x}_2), \textrm{mix}(x_3, \tilde{x}_3), ... $}
where $x_i$ are sampled with uniform classes (50% proportion) and $\tilde{x}_i$ are sampled indiscriminatively (hence with the original 78% proportion). Overall we get something inbetween (64%).

- **Resampling (uniform cl.) + concatenated pairs** uses the same instances as above without the mixing:
{$ x_1, \tilde{x}_1, x_2, \tilde{x}_2, x_3, \tilde{x}_3, ... $}
The proportion of labels is thus also ~64%.

Yes, we performed experiments on datasets with various $\lambda$ and found very little impact.