Paper: Mixup Your Own Pairs (ICLR-2024)

Reviewer comment (Experiments):
No sufficient comparisons are made to demonstrate the solidness of this work. Since this paper proposed a data augmentation method based on mixup, the performance of SupReMix should be compared with other mixup approaches and not just the baseline SupCon. Besides regular Mixup and manifold Mixup, there are many, many other mixup approaches, for instance, C-Mixup [1], local-Mixup [2], and manifold intrusion [3].

Author response:
We would like to thank the reviewer for the valuable and constructive comments and suggestions. We have now performed new experiments and carefully revised our manuscript to address your concerns. We believe that the quality and clarity of our manuscript has been improved.

$\textbf{a) and  b) More comparisons with Mixup methods and regression learning methods.}$ We thank the reviewer for pointing out the lack of comparison with other Mixup methods and regression learning methods. We firstly would like to hightlight that our SupReMix is a novel approach for $\textbf{representation learning}$ for regression tasks that stands apart from traditional $\textbf{regression learning}$ methods like C-Mixup[1]. As recommended, below we conducted two additional experiments using the AgeDB dataset: 
(1) we evaluated the regression performance of SupReMix (with pre-training followed by a linear probe) in comparison to Mixup methods and regression learning methods (with train from starch) including local-mixup[2], AdaMixup[3], C-Mixup[1], ordinal regression[5] and moving window regression (MWR)[4]. SupReMix has achieved better performance than $\textbf{ALL}$ of them; (from the first two parts of the table)
(2) we demonstrated that integrating SupReMix with regression learning methods (with pretraining followed by linear probe with regression learning methods) can further enhance their performance (from the bottom part of the table below).


 | Metrics                      | MAE ↓    | MSE ↓    | GM ↓    |
| -----------------------------| -------- | -------- | ------- |
| **Regression Learning (Train from scratch)** |
| Vanilla                      | 7.77     | 101.60   | 5.05    |
| local-Mixup                  | 7.89     | 102.33   | 5.19    |
| AdaMixup                     | 7.92     | 103.21   | 5.23    |
| C-Mixup                      | 7.66     | 99.82    | **4.82**|
| Ordinal                      | 7.59     | 97.51    | 4.99    |
| MWR                          | **7.47** | **93.26**| 4.83    |
| **Representation Learning (Linear Probing)** |
| SupCon                       | 8.23     | 113.91   | 5.31    |
| SupCR                        | 7.86     | 102.84   | 5.15    |
| **SupReMix (ours)**          | **7.12** | **87.65**| **4.58**|
| **SupReMix + Regression Learning** |
| SupReMix + Ordinal           | 7.35     | 93.42    | 4.72    |
| **SupReMix + MWR**           | **7.07** | **85.33**| **4.51**|
| SupReMix + C-Mixup           | 7.17     | 90.21    | 4.79    |
| GAINS (**Best Joint** vs. Vanilla (%)) | **+9.0** | **+16.0**| **+10.7**|
| GAINS (**Best Joint** vs. Best RegL (%)) | **+5.2** | **+8.2** | **+6.4** |

We also performed additional performance comparisons between SupReMix with C-Mixup[1] across four different datasets (see the table below). Here we show that SupReMix can **consistently outperform C-Mixup** train from scratch and **bring improvement for C-Mixup** with the SupReMix pretrain (bottom row of the table below).

| Metrics (MAE ↓)     | AgeDB  | IMDB  | HCP   | UKB   |
| -------------------- | ------ | ----- | ----- | ----- |
| C-Mixup              | 7.66   | 5.43  | 9.07  | 3.00  |
| **SupReMix**         | 7.12   | 5.38  | 8.43  | 2.97  |
| **SupReMix + C-Mixup** | **7.17** | **5.21** | **8.23** | **2.90** |
| GAINS (Joint VS. C-Mixup(%)) | **+6.4** | **+4.1** | **+9.3** | **+3.3** |

Reviewer comment (Experiments):
Since the idea of contrastive learning is towards the clustering effect, one would naturally think that applying contrastive learning to regression is not viable. Then it would only make sense to compare SuperReMix to regression frameworks without contrastive learning, for example, Moving Window Regression [4].

Author response:
We would like to thank the reviewer for the valuable and constructive comments and suggestions. We have now performed new experiments and carefully revised our manuscript to address your concerns. We believe that the quality and clarity of our manuscript has been improved.

$\textbf{a) and  b) More comparisons with Mixup methods and regression learning methods.}$ We thank the reviewer for pointing out the lack of comparison with other Mixup methods and regression learning methods. We firstly would like to hightlight that our SupReMix is a novel approach for $\textbf{representation learning}$ for regression tasks that stands apart from traditional $\textbf{regression learning}$ methods like C-Mixup[1]. As recommended, below we conducted two additional experiments using the AgeDB dataset: 
(1) we evaluated the regression performance of SupReMix (with pre-training followed by a linear probe) in comparison to Mixup methods and regression learning methods (with train from starch) including local-mixup[2], AdaMixup[3], C-Mixup[1], ordinal regression[5] and moving window regression (MWR)[4]. SupReMix has achieved better performance than $\textbf{ALL}$ of them; (from the first two parts of the table)
(2) we demonstrated that integrating SupReMix with regression learning methods (with pretraining followed by linear probe with regression learning methods) can further enhance their performance (from the bottom part of the table below).


 | Metrics                      | MAE ↓    | MSE ↓    | GM ↓    |
| -----------------------------| -------- | -------- | ------- |
| **Regression Learning (Train from scratch)** |
| Vanilla                      | 7.77     | 101.60   | 5.05    |
| local-Mixup                  | 7.89     | 102.33   | 5.19    |
| AdaMixup                     | 7.92     | 103.21   | 5.23    |
| C-Mixup                      | 7.66     | 99.82    | **4.82**|
| Ordinal                      | 7.59     | 97.51    | 4.99    |
| MWR                          | **7.47** | **93.26**| 4.83    |
| **Representation Learning (Linear Probing)** |
| SupCon                       | 8.23     | 113.91   | 5.31    |
| SupCR                        | 7.86     | 102.84   | 5.15    |
| **SupReMix (ours)**          | **7.12** | **87.65**| **4.58**|
| **SupReMix + Regression Learning** |
| SupReMix + Ordinal           | 7.35     | 93.42    | 4.72    |
| **SupReMix + MWR**           | **7.07** | **85.33**| **4.51**|
| SupReMix + C-Mixup           | 7.17     | 90.21    | 4.79    |
| GAINS (**Best Joint** vs. Vanilla (%)) | **+9.0** | **+16.0**| **+10.7**|
| GAINS (**Best Joint** vs. Best RegL (%)) | **+5.2** | **+8.2** | **+6.4** |

We also performed additional performance comparisons between SupReMix with C-Mixup[1] across four different datasets (see the table below). Here we show that SupReMix can **consistently outperform C-Mixup** train from scratch and **bring improvement for C-Mixup** with the SupReMix pretrain (bottom row of the table below).

| Metrics (MAE ↓)     | AgeDB  | IMDB  | HCP   | UKB   |
| -------------------- | ------ | ----- | ----- | ----- |
| C-Mixup              | 7.66   | 5.43  | 9.07  | 3.00  |
| **SupReMix**         | 7.12   | 5.38  | 8.43  | 2.97  |
| **SupReMix + C-Mixup** | **7.17** | **5.21** | **8.23** | **2.90** |
| GAINS (Joint VS. C-Mixup(%)) | **+6.4** | **+4.1** | **+9.3** | **+3.3** |

Reviewer comment (Evaluation):
Results in Tab. 1-6 show incremental improvement over the baseline. However, the gain is provided but is w.r.t to vanilla, not second best.

Author response:
**c) Gain w.r.t to the Second Best.** Thank you for highlighting the importance of comparing our results with the second-best method, not just the vanilla baseline. Please note that vanilla is the second-best for some datasets (AgeDB, IMDB, HCP). For clarity, we have added underscores of the second-best performance metrics in Table 1-6.

Reviewer comment (Presentation):
Section 2 seems somewhat out-of-place. It doesn't quite follow naturally from Section 1, and it is not entirely clear what the main takeaway should be or how it connects to the rest of the paper.

Author response:
For Section 2, the main motivation is to show that the direct application of contrastive learning for classification to regression can only learn **discontinuous representations that are insensitive to label permutation** (Section 2.1) and **swift saturation** of training with conventional positive/negative contrastive pairs (Section 2.2). ("various interesting empirical studies of regression tasks, e.g., visualizations of logit distribution and latent space. These findings are supportive of the proposed method and might be inspiring for designing better algorithms for regression tasks." from reviewer 4GQV). Section 2.1 illustrates that SupCon is deficient in ordinality-awareness as it consistently generates embeddings that exhibit discrete clustering patterns, regardless of whether the labels are genuine or permuted. In Section 2.2, we observe that SupCon also lacks in presenting sufficient hardness; notably fewer contrastive pairs contribute to training during the middle or late stages. These observations directly relate to the two key aspects we previously highlighted: ordinality-awareness and hardness. To address these issues, we introduce Mix-pos and Mix-neg with weights. This enhancement aims to equip the model with a better grasp of ordinality and to amplify the level of hardness in contrastive pairs. We have modified the text to make it more explicit.

Reviewer comment (Presentation):
Equation (1) and (3) are difficult to read/parse as they are quite "dense".

Author response:
For equation (1) and equation (3), we have made some modifications to **parse them** further for better readability. We change the previous set of expressions into:

$z_{m,i}^{-} = \lambda_1 \cdot z_{m,i} + (1-\lambda_1) \cdot z', \quad
\text{where} \ z' \in I\backslash I_m, \ \lambda_1 \sim \text{Beta}(\alpha, \beta), \
\text{and} \ \overline{m} = \lambda_1 \cdot m + (1-\lambda_1) \cdot m'$ and
$z_{m,i}^{+} = \lambda_2 \cdot z_{m',i'} + (1-\lambda_2) \cdot z_{m'',i''}, \quad
\text{where} \ i'<i<i'', \ i-i', i''-i \leq \gamma, \
\text{and} \ \lambda_2 \cdot m' + (1-\lambda_2) \cdot m'' = m$

Reviewer comment (Theory):
It is not clear what the theoretical results in Section 3.4 are supposed to tell me, or what should be my main takeaway.

Author response:
The theoretical results in Section 3.4 are "theoretical explanations" (reviewer 4GQV) for our SupReMix loss function. In Theorem 1, the key insight is that SupReMix amplifies the penalty for more distant negative pairs compared to closer ones by assigning specific weights to these pairs. Meanwhile, Lemma 1 and Theorem 2 collectively highlight that the SupReMix loss function not only possesses a tight bound but also ensures both the ordinality and continuity of representations as they converge towards the infimum. We have concisely summarized the main takeaway in Section 3.4 to more effectively communicate the key message of this section.

Reviewer comment (Experiments):
The proposed method is only applied to regression problems with 1D targets. It is not clear whether or not it could be extended.

Author response:
Thank you for this important feedback. Firstly, even if ordinality has been shown to be important in regression tasks, it is undefined for vectors in dimensions greater than one. The topology on $\mathbb{R}^n$ does not constitute an order topology, making the application of our Mix-pos method to outputs in dimensions greater than one non-trivial. In Mix-pos, obtaining a weights vector (whose dimension matches that of the label) by solving a linear system is theoretically feasible. However, this solution is not always assured due to the potential linear independence between selected samples and the anchor. And it is not scalable when increasing the dimension of regression labels. Future research could focus on exploring methods to preserve ordinality in regression representations when dealing with higher-dimensional labels. However, our Mix-neg approach and weights design can be straightforwardly applied to higher-dimensional settings. To illustrate this, we have included additional experiments to perform regression in gaze prediction (two outputs), demonstrating that our Mix-neg and weight adjustments can lead to significant improvements. We have added one more limitation section (Section E) in our submission to discuss this point.

Below we show the performance comparison of SupReMix (w/o Mix-pos) with Vanilla and other contrastive learning counterparts (SupCon and SupCR). We carried out gaze estimation tasks with the subsampled (due to the time limitation) MPIIFaceGaze dataset to evaluate the effectiveness of SupReMix. In terms of the calculation of label distance, we use cosine distance since the dimension of the target is larger than one. (More details about the experiment are in Appendix D.7.) We show that SupReMix significantly outperforms Vanilla and other contrastive learning methods.

| Metrics               | Angular ${\downarrow}$ | $R^{2}$ ${\uparrow}$ |
| ----------------------| ----------------------- | --------------------- |
| Vanilla               | 5.72                    | 0.774                 |
| SupCon                | 6.01                    | 0.767                 |
| SupCR                 | 5.48               | 0.801             |
| **SupReMix (w/o Mix-pos)** | **5.14**      | **0.827**             |
| GAINS (**Ours** VS. Vanilla(%)) | **+10.1** | **+6.8** |
| GAINS (**Ours** VS. SupCR(%))   | **+4.2**  | **+3.2** |

Reviewer comment (Experiments):
The proposed method is only applied to datasets where the regression targets take on a relatively small set of different values (see Table 12), for example age in the interval [0, 100] with a bin size of 1. It is not entirely clear whether or not the method is somehow limited to such regression problems.

Author response:
"Could it be applied also to problems with 'truly continuous' regression targets?" The answer is a definite **YES**. SupReMix could be adapted to **ANY** bin size within the target range. In fact, smaller bin sizes are particularly advantageous as they promote greater continuity in the representations. This is due to the increased diversity of candidates available for the mixup process. Crucially, SupReMix does not depend on having samples with identical labels. It functions effectively even in scenarios where continuous variation is extreme, such as cases where each sample has a unique label. For Mix-neg, the anchor is combined with all other samples that bear different labels. In the case of Mix-pos, samples within a specified window and possessing distinct labels from the anchor are chosen for mixup. An important feature of this approach is its ability to always find a positive weight that combines a larger and a smaller label to match the anchor's label. This is achieved by simply solving a linear equation with one variable. The design of SupReMix thus ensures both flexibility and precision in handling a wide range of label variations. The bin sizes selected for our experiments were determined by the inherent characteristics of each dataset, independent of the design of our model.

Reviewer comment (Experiments):
Could the proposed method be extended also to regression problems where the targets are multi-dimensional?

Author response:
Thank you for this important feedback. Firstly, even if ordinality has been shown to be important in regression tasks, it is undefined for vectors in dimensions greater than one. The topology on $\mathbb{R}^n$ does not constitute an order topology, making the application of our Mix-pos method to outputs in dimensions greater than one non-trivial. In Mix-pos, obtaining a weights vector (whose dimension matches that of the label) by solving a linear system is theoretically feasible. However, this solution is not always assured due to the potential linear independence between selected samples and the anchor. And it is not scalable when increasing the dimension of regression labels. Future research could focus on exploring methods to preserve ordinality in regression representations when dealing with higher-dimensional labels. However, our Mix-neg approach and weights design can be straightforwardly applied to higher-dimensional settings. To illustrate this, we have included additional experiments to perform regression in gaze prediction (two outputs), demonstrating that our Mix-neg and weight adjustments can lead to significant improvements. We have added one more limitation section (Section E) in our submission to discuss this point.

Below we show the performance comparison of SupReMix (w/o Mix-pos) with Vanilla and other contrastive learning counterparts (SupCon and SupCR). We carried out gaze estimation tasks with the subsampled (due to the time limitation) MPIIFaceGaze dataset to evaluate the effectiveness of SupReMix. In terms of the calculation of label distance, we use cosine distance since the dimension of the target is larger than one. (More details about the experiment are in Appendix D.7.) We show that SupReMix significantly outperforms Vanilla and other contrastive learning methods.

| Metrics               | Angular ${\downarrow}$ | $R^{2}$ ${\uparrow}$ |
| ----------------------| ----------------------- | --------------------- |
| Vanilla               | 5.72                    | 0.774                 |
| SupCon                | 6.01                    | 0.767                 |
| SupCR                 | 5.48               | 0.801             |
| **SupReMix (w/o Mix-pos)** | **5.14**      | **0.827**             |
| GAINS (**Ours** VS. Vanilla(%)) | **+10.1** | **+6.8** |
| GAINS (**Ours** VS. SupCR(%))   | **+4.2**  | **+3.2** |

Reviewer comment (Experiments):
Is the proposed method somehow limited to regression problems such as age estimation, in which the targets take on a relatively small discrete set of values? Could it be applied also to problems with "truly continuous" regression targets? What happens if, within a given batch, it is not possible to find examples with exactly the same regression target as the anchor?

Author response:
"Could it be applied also to problems with 'truly continuous' regression targets?" The answer is a definite **YES**. SupReMix could be adapted to **ANY** bin size within the target range. In fact, smaller bin sizes are particularly advantageous as they promote greater continuity in the representations. This is due to the increased diversity of candidates available for the mixup process. Crucially, SupReMix does not depend on having samples with identical labels. It functions effectively even in scenarios where continuous variation is extreme, such as cases where each sample has a unique label. For Mix-neg, the anchor is combined with all other samples that bear different labels. In the case of Mix-pos, samples within a specified window and possessing distinct labels from the anchor are chosen for mixup. An important feature of this approach is its ability to always find a positive weight that combines a larger and a smaller label to match the anchor's label. This is achieved by simply solving a linear equation with one variable. The design of SupReMix thus ensures both flexibility and precision in handling a wide range of label variations. The bin sizes selected for our experiments were determined by the inherent characteristics of each dataset, independent of the design of our model.

Reviewer comment (Evaluation):
Section 4, Evaluation Schemes: The linear probing protocol is used for 4 out of 6 datasets. Any particular reason why? Why not for all 6? Or, why do you finetune the whole network for those 2 particular datasets?

Author response:
For the neuroimaging datasets (HCP and UKB), we followed the previous work [1] and [2] that fine-tuned the whole network.

Reviewer comment (Presentation):
Table 9 and 10 are interesting results, but for which dataset(s) are these results?

Author response:
Table 9 was based on AgeDB. We have added the information in the tables. Table 10 was based on AgeDB and STS-B (indicated in the header row).

Reviewer comment (Writing):
Section 1, "its dependency on data augmentation restricts its applicability to domains where effective augmentation techniques are lacking": This should be the other way around — the applicability is restricted to domains where effective augmentation methods do exist.

Author response:
As suggested, we have changed the sentence to "Nevertheless, its dependency on data augmentation restricts its applicability to domains where effective augmentation techniques have been established, such as time series or tabular data."

Reviewer comment (Writing):
Section 3, "which encodes input to embeddings": should be "input" → "inputs".

Author response:
We have changed the "input" to "inputs," thank you for pointing this out.

Reviewer comment (Presentation):
The paper title could be modified to be more descriptive, as the current title gives no clear clue that the paper studies the important problem of representation learning for regression.

Author response:
As suggested, we have changed our title to "Mixup Your Own Pairs: Supervised Contrastive Learning for Regression with Mixup." Thank you for your excellent suggestion.

Reviewer comment (Experiments):
I think the biggest weakness is the lack of comparisons with related works in regression -- most notably C-Mixup (Yao et al., NeurIPS 2022) -- and vanilla mixup as well, as it has been shown to be a competitive baseline even on regression. I think they may be better than the ERM (vanilla) baseline in the experiments. Besides, the C-mixup method is also closely related in the sense that they choose mixup pairs that are close in function value (albeit without considering ordinality) so its worth a deeper comparison even methodologically. It might provide some insights into the strengths of SupReMix. For example, does adding ordinality with vanilla mixup or C-mixup boost performance of each method?

Author response:
**More Comparisons (C-Mixup):**
Thank you for your constructive comment. As recommended, we have added performance comparison with C-Mixup (training-from-scratch) on four datasets. Moreover, as SupReMix is a representation learning method, it can **jointly** work with C-Mixup by using SupReMix pretraining and C-Mixup during the linear probe. We demonstrate that, compared to training from scratch, utilizing our SupReMix pre-training consistently yields improvements.

| Metrics (MAE↓) | AgeDB | IMDB | HCP | UKB |
| -------------- | ----- | ---- | --- | --- |
| C-Mixup | 7.66 | 5.43 | 9.07 | 3.00 |
| **SupReMix** | **7.12** | 5.38 | 8.43 | 2.97 |
| **SupReMix + C-Mixup** | 7.17 | **5.21** | **8.23** | **2.90** |
| GAINS (Joint VS. C-Mixup(%)) | **+6.4%** | **+4.1%** | **+9.3%** | **+3.3%** |

**Pair Selection Strategy:**
We appreciate your in-depth advice on the relation of SupReMix with C-Mixup in terms of "pair selection strategy". In short, our pair selection strategy is in the same direction but different from C-Mixup's. Although both of us consider the label distance, C-Mixup will only put a higher probability of mixup for pairs with smaller label distances. SupReMix, as a contrastive learning method, treats positive pairs mixup (Mix-pos) and negative pairs mixup (Mix-neg) differently. More specifically, SupReMix takes a window size (γ) of label distance for Mix-pos and leaning-toward-anchor beta distribution for sampling anchor-inclusive pairs (Mix-neg). Mix-Pos strongly depends on local linearity assumption, therefore we do not consider samples with large label distance (even with low probability) for hard positive mixtures. For performance comparison, we replaced our pair selection strategy with C-Mixup's and experimented on the AgeDB dataset. We have shown that our proposed approach still outperforms C-Mixup (see the table below).

| Mixup Strategy | MAE↓ | MSE↓ | GM↓ |
| -------------- | ---- | ---- | --- |
| C-Mixup | 7.22 | 92.68 | 4.83 |
| Ours | **7.12** | **87.65** | **4.58** |

Reviewer comment (Reproducibility):
The framework fundamentally relies on ordering the training data according to function value -- how is this implemented in practice? Is the ordering done on each mini batch?

Author response:
**No Ordering is Needed:**
Thanks for raising this question. We would like to clarify that our method does not require a specific order for selecting contrastive pairs. For hard negatives, the anchor is mixed with all other samples in the same mini-batch that have differing labels. Regarding hard positives, the selection process involves identifying samples within a predefined γ window. This is achieved by first calculating the label distance between the anchor and all other samples. We then select the samples that fall within the window. This process can be efficiently implemented through matrix subtraction and identifying the minimum values.

Reviewer comment (Reproducibility):
Are the backbone networks trained from scratch or do you use pre-trained variants followed by fine-tuning?

Author response:
**All the backbone networks were trained from scratch (if you refer to the contrastive learning stage itself).**

Reviewer comment (Experiments):
A limitation that is unaddressed here, is that the current method does not generalize to even simpler regression problems where the output function has dimensionality $>1$.

Author response:
**Higher Dimensional Output:**
Thank you for this important feedback. Firstly, even if ordinality has been shown to be important in regression tasks, it is undefined for vectors in dimensions greater than one. The topology on $\mathbb{R}^n$ does not constitute an order topology, making the application of our Mix-pos method to outputs in dimensions greater than one non-trivial. In Mix-pos, obtaining a weights vector (whose dimension matches that of the label) by solving a linear system is theoretically feasible. However, this solution is not always assured due to the potential linear independence between selected samples and the anchor. And it is not scalable when increasing the dimension of regression labels. Future research could focus on exploring methods to preserve ordinality in regression representations when dealing with higher-dimensional labels. However, our Mix-neg approach and weights design can be straightforwardly applied to higher-dimensional settings. To illustrate this, we have included additional experiments to perform regression in gaze prediction (two outputs), demonstrating that our Mix-neg and weight adjustments can lead to significant improvements. We have added one more limitation section (Section E) in our submission to discuss this point.

Below we show the performance comparison of SupReMix (w/o Mix-pos) with Vanilla and other contrastive learning counterparts (SupCon and SupCR). We carried our gaze estimation tasks with the subsampled (due to the time limitation) MPIIFaceGaze dataset to evaluate the effectiveness of SupReMix. In terms of the calculation of label distance, we used cosine distance since the dimension of the target is larger than one. (More details about the experiment are in Appendix D.7.) We show that SupReMix **significantly outperforms** Vanilla and other contrastive learning methods.

| Metrics | Angular↓ | R²↑ |
| ------- | -------- | --- |
| Vanilla | 5.72 | 0.774 |
| SupCon | 6.01 | 0.767 |
| SupCR | 5.48 | 0.801 |
| **SupReMix (w/o Mix-pos)** | **5.14** | **0.827** |
| GAINS (Ours VS. Vanilla(%))| **+10.1** | **+6.8** |
| GAINS (Ours VS. SupCR(%))| **+4.2** | **+3.2** |

Reviewer comment (Writing):
Finally, I find the notational style in the paper is a bit confusing and not easy to read. It may help the reader to simplify or modify the notation to benefit legibility .. for e.g. indices, function values, embeddings are all in italics which is not typical and hard to distinguish. Please fix these.

Author response:
**Notations:**
Thank you for your advice. We have made the following modifications to enhance clarity and conformity:
(1) The text in equations has been altered from italic to normal font. For instance, the word "with" in Theorem 1 now appears in standard font. (2) Vectors are now in bold, aligning with common mathematical practice, e.g., vector **z** used for embeddings. (3) The function notation g has been replaced with a Greek letter ρ.

Reviewer comment (Presentation):
Figures 1 A and B, do not convey the message of the method clearly.. and is a bit confusing. I think this can be improved to make the proposed method more explicit.

Author response:
**Figures:**
Thank you for your suggestion to improve the figure of schematic overview. We have made modifications to make the proposed method more explicit: (1) In panel A, we use red and green to color code the Mix-Pos and Mix-Neg; (2) In panel B, we change the names of our hard pairs to "Mix-Pos" and "Mix-Neg", to be consistent with panel A.

Reviewer comment (Novelty):
(**W1**) Lack of novelty compared to existing methods. The studied problem and proposed method are not novel enough. As known to all, the hard sample mining problem has been explored since 2020, there are also some relevant methods for regression methods (SupCon, SupCR, Decoupled CL [1]). Meanwhile, the proposed SupReMix utilizes similar strategies to design hard samples as MoCHi [2] and i-Mix [3] for contrastive pre-training and uses similar anchor and sample selection strategies as C-Mixup [4].

Author response:
We appreciate the reviewer's assessment regarding the novelty of our proposed method. It provides us with an opportunity to clarify the contributions and innovative aspects of our work. The novelty of our work can be summarized in the following three key aspects:

**(1) Contribution to an Underexplored Domain:**
We wish to highlight that the majority of previously proposed hard sample mining methods, including those like SupCon and Decoupled CL you mentioned, predominantly focus on classification rather than **regression**. The direct application of them to regression leads to sub-optimal performance due to lack of ordinality-awareness and hardness. Regression is "a domain to which a lot of our insights and beliefs from classification may not generalize" (reviewer gvot) and "an important and interesting problem that I think should be studied more" (reviewer DdrH). "The idea of applying contrastive learning to regression is interesting" (reviewer ggUN). To our knowledge, SupCR is the only method centered on regression (We have thoroughly compared with it and provided analysis in our submission even if it was an Arxiv before submission), yet it faces a significant limitation: the necessity for data augmentation, which may not be suitable for modalities like time series or tabular data. SupReMix stands out as the first supervised contrastive learning framework for regression that does **NOT rely on data augmentation** ("This is significant" from reviewer gvot). Furthermore, earlier strategies such as MoCHi and i-Mix were implemented in a self-supervised manner, not in **supervised** training. Our approach utilizes regression labels to foster ordinality and continuity, aspects that were overlooked in previous methods.

**(2) Technical Novelty:**
The prior methods in contrastive learning and hard sample mining, including all those you've mentioned like SupCon, SupCR, Decoupled CL, MoCHi, and i-Mix, primarily focus on identifying hard negatives, leaving the concept of **hard positives** largely unexplored. We are grateful for your recognition of our "various interesting empirical studies of regression tasks, including visualizations of logit distribution and latent space." These studies underline the importance of incorporating hard positives in regression analysis. Our approach introduces the novel hard positive pairs in regression by combining two negative samples in such a way that their convex combination of labels matches the anchor's label. It aims to promote local linearity. SupReMix represents a groundbreaking approach in addressing the issue of hard positives in the regression context and proposing a solution leading to superior regression performance. Even in the broader scope of contrastive learning in general, hard positives remain an underexplored area. Wu et al. 2023, as cited in our related work, appears to be the only study delving into this concept, albeit in a self-supervised learning framework. On the other hand, SupReMix is the first contrastive learning framework that incorporates **label distance** into contrastive loss, providing a more fundamental approach to handle negative pairs, and it is supported by the concept of Distance Magnifying we proposed and theoretically analyzed.

**(3) Theoretical Contribution:**
SupReMix is supported by two theorems and a lemma we established, offering novel insights in supervised contrastive learning for regression. We demonstrate that by assigning weights to contrastive pairs based on label distance, our approach amplifies the penalty for negative pairs that are more distant relative to nearer ones. Besides, we establish that the SupReMix loss function has a tight bound, and it guarantees the ordinality and continuity of representations as it approaches the infimum.

Reviewer comment (Experiments):
(**W2**) Weak experiments. Despite the authors providing extensive comparison results with two directly relevant contrastive learning methods, more related baselines should be compared in various experimental settings. Firstly, the authors adopt the implementation from C-Mixup [4] while not comparing it with it. I suggest the authors add more comparison experiments with existing mixup methods for regression (e.g., Vanilla Mixup, C-Mixup [4], ManifoldMix [5], etc) and general hard sample mining contrastive pre-training methods with mixup augmentations (e.g., MoCHi [2] and i-Mix [3]).

Author response:
**Comparisons with related work (W2):** We appreciate the important comment. We firstly would like to highlight that our SupReMix is a novel approach for $\textbf{representation learning}$ for regression tasks that stands apart from traditional $\textbf{regression learning}$ methods like C-Mixup[4]. As recommended, we conducted two new experiments to validate the effectiveness of SupReMix: (1) we evaluated the regression performance of SupReMix (with pre-training followed by a linear probe) in comparison to methods (both training-from-scratch and pretrain + linear probe) including MoCHi[2], i-Mix[3], C-Mixup[4], Vanilla Mixup, and M-Mixup[6]. SupReMix has achieved better performance than **ALL** of them using AgeDB dataset (see the top two parts of the table below); (2) we demonstrated that integrating SupReMix with C-Mixup can further enhance C-Mixup's performance (bottom part of the table below).

| Metrics | MAE ↓ | MSE ↓ | GM ↓ |
| ------- | ----- | ----- | ---- |
| **Regression Learning (Train from scratch)** | | | |
| Vanilla | 7.77 | 101.60 | 5.05 |
| Mixup | 8.03 | 119.51 | 5.78 |
| M-Mixup | 8.02 | 104.33 | 5.34 |
| C-Mixup | 7.66 | 99.82 | 4.82 |
| **Representation Learning (Linear Probe)** | | | |
| MoCHi | 9.07 | 132.50 | 5.93 |
| i-Mix | 8.50 | 117.96 | 5.52 |
| SupCon | 8.23 | 113.91 | 5.31 |
| SupCR | 7.86 | 102.84 | 5.15 |
| **SupReMix (ours)** | **7.12** | **87.65** | **4.58** |
| **SupReMix + Regression Learning** | | | |
| SupReMix + C-Mixup | 7.17 | 90.21 | 4.79 |

We also performed additional performance comparisons between SupReMix with C-Mixup[4] across four different datasets (see the table below). Here we show that SupReMix can **consistently outperform C-Mixup** train from scratch and **bring improvement for C-Mixup** with the SupReMix pretrain (bottom row of the table below).

| Metrics (MAE ↓) | AgeDB | IMDB | HCP | UKB |
| --------------- | ----- | ---- | --- | --- |
| C-Mixup | 7.66 | 5.43 | 9.07 | 3.00 |
| **SupReMix** | **7.12** | 5.38 | 8.43 | 2.97 |
| **SupReMix + C-Mixup** | 7.17 | **5.21** | **8.23** | **2.90** |
| GAINS (**Joint** VS. C-Mixup (%)) | **+6.4** | **+4.1** | **+9.3** | **+3.3** |

Reviewer comment (Experiments):
(**W2**, continued) The authors should not only evaluate on self-supervised pre-training and fine-tuning setting but should conduct the practical training-from-scratch setting as well.

Author response:
**Comparisons with related work (W2):** We appreciate the important comment. We firstly would like to highlight that our SupReMix is a novel approach for $\textbf{representation learning}$ for regression tasks that stands apart from traditional $\textbf{regression learning}$ methods like C-Mixup[4]. As recommended, we conducted two new experiments to validate the effectiveness of SupReMix: (1) we evaluated the regression performance of SupReMix (with pre-training followed by a linear probe) in comparison to methods (both training-from-scratch and pretrain + linear probe) including MoCHi[2], i-Mix[3], C-Mixup[4], Vanilla Mixup, and M-Mixup[6]. SupReMix has achieved better performance than **ALL** of them using AgeDB dataset (see the top two parts of the table below); (2) we demonstrated that integrating SupReMix with C-Mixup can further enhance C-Mixup's performance (bottom part of the table below).

| Metrics | MAE ↓ | MSE ↓ | GM ↓ |
| ------- | ----- | ----- | ---- |
| **Regression Learning (Train from scratch)** | | | |
| Vanilla | 7.77 | 101.60 | 5.05 |
| Mixup | 8.03 | 119.51 | 5.78 |
| M-Mixup | 8.02 | 104.33 | 5.34 |
| C-Mixup | 7.66 | 99.82 | 4.82 |
| **Representation Learning (Linear Probe)** | | | |
| MoCHi | 9.07 | 132.50 | 5.93 |
| i-Mix | 8.50 | 117.96 | 5.52 |
| SupCon | 8.23 | 113.91 | 5.31 |
| SupCR | 7.86 | 102.84 | 5.15 |
| **SupReMix (ours)** | **7.12** | **87.65** | **4.58** |
| **SupReMix + Regression Learning** | | | |
| SupReMix + C-Mixup | 7.17 | 90.21 | 4.79 |

We also performed additional performance comparisons between SupReMix with C-Mixup[4] across four different datasets (see the table below). Here we show that SupReMix can **consistently outperform C-Mixup** train from scratch and **bring improvement for C-Mixup** with the SupReMix pretrain (bottom row of the table below).

| Metrics (MAE ↓) | AgeDB | IMDB | HCP | UKB |
| --------------- | ----- | ---- | --- | --- |
| C-Mixup | 7.66 | 5.43 | 9.07 | 3.00 |
| **SupReMix** | **7.12** | 5.38 | 8.43 | 2.97 |
| **SupReMix + C-Mixup** | 7.17 | **5.21** | **8.23** | **2.90** |
| GAINS (**Joint** VS. C-Mixup (%)) | **+6.4** | **+4.1** | **+9.3** | **+3.3** |

Reviewer comment (Experiments):
(**W3**) Usage of hyper-parameters. Firstly, many hyper-parameters are used in SupReMix compared to SupCon and mixups for regression tasks. The most widely used mixup augmentations are

Author response:
**Hyper-parameters (W3):**
We appreciate this useful point. To clarify, SupReMix introduces **ONLY ONE** additional hyper-parameter: window size (denoted as $\gamma$) for Mix-pos. This parameter is chosen based on the range of regression targets. Table 10 demonstrates that $\gamma$ is arguably robust: moderate ranges of $\gamma$ consistently enhance performance. Another consideration may be related to the beta distribution for Mix-neg. In Table 11, by experimenting with different shapes of distribution on six datasets with different modalities for different tasks, beta(2,8) consistently outperforms other variants. It indicates that as long as the level of difficulty is maintained at a certain level, i.e., the beta distribution leans towards the anchor, the algorithm can perform the task robustly.

Reviewer comment (Novelty):
(**W4**) Overlook of some related works. As mentioned above, some general and relevant mixup augmentations [3, 5, 6, 7] should be included as background knowledge. Meanwhile, relevant hard sample mining techniques [2, 5, 7] in contrastive learning should be included and discussed in the related work section.

Author response:
**Related Work (W4):**
Thank you for pointing out these useful references. As recommended, we have included the following sentences to highlight these studies and clarify the research gaps. (In Introduction, second paragraph).

"... Nevertheless, they mainly focused on hard negatives with hard positives underexplored, and the hardness of contrastive pairs in contrastive learning for regression remains inadequately examined. Data mixing techniques (Zhang et al., 2018; Verma et al., 2019; Shen et al., 2022), commonly used for data augmentation, have been used to create hard samples in previous contrastive learning methods for classification (Kalantidis et al., 2020; Lee et al., 2020; Liu et al., 2023). However, these methods do not leverage the label distance to differentiate the hardness among hard negative mixtures. ..."