Paper: FragSel: Fragmented Selection for Noisy Label Regression (ICLR-2024)

Reviewer comment (Novelty):
Some highly related references in noisy label learning are missing. For example, the transition matrix methods [3-5], and the hybrid methods [6,7].

Author response:
**W2. Include Transition matrix and Hybrid methods in related works.**

We appreciate the valuable insight of the related works! We appropriately cited them within the main body of the manuscript and dedicated additional sections for Transition Matrix, Object detection, and Hybrid Methods in the Appendix.

Reviewer comment (Reproducibility):
The description of the proposed algorithm procedure and the experiment setting can be introduced more clearly. I have some questions and suggestions: 1) Are the prediction-based or representation-based sample selections used together in the proposed method? If not, when to use the prediction-based sample selection, and when to representation-based one? 2) How to inject symmetric label noise in regression labels? 3) The pseudo-code of the proposed algorithm will help a lot for the readers who want to understand the detailed design.

Author response:
**W3Q1. Clarify the combining of prediction and representation-based selections.**

We revise our notation to follow our Ablation study in Appendix E.6 (Table 5) by introducing $S^p$ and $S^r$, where $p$ represents prediction and $r$ represents regression. In FragSel, we utilize the union of $S^p$ and $S^r$, denoted as $S^p \cup S^r$.

**W3Q2. How to inject symmetric label noise in regression labels?**

We adhere to the standard classification setting’s symmetric label noise [1], where a percentage of samples are uniformly flipped into other labels.

[1] Yi, K., & Wu, J. Probabilistic end-to-end noise correction for learning with noisy labels. In CVPR, 2019.

**W3Q3. Pseudo-code of FragSel**

As per the reviewer’s suggestion, we include a pseudo-code of the proposed algorithm in the Appendix.

Reviewer comment (Presentation):
Figure 1 is hard to understand. It includes too much information that has not yet been explained.

Author response:
**W1. Figure 1 Update**

We thank the reviewer for the insight! We have relocated Figure 1 and incorporated supplementary information to improve reader comprehension.

Reviewer comment (Novelty):
I think it is already quite well known that samples with similar features tend to exhibit similar labels, and many studies have assumes that properties; the validity of Semi supervised learning, pseudo labeling stems from this assumption. Therefore, I think that the novelty of this paper may be limited from several previous sample selection based methods, since I think the method proposed in this paper is the combinations of the previous studies (suggested in the classification task).

Author response:
**W2. Label Feature Correlation is widely assumed, is FragSel a combination of previous studies from noisy label classification?**

We concur that the assumption regarding the correlation between labels and features has been previously investigated in various domains, which is the reason why we dealt with a detailed section as an extended related section in the Appendix (C.1)! However, given the extensive prior research on noisy labeled learning within the context of classification, we wish to emphasize its significance as the primary characteristic to be researched when developing a solution for handling noisy labels in regression tasks.

The terms 'Neighborhood' and 'Contrasting' are prevalent keywords within the extensive body of literature on noisy label classification [1-11], and their prominence is well-justified, given their effectiveness in selecting confident samples from uncalibrated neural network outputs. However, their application in the context of noisy labeled regression remains unexplored. In addition, we present several notable distinctions from previous research. Most importantly, every unique aspect of FragSel stems from its foundational reliance on **contrastive fragments**. It is what leads to the independent contrastive training as well as the mixture-based probabilistic selection framework.

- Specifically, the mixture-based probabilistic framework of neighboring contrastive representations uniquely consists of a prior weighting based on relative distances amongst the mixtures (fragments), as well as a two-part agreement (self & neighboring) within the single mixture as well as amongst them.

- The contrastive fragment-based training of the representations results in improved neighbor representations and, consequently, enhanced sampling

[1] Li, J., et al. Neighborhood collective estimation for noisy label identification and correction. ECCV, 2022.

[2] Zhu, Z., et al. Detecting corrupted labels without training a model to predict. ICML, 2022.

[3] Shao, H. C., et al. Ensemble learning with manifold-based data splitting for noisy label correction. IEEE Transactions on Multimedia, 2022.

[4] Wu, P., et al. A topological filter for learning with label noise. NeurIPS, 2020.

[5] Iscen, A., et al. Learning with neighbor consistency for noisy labels. CVPR, 2022.

[6] Xu, R., et al. Neighborhood-regularized self-training for learning with few labels. AAAI, 2023.

[7] Li, S., et al. Selective-supervised contrastive learning with noisy labels. CVPR, 2022.

[8] Li, J., et al. Learning from noisy data with robust representation learning. ICCV, 2021.

[9] Ortego, D., et al. Multi-objective interpolation training for robustness to label noise. CVPR, 2021.

[10] Zhang, X., et al. Codim: Learning with noisy labels via contrastive semi-supervsied learning. arXiv preprint arXiv:2111.11652, 2021.

[11] Huang, Z., et al. Twin contrastive learning with noisy labels. CVPR, 2023.

Reviewer comment (Theory):
I cannot understand yet why the method the authors suggests fits especially for the regression task. Can't it be applied to classification task?

Author response:
**W3. Why use FragSel only on Regression and not Classification?**

We focus on the task of noisy label regression because it is a recurring challenge in commercial applications and possesses unique characteristics compared to classification. However, while extending this approach to classification is feasible, it may necessitate specific approximations and the incorporation of prior knowledge to implement contrastive fragmentation based on the label-feature relationship assumption. One avenue worth investigating involves leveraging the CLIP model [1] to acquire label embeddings for the purpose of quantifying the distance relationship within the label space.

[1] Radford, A., et al. Learning transferable visual models from natural language supervision. In ICML, 2021.

Reviewer comment (Theory):
It is known that data points with similar features tend to exhibit similar label values. However, including noisy labeled data samples, it corrupts these similarity the model learns because the model tries to fit all data samples, which is also the problem of learning noisy data. Therefore, for managing noisy data, can we use the similar feature-similar label property as it is? Or should we use additional tricks for managing the problem?

Author response:
**Q1. Does label feature correlation still hold when learning with noisy data? Should we use additional tricks?**

We also acknowledge that the similarity relations learned by the model can be susceptible to disruptions. Therefore, the adoption of a robust algorithm holds paramount significance [1, 2]!

We gently remind that FragSel effectively addresses this challenge through a multifaceted approach. This encompasses data fragmentation to facilitate stable training via cross-entropy loss, the utilization of contrastive pairing, the incorporation of Mixture of Neighbor agreements, and the introduction of jittering techniques to enhance regularization.

Furthermore, the incorporation of "additional tricks" can prove advantageous. Notably, FragSel exhibits compatibility with various existing techniques, as demonstrated in Table 4, including SCE, co-teaching, and cmixup.

[1] Zhang, C., et al. Understanding deep learning requires rethinking generalization. In ICLR, 2017.

[2] Li, J., et al. How does a neural network’s architecture impact its robustness to noisy labels?. In NeurIPS, 2021.

Reviewer comment (Theory):
Why should we select samples with framentation? (empirically, okay. Any theoretical idea?)

Author response:
**Q2. Theory of Fragmentation**

FragSel operates by partitioning data samples into fragments and leveraging trained feature extractors for sample selection through collective modeling. This approach can be interpreted as a theoretically well-grounded Mixture-of-Experts (MoE), in which individual experts focus on specific subspaces of the problem through data partitioning [1, 2]. Through fragmentation, we can take MoE’s advantages in terms of computational scalability, handling mixed-type data, and reducing output variance [1]. It is worth highlighting that, as each network is trained on a distinct training set, MoE can effectively circumvent concurrent failures, thereby preventing error propagation among networks and ultimately enhancing generalization performance [3].

In addition to the theoretical benefits associated with MoE, FragSel also addresses the issue of feature extractors memorizing incorrect labels when dealing with the noisy label problem. This is achieved by incorporating both self and neighbor agreements (Eq. 5), which bolsters robustness by preventing coincidental failures among feature extractors.

Furthermore, the use of contrastive pairing for fragments enhances the training of feature extractors. Maximizing the distances between paired fragments ensures a substantial margin between their representations, which is pivotal for guiding robust training and promoting model generalizability [4, 5, 6]. While FragSel capitalizes on the distinctiveness of contrastiveness between labels, it's important to note that there are other well-grounded approaches that leverage label correlations, such as pairing labels in multi-label classification [7, 8], constructing label relation graphs [9], or employing label embedding techniques [10].

Moreover, the process of fragmentation introduces the possibility of transforming a regression problem into a classification problem. As theoretically analyzed in [11], while Mean Squared Error (MSE) loss overlooks the marginal entropy of representation, Cross-Entropy (CE) loss maximizes it. Consequently, classification offers a more stable training paradigm compared to regression. For a comprehensive understanding of these concepts, please refer to the response to Q3 from reviewer 3QwR for detailed explanations.

[1] Yuksel, S. E., et al. Twenty years of mixture of experts. IEEE transactions on neural networks and learning systems, 2012.

[2] Masoudnia, S., & Ebrahimpour, R. Mixture of experts: a literature survey. Artificial Intelligence Review, 2014.

[3] Sharkey, A. J., & Sharkey, N. E. Combining diverse neural nets. The Knowledge Engineering Review, 1997.

[4] Shawe-Taylor, J., & Cristianini, N. Robust bounds on generalization from the margin distribution. 1998.

[5] Grønlund, A., et al. Margin-based generalization lower bounds for boosted classifiers. In NeurIPS, 2019.

[6] Grønlund, A., et al. Near-tight margin-based generalization bounds for support vector machines. In ICML, 2020.

[7] Ghamrawi, N., & McCallum, A. Collective multi-label classification. ACM international conference on Information and knowledge management. 2005.

[8] Read, J., et al. Classifier chains for multi-label classification. Machine learning, 2011.

[9] Deng, J., et al. Large-scale object classification using label relation graphs. In ECCV, 2014.

[10] Akata, Z., et al. Label-embedding for attribute-based classification. In CVPR, 2013.

[11] Boudiaf, M., et al. A unifying mutual information view of metric learning: cross-entropy vs. pairwise losses. In ECCV, 2020.

Reviewer comment (Experiments):
Number of fragmentation would matter...

Author response:
**Q3. Number of Fragments**

We concur that determining the optimal number of fragments can be an important consideration. This is why we have included a discussion of the current limitations of FragSel in the Appendix, as well as an exploration of feasible approaches to mitigate these limitations. We reiterate that determining the optimal number of fragments without empirical search is a challenging task with related topics just beginning to be explored, as it involves striking a balance between task difficulty [1, 2] and instance difficulty [3, 4].
In this study, we have gained valuable insights, demonstrating that significant performance improvements are consistently achieved across diverse domains and in the presence of various types of noise, even when the number of fragments is fixed at four, as it was throughout all our experiments detailed in the main manuscript. Furthermore, to provide a more comprehensive understanding of the impact of fragment numbers, we conducted a detailed analysis of their effects in Figures 7 and 8.

[1] Mao, Y., et al. Metaweighting: learning to weight tasks in multi-task learning. ACL, 2022.

[2] Guo, M., et al. Dynamic task prioritization for multitask learning. ECCV, 2018.

[3] Ethayarajh, K., et al. Understanding dataset difﬁculty with V-usable information. ICML, 2022.

[4] Baldock, R., et al. Deep learning through the lens of example difﬁculty. NeurIPS, 2021.

Reviewer comment (Evaluation):
When checking the empirical performances of the proposed method (FragSel-R v.s. FragSel-D), it seems that the role of classification-based feature extractor has much larger effect on the performance than the regression based method. And the performance of FragSel-R is not consistently better than baselines.

Author response:
**Q3(W2). Performance of FragSel-D (classification) vs. FragSel-R (regression)? FragSel-R performance consistency?**

Given the empirical advantages of classification training over regression, it is currently a common practice to address regression-type problems by treating them as classification tasks through the process of binning [1-3].

A theoretical explanation for the greater stability of training in classification, specifically using cross-entropy as opposed to regression (such as Mean Squared Error, MSE), as posited by [4], is rooted in the concept that deep neural networks aim to maximize the mutual information between the learned representation $Z$ and the target variable $Y$. Mutual information, denoted as $I(Z; Y )$, can be defined as the difference between the marginal entropy $H(Z)$ and the conditional entropy $H(Z | Y )$. A high value of $I(Z; Y )$ indicates that the marginal entropy $H(Z)$ is high, signifying that the features in $Z$ are diverse or spread out, while the conditional entropy $H(Z | Y )$ is low, implying that the features related to common target values are close or similar. Classification achieves both of these objectives, as demonstrated by [5]. On the other hand, [6] provides evidence that regression primarily minimizes $H(Z | Y )$ but pays less attention to reducing $H(Z)$. Consequently, the learned representations $Z$ obtained from regression tend to have a lower marginal entropy.

Despite the superior performance of classification-based feature extractor training, the regression-based method offers advantages in precisely handling noise. While CE treats all errors equally without emphasizing severe misses, MSE has the capability to differentiate the severity of noise. Given the crucial importance of addressing varying degrees of noise in noisy label regression algorithms, this represents a research direction that can be effectively explored via specialized regression techniques.

As an auxiliary experiment, we additionally present the performance of the co-trained FragSel-R (referred to as Co-FragSel-R). Co-FragSel-R demonstrates its superiority over the majority of baseline methods, particularly when compared to other co-trained baseline methods such as CNLCU-S, CNLCU-H, and Co-Selfie.

[1] Cao, Y., et al. Estimating depth from monocular images as classiﬁcation using deep fully convolutional residual networks. IEEE Transactions on Circuits and Systems for Video Technology, 2017.

[2] Liu, L., et al. Counting objects by blockwise classiﬁcation. IEEE Transactions on Circuits and Systems for Video Technology, 2019.

[3] Van den Oord, A., et al. Pixel recurrent neural networks, In ICML, 2016.

[4] Shwartz-Ziv, R., & Tishby, N. Opening the black box of deep neural networks via information. arXiv preprint arXiv:1703.00810, 2017.

[5] Boudiaf, M., et al. A unifying mutual information view of metric learning: cross-entropy vs. pairwise losses. In ECCV, 2020.

[6] Zhang, S., et al. Improving deep regression with ordinal entropy, In ICLR, 2023.

Reviewer comment (Experiments):
Is there a rationale behind the authors' choice to split fragments based on equal length rather than equal size? For instance, considering the age distribution of heart disease, it may be less common in children, resulting in fewer cases. Would splitting the data in intervals of 0-10, 10-20, etc., be suitable given such disparities?

Author response:
**Q1&Q2. Why Fragment based on Equal Length than Equal Size? Why edge weights via distance between the closest labels of two fragments?**

The foundational principle underlying FragSel is to maximize the contrast or distances between fragments, as it has been empirically demonstrated to be essential for robust training and generalizability [1, 2, 3]. Therefore, while we could conceivably consider equal fragment lengths in our approach, using equal sizes for fragment partitioning may lead to inadequate label separation among contrasting pairs, especially within densely populated label intervals. Consequently, we have chosen to employ equal fragment lengths to fully harness the advantages of contrastive pairing. Additionally, although employing the equal size criterion could assist in addressing dataset imbalances, if they exist, this concern can also be mitigated through post-selection balancing and the utilization of established imbalanced regression techniques, as detailed in reviewer PSFZ’s Q4.

For the same rationale, in the assessment of edge weights, the utilization of nearest boundaries ensures a greater contrast in comparison to the centroid distances mentioned by the reviewer.

[1] Shawe-Taylor, J., & Cristianini, N. Robust bounds on generalization from the margin distribution. 1998.

[2] Grønlund, A., et al. Margin-based generalization lower bounds for boosted classifiers. In NeurIPS, 2019.

[3] Grønlund, A., et al. Near-tight margin-based generalization bounds for support vector machines. In ICML, 2020.

Reviewer comment (Theory):
In step 2 of the proposed contrastive fragmentation algorithm, for completing the graph, could authors explain a bit more about the whole process, i.e., why the edge weight is decided by the distance between the closest samples of the two fragments, instead of the distance between two centroids.

Author response:
**Q1&Q2. Why Fragment based on Equal Length than Equal Size? Why edge weights via distance between the closest labels of two fragments?**

The foundational principle underlying FragSel is to maximize the contrast or distances between fragments, as it has been empirically demonstrated to be essential for robust training and generalizability [1, 2, 3]. Therefore, while we could conceivably consider equal fragment lengths in our approach, using equal sizes for fragment partitioning may lead to inadequate label separation among contrasting pairs, especially within densely populated label intervals. Consequently, we have chosen to employ equal fragment lengths to fully harness the advantages of contrastive pairing. Additionally, although employing the equal size criterion could assist in addressing dataset imbalances, if they exist, this concern can also be mitigated through post-selection balancing and the utilization of established imbalanced regression techniques, as detailed in reviewer PSFZ’s Q4.

For the same rationale, in the assessment of edge weights, the utilization of nearest boundaries ensures a greater contrast in comparison to the centroid distances mentioned by the reviewer.

[1] Shawe-Taylor, J., & Cristianini, N. Robust bounds on generalization from the margin distribution. 1998.

[2] Grønlund, A., et al. Margin-based generalization lower bounds for boosted classifiers. In NeurIPS, 2019.

[3] Grønlund, A., et al. Near-tight margin-based generalization bounds for support vector machines. In ICML, 2020.

Reviewer comment (Evaluation):
Regarding the performance of FragSel-R,-D, it seems that the classification-based feature extractor has much better effect than the regression based method. And the performance of FragSel-R is not consistently better than baselines.

Author response:
**Q3(W2). Performance of FragSel-D (classification) vs. FragSel-R (regression)? FragSel-R performance consistency?**

Given the empirical advantages of classification training over regression, it is currently a common practice to address regression-type problems by treating them as classification tasks through the process of binning [1-3].

A theoretical explanation for the greater stability of training in classification, specifically using cross-entropy as opposed to regression (such as Mean Squared Error, MSE), as posited by [4], is rooted in the concept that deep neural networks aim to maximize the mutual information between the learned representation $Z$ and the target variable $Y$. Mutual information, denoted as $I(Z; Y )$, can be defined as the difference between the marginal entropy $H(Z)$ and the conditional entropy $H(Z | Y )$. A high value of $I(Z; Y )$ indicates that the marginal entropy $H(Z)$ is high, signifying that the features in $Z$ are diverse or spread out, while the conditional entropy $H(Z | Y )$ is low, implying that the features related to common target values are close or similar. Classification achieves both of these objectives, as demonstrated by [5]. On the other hand, [6] provides evidence that regression primarily minimizes $H(Z | Y )$ but pays less attention to reducing $H(Z)$. Consequently, the learned representations $Z$ obtained from regression tend to have a lower marginal entropy.

Despite the superior performance of classification-based feature extractor training, the regression-based method offers advantages in precisely handling noise. While CE treats all errors equally without emphasizing severe misses, MSE has the capability to differentiate the severity of noise. Given the crucial importance of addressing varying degrees of noise in noisy label regression algorithms, this represents a research direction that can be effectively explored via specialized regression techniques.

As an auxiliary experiment, we additionally present the performance of the co-trained FragSel-R (referred to as Co-FragSel-R). Co-FragSel-R demonstrates its superiority over the majority of baseline methods, particularly when compared to other co-trained baseline methods such as CNLCU-S, CNLCU-H, and Co-Selfie.

[1] Cao, Y., et al. Estimating depth from monocular images as classiﬁcation using deep fully convolutional residual networks. IEEE Transactions on Circuits and Systems for Video Technology, 2017.

[2] Liu, L., et al. Counting objects by blockwise classiﬁcation. IEEE Transactions on Circuits and Systems for Video Technology, 2019.

[3] Van den Oord, A., et al. Pixel recurrent neural networks, In ICML, 2016.

[4] Shwartz-Ziv, R., & Tishby, N. Opening the black box of deep neural networks via information. arXiv preprint arXiv:1703.00810, 2017.

[5] Boudiaf, M., et al. A unifying mutual information view of metric learning: cross-entropy vs. pairwise losses. In ECCV, 2020.

[6] Zhang, S., et al. Improving deep regression with ordinal entropy, In ICLR, 2023.

Reviewer comment (Experiments):
While the authors assert that the sole hyperparameters of the framework are the number of fragments (F), the parameter K utilized for KNN-based prediction, and the extent of jittering applied for regularization, the influence of each on the results presented in Table 1 remains unclear. Could the authors provide additional insight into how these hyperparameters affect the outcomes?

Author response:
**Q5. Do Further Hyperparameter Analysis**

In Figures 7 and 8, we already investigate different fragment numbers and the influence of each on the results in Table 1 on IMDB-Clean-B as well as SHIFT15M-B dataset. As per the reviewer’s suggestion, we also include an analysis of $K$ for KNN ([Figure. 9, 10.](https://i.imgur.com/Kyxhl3A.png)) and $J$ for jittering’s influence ([Figure. 11, 12.](https://i.imgur.com/8BPt5Oi.png)).

The hyperparameter $K$ determines the number of neighbors considered when assessing self/neighbor agreement from a representation perspective. With an increase in the value of $K$, the criteria for agreement become more stringent. Consequently, as $K$ increases, a greater number of confident samples are selected, resulting in a reduction in the Selection rate, ERR. However, it is worth noting that the influence of $K$ on Mean Relative Absolute Error (MRAE) performance exhibits slight fluctuations across different datasets.

The hyperparameter $J$ controls the buffer range for jittering, which, in turn, determines the level of regularization applied via neighborhood jittering. Increasing the value of $J$ results in stronger regularization, effectively preventing overfitting. However, excessive regularization, as observed when $J = 0.10$, results in adverse effects during training.

Specifically, in Figure 11(a) (IMDB-Clean-B Symmetric 40%), the feature extractors exhibit similar convergence patterns when $J = 0.05$ or $J = 0.10$. Consequently, comparable performance is observed in Selection Rate and MRAE. Yet, in Figure 11(b) (IMDB-Clean-B Symmetric 40%), the ERR of $J = 0.05$ is smaller than that of $J = 0.10$, leading to improved MRAE performance for $J=0.05$.

Similar effects are observed in the SHIFT15M-B dataset, as depicted in Figure 12 (SHIFT15M-B).