L EARNING WITH N ON -U NIFORM L ABEL N OISE : A C LUSTER -D EPENDENT S EMI -S UPERVISED A PPROACH
Anonymous authors
Paper under double-blind review
A BSTRACT
Learning with noisy labels is a challenging task in machine learning.
Most existing methods explicitly or implicitly assume uniform label noise across all samples.
In reality, label noise can be highly non-uniform in the feature space, e.g. with higher error rate for more difficult samples.
Some recent works consider instance-dependent label noise but they require additional information such as some cleanly labeled data and confidence scores, which are usually unavailable or costly to obtain.
In this paper, we consider learning with non-uniform label noise that requires no such additional information.
We propose a cluster-dependent sample selection algorithm followed by a semi-supervised training mechanism based on the cluster-dependent label noise.
Inspired by stratified sampling, the proposed sample selection method increases the consistency of sample space by forcing the selection of clean samples from the entire feature space.
Despite its simplicity, the proposed method can distinguish clean data from the corrupt ones more precisely and achieve state-of-the-art performance on most image classification benchmarks, especially when the number of training samples is small and the noise rate is high.
I NTRODUCTION
Deep Neural Networks (DNNs) have achieved great success in various machine learning tasks, such as in computer vision, natural language processing, and information retrieval.
Unfortunately, their successes heavily rely on the carefully labeled data, which are expensive and time-consuming to collect.
Online queries (Xie et al., 2019) and crowdsourcing (Yu et al., 2018) are cheap alternatives, which would produce datasets with noisy labels.
Furthermore, the datasets in medical applications are typically small and domain expertise is required to annotate medical data, which however often suffers from inter- and intra-observer variability (Han et al., 2020; Xue et al., 2022).
Song et al.
(2022) reports that the ratio of corrupted labels in real-world datasets range from 8.0% to 38.5%.
Due to the universal approximation ability of DNNs (Hornik et al., 1989), they can easily memorize and eventually overfit to the corrupted labels, leading to poor generalization (Zhang et al., 2017).
Efforts have been taken to robust learning paradigms under noisy labels (Fr´enay & Verleysen, 2014;
Han et al., 2020; Song et al., 2022).
Generally, existing methods on learning with noisy labels can be categorized into two groups: loss correction methods (Patrini et al., 2017; Xia et al., 2019;
Yao et al., 2020; Xu et al., 2019; Xia et al., 2020b; Berthon et al., 2021) and sample selection methods (Han et al., 2018; Yu et al., 2019; Li et al., 2020; Bai et al., 2021).
Methods in the first category mainly model label noise with label transition matrix.
Patrini et al.
(2017); Xia et al.
and Yao et al.
(2020) assumed that label noises were class-dependent but instance-independent (note as class-dependent noise, CDN).
Xia et al.
(2020b) and Berthon et al.
(2021) proposed to model instance-dependent noise (IDN).
Obviously, the IDN transition matrices are more realistic (see Figure 2) but unidentifiable in general (Liu, 2022), which require a large number of parameters to be estimated.
Additional information (Berthon et al., 2021) or extra assumption (Xia et al., 2020b) is required to estimate instance-dependent transition matrix.
The methods of second type are designed to select confident clean samples from noisy datasets based on the memorization effect of DNNs (Arpit et al., 2017), which tend to learn simple patterns first before fitting the corrupt samples.
Han et al.
(2018) and Yu et al.
(2019) train two networks simultaneously where each network selects small-loss samples to train the other one.
Furthermore,
[Figure]
[Figure]
Existing works explicitly or implicitly rely on the assumption of uniform noise, while the noise rate of real dataset range from 25% ∼ 60% in Figure 2.
The methods of CDN transition matrix explicitly consider all the samples in the same class have same noise rate and methods of IDN need additional information and extra assumption, which are not realistic and have poor performance experimentally.
Without explicit assumption of uniform noise rate, the small-loss trick would still select simple patterns first but regard all samples in the hard regions as corrupt data (though part of the samples have correct labels), resulting in inconsistent sample distribution (Cheng et al., 2020).
Notably, Liu & Wang (2021) reports that increasing label noise to balance can return more accurate models.
To empirically verify the poor performance of existing methods on non-uniform noisy data, we conduct a demo experiment on moon data with gradually changed noise rate, shown in Figure 1.
Non-uniform label noise (IDN, noise rate range from 0% to 45% ) is added to the moon dataset and different methods (Xia et al., 2019; Li et al., 2020) are employed to learn the noise-robust classifier.
As illustrated in Figures 1b and 1c, both existing loss correction and sample selection methods fail to classify the moon dataset.
Clearly, T-Revision with transition matrix explicitly suppose that all the samples share the same noise rate, which is contradictory to the real noise distribution and leads to the poor performance.
Although the small-loss trick of DivideMix does not require the same noise rate, no sample in the high noise rate region is selected despite the existence of correct samples, resulting in poor decision boundary.
To address the above problem of inconsistent sample selection problem, we assume that label noise is dependent on the clusters of features (Cluster-Dependent Noise, CluDN, Definition 3.1 for more details), i.e. the samples share the same noise rate in the same cluster while have different noise rate among clusters.
Based on the idea of stratified sampling, we propose a cluster-dependent sample selection algorithm followed by a semi-supervised training mechanism, which is named as ClusterMix.
Cluster-dependent sample selection and semi-supervised training are conducted in turn and precise clean data would be selected progressively.
Based on the cluster-dependent sample selection, samples in both simple and hard regions can be selected which eliminate the inconsistency defect of existing sample selection methods.
Experimental results on real non-uniform datasets verify the effectiveness of our proposed method, which outperforms all the baseline methods when the training set is small and noise rate is high.
The rest of the paper is organized as follows.
In Section 2, we review the related work briefly.
In
Section 3, we first give the our intuition analysis of cluster-dependent label noise and then introduce our proposed ClusterMix algorithm in details.
Section 4 displays the experimental results on various image classification benchmarks.
Finally, concluding remarks are given in Section 5.
R ELATED W ORK
Learning with label noise has been extensively studied (Fr´enay & Verleysen, 2014; Han et al., 2020;
Song et al., 2022).
Generally, existing methods on learning with noisy labels can be categorized into two groups: loss correction methods and sample selection methods.
The loss correction methods aim to design risk-consistent training mechanism based on loss correction, while the sample selection methods try to select clean samples from the entire noisy dataset and recently combine with semisupervised learning.
As the experimental results demonstrates the effectiveness of sample selection methods, we will focus on the sample selection methods in this paper.
Learning with loss correction.
To deal with label noise, methods have been proposed to model the relationship between clean labels and noisy labels by transition matrix and build loss functions to correct it.
Patrini et al.
Goldberger & Ben-Reuven (2017); Xia et al.
(2019) and Zhu et al.
(2021) provide methods to estimate transition matrix and use forward and backward procedures for loss correction.
Besides, Xu et al.
(2019) proposes a information-theoretic loss function, which is provably robust to instance-independent label noise.
However, the instance-independent label noise used in these methods is inconsistent with the non-uniform label noise in reality.
To deal with the instance-dependent label noise, methods (Xia et al., 2020b; Yang et al., 2021; Berthon et al., 2021) estimate a separate transition matrix for each samples.
Nevertheless, the number of matrices parameters is large, resulting in large estimation error in practical applications.
For example, given a dataset with 10000 samples and 10 classes, 10 6 parameters are needed.
Furthermore, Xia et al.
(2020a); Zhu et al.
(2021) and Zhang et al.
(2021) propose to estimate cluster-dependent transition matrices.
Although these methods have made certain progress, they are hard to handle a large number of classes and struggling to estimate accurately for heavy noise.
Sample selection.
The second strand tries to select clean samples from the noisy dataset, by exploiting the memorization effect of DNNs (Arpit et al., 2017).
A common method is to treat samples with small loss as clean ones (Shen & Sanghavi, 2019).
To utilize the property, Co-teaching (Han et al., 2018) train two networks simultaneously and let them select clean samples for each other within each mini-batch.
Co-teaching+ (Yu et al., 2019) improves Co-teaching by maintaining disagreement between the two networks.
MentorNet (Jiang et al., 2018) use a mentor network to select confident clean samples for the training of student network.
To utilize the corrupt samples, some works (Li et al., 2020; Bai et al., 2021; Chen et al., 2021) treat un-selected data as unlabeled data and conduct semi-supervised learning (SSL) mechanism to train networks.
DivideMix (Li et al., 2020) employs two Gaussian Mixture Model (GMM) to select clean samples and MixMatch (Berthelot et al., 2019) strategy to leverage corrupt examples with SSL frameworks.
PES (Bai et al., 2021) select confident clean samples by progressive early stopping.
SOP (Liu et al., 2022) propose to model the label noise and learn to separate it from the data by over-parameterization.
However, existing methods usually adopt unified selection criteria across all samples, where all the hard samples would be regarded as corrupt ones, resulting in inconsistent data space.
In contrast, our method is designed to select samples from each cluster separately.
With variable selection criteria on different clusters, our method is able to better exploit the whole data space and thus achieve superior performance.
M ETHOD 3. P RELIMINARIES
Suppose ( X, Y ) ∈
( X , Y ) are drawn from an unknown joint distribution P D , where X ⊂ R
d is the feature space and Y ⊂ { 0 , 1 } C is the clean label space in a one-hot manner, where d is the feature dimension and C is the number of categories.
Then D = { ( x n , y n ) } n ∈ [ N ] is a clean training dataset with N samples.
[Figure]
[Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure]
rate distribution 0.
N o i s e r a t e
[Figure]
Instead of observing clean label Y , people can only collect noisy label ˜
Y in various real-world scenarios.
Let noisy dataset ˜ D
= { ( x n , ˜ y n ) } n ∈ [ N ] .
We denote ˜ y n as corrupt label if ˜ y
n ̸ = y n and clean otherwise.
The noise rate for sample x is denote as τ ( x ) =
Pr(˜
y ̸
= y | x ) .
And the noise rate for the whole dataset is,
[Equation]
In the sample selection methods, the entire dataset ˜ D would be grouped to K clusters and finally divided into a clean subset ˜ D clean = { ( x n , ˜ y n ) : ( x n , ˜ y
[Equation]
) ∈
[Equation]
˜
[Equation]
n = y n } and a corrupted subset ˜ D corrupt = { ( x n , ˜ y n ) : ( x n , ˜ y n ) ∈ ˜ D & ˜ y
n ̸ = y n } .
Our aim is to learn a robust classifier f ( · , θ ) :
X → Y based only on corrupted dataset ˜ D .
3. I NTUITIONS
As shown in Figure 2, the CDN model, which assumes that all samples in the same class share same noise rate, is unrealistic in general scenarios.
Besides, IDN model is proposed to fit the real noise but suffers the non-identifiability (Liu, 2022).
Specifically, different combination of clean posterior
P ( Y | X ) and instance noise rate τ ( X ) can lead to the same noisy posterior P ( ˜ Y | X ) .
To make a trade-off between flexibility and identifiability, we assume that the real label noise is dependent on the clusters.
The samples in the same cluster share the same noise rate, as stated in Definition 3.1.
Definition 3.1 (Cluster-dependent Label Noise Model) .
Suppose a noisy dataset ˜ D can be divided into multiple clusters based on features.
Cluster-Dependent Noise Model (CluDN) assumes that the samples in each cluster share the same corrupt probability and the corrupt probabilities among clusters can vary greatly.
In the paper, clusters are treated as the finer hierarchy than the classes (illustrated in Figures 3a and 3b), i.e. one class can contain multiple clusters.
On the contrary, clusters are not always the sub-group of the classes, as the clusters on the class boundary can cross two- or multi- classes.
To further explore the validity of the proposed noise model, corresponding noise rate distribution are visualized in Figure 3c, which illustrates the non-uniform noise distribution within each class.
Besides, Appendix E displays several images of two example clusters, which gives a potential source of non-uniform noise on real dataset.
To design a robust learning mechanism under non-uniform noise, we combine the CluDN model with existing sample selection methods.
In CluDN model, samples among different clusters have different patterns and thus unified selection criteria among clusters is inappropriate under non-uniform noise.
Based on the idea of stratified sampling, we propose to select clean samples for each cluster separately and aggregate them together to get the clean data ˜ D clean and corrupt data ˜ D corrupt .
More details will be discussed in the following subsection.
[Figure]
3. C LUSTER M IX : S EMI - SUPERVISED L EARNING WITH C LUSTER -D EPENDENT S AMPLE S ELECTION
Based on CluLN, we propose ClusterMix, a robust semi-supervised algorithm with clusterdependent sample selection module.
An overview of the framework is shown in Figure 4.
To avoid the accumulation of confirmation bias in self-training, we train two identical network with different random initialization simultaneously, following Han et al.
Yu et al.
and
Li et al.
As shown in Figure 4, the algorithm consists of two parts, i.e. cluster-based data division module and SSL training paradigm, specifically MixMatch (Berthelot et al., 2019).
The cluster-dependent data division module will first divide the entire dataset ˜ D into clean subset ˜ D clean and corrupt subset ˜
D corrupt based on cluster and loss information.
Then in the second part, the clean set will be treated as labeled data while corrupt one as unlabeled data.
The SSL module use the label co-refinement and co-guess techniques (Li et al., 2020) to generate pseudo labels and MixMatch technique (Berthelot et al., 2019) to perform SSL regularization.
Cluster-Dependent Data Division.
As shown above, the samples in different clusters have different noise rate.
And samples with high noise rate are more difficult to classify, i.e. all of them would have larger losses.
Thus, the samples in the ambiguous feature space would all be regarded as corrupted instances, resulting in inconsistent data distribution.
To address the challenge of non-uniform label noise, we propose to select clean and corrupt samples within clusters.
Inspired by the idea of divide-and-conquer, we propose to first divide the entire dataset into several cluster-based subset.
According to the CluDN model, the samples in the same cluster have the same noise rate and thus have comparable prediction losses.
Therefore by using the small-loss trick on each cluster, an independent 2-components GMM is used to fit the per-sample losses and the samples clean probability ω
i = p ( g | l i ) , where g is the component with lower mean and l i is the loss of sample i .
Finally, the clean samples ( ω i > 0 . 5 ) in all clusters will merge together as the labeled data while the corrupted ones as the unlabeled data.
The visualization results in Appendix C illustrate the effectiveness of the proposed cluster-dependent sample selection strategy, which actually adds another dimension and thus have variable criteria for different clusters.
The proposed method can select more precise and adequate clean samples and thus achieve better evaluation accuracy.
refinement and co guessing.
in mini-batch.
( ∼ i = if vice
[Equation]
end end
Pseudo label generation.
To account for the label noise, label co-refinement is employed to modified the labeled data and label co-guessing is used to generate pseudo label for unlabeled data using the aggregate output of two networks.
More details are shown in Li et al.
and the step is used in Algorithm 2, line 7.
By the label co-refinement and co-guessing above, pseudo labels are generated for both labeled dataset X and unlabeled dataset U .
Semi-supervised learning - MixMatch.
With the generated pseudo labels, MixMatch (Berthelot et al., 2019) is conducted for SSL, which utilizes unlabeled data by combining consistency regularization and entropy minimization with the MixUp (Zhang et al., 2018) augmentation.
For a pair of samples ( x 1 , ˆ y 1 ) and ( x 2 , ˆ y 2 ) , the mixed sample ( x ′ , y ′ ) is computed by
[Equation]
The mixed sets are denoted as X ′ and U ′ .
Finally parameters Θ 1 , Θ 2 are trained with cross-entropy (CE) loss on X ′ and mean-squared-error (MSE) loss on U ′ .
Clustering.
Different from the moon data shown in Figure 1 which can be grouped on the raw 2-dimensional data space, we use the features extracted by networks to divide the general dataset into clusters.
Clustering and network training take turns in each epoch, where clusters are used for robust feature extraction and conversely extracted features would benefit the precise clustering.
Various clustering methods have been proposed recently, e.g. partition-based clustering, spectral clustering, manifold clustering, and subspace clustering etc.
As our method is robust to the clustering
[Table]
algorithm, we adopt K-Means for convenience.
Besides, experimental result shows the robustness of our method to the clusters number.
E XPERIMENTS 4. D ATASETS AND I MPLEMENTATION D ETAILS
Datasets: We evaluate our method on three real noisy dataset CIFAR-10N, CIFAR-100N (Wei et al., 2022),
Clothing1M (Xiao et al., 2015), and (mini) WebVision (Li et al., 2017).
CIFAR-10N and
CIFAR-100N equipe the training sets of CIFAR-10 and CIFAR-100 (Krizhevsky et al., 2009) with human-annotated real-world noisy labels coleected from Amazon Mechanical Turk.
Both CIFAR10 and CIFAR-100 contain 50K training images and 10K test images with size 32 × 32 .
CIFAR-10N provides several set of real-world noisy labels for training set, i.e. Aggregate (noise rate τ = 9 . 03% ),
Random 1( τ = 17 .
23% ), and Worst ( τ = 40 .
21% ) etc.
And CIFAR-100N provides noisy labels, denoted as Fine ( τ = 40 .
To further discover the performance under different size of training set, we randomly select training samples from CIFAR-10N dataset, where the noise rate barely changes (variation range < 1% ), as shown in Table 2.
Clothing1M is a large-scale clothing dataset with real-world noisy labels, whose images are crawled from several online shopping websites and labels are generated from the surrounding texts.
In our setting, we use 1M training images with noisy label for training and 10K test images with clean label for evaluation.
WebVision contains 2.4 million images crawled from the web whose training set contains many real-world noisy labels.
Then following previous work (Li et al., 2020; Chen et al., 2019), baseline methods are compared on the first 50 classes of the Google image subset (approximately 66 thousand images).
Models trained on mini WebVision are evaluated on both WebVision and ImageNet
ILSVRC12 validation set.
Baselines: As introduced before, existing methods consist of loss correction and sample selection methods.
To make comprehensive comparison, We select state-of-the-art methods from different categories.
Specifically, (1) CE (Standard): standard training with cross-entropy loss.
(2) T-Revision (Xia et al., 2019): loss correction with label transition matrix estimated without anchor points.
PTD (Xia et al., 2020b): estimate part-dependent label noise based on non-negative matrix factorization (NMF).
(4) ELR+ (Liu et al., 2020): prevent memorization of the false labels by early-learning regularization.
(5) DivideMix (Li et al., 2020): uses data division and MixMatch (Berthelot et al., 2019) mechanism to leverage corrupt data.
(6) SOP (Liu et al., 2022): model label noise via another sparse over-parameterization and separate the underlying corruptions by exploiting implicit algorithmic regularizations .
Network structure and parameters: Our method is implemented by PyTorch v1.12.
Baseline methods are implemented by the public codes with the same hyper-parameter settings in the original papers.
For the methods with multiple networks, final test accuracy is evaluated with the average among the networks.
To guarantee the comparability, we use the same backbone networks among the baseline methods for the same task.
For the CIFAR-10/100N datasets, we use 18/34-layer PreAct ResNet as backbone (He et al., 2016), following Li et al.
The networks are trained for 300 epochs and SGD is used with a mo-
[Table]
mentum of 0.9 and a weight decay of 0.0005. The initial learning rate is set as 0.02 and reduced by a factor of 10 after the 150-th epoch.
The batch size is 128, clustering interval is 10 epochs, and K-Means is used as cluster method.
10 warm-up epochs are used for CIFAR-10 and 30 warm-up epochs for CIFAR-100.
For Clothing1M datasets, we use a 50-layer ResNet with ImageNet pretrained weights.
The networks are trained for 100 epochs and SGD with a momentum of 0.9, a weight decay of 0.001 is used.
The initial learning rate is set as 0.002 and reduced by a factor of 10 after the 50-th epoch.
The batch size is 32, clustering interval is 10 epochs, K-Means is used as cluster method, and 5 warm-up epochs are used.
As Clothing1M is a large dataset with 1 million images, we randomly select 1000 batches with in each epoch.
For (mini) WebVision dataset, we use the inception-resnet v2 (Szegedy et al., 2017), following the previous work.
The networks are trained for 100 epochs and SGD with a momentum of 0.9, a weight decay of 0.001 is used.
The initial learning rate is set as 0.01 and reduced by a factor of 10 after the 50-th epoch.
The batch size is 32, clustering interval is 10 epochs, K-Means is used as cluster method, and 5 warm-up epochs are used.
More parameters keep same with Li et al.
4. C LASSIFICATION A CCURACY E VALUATION
In this section, experiment results are shown on original CIFAR-N datasets, CIFAR-N datasets with different training samples, CIFAR-10 dataset with synthetic noise, clothing1M dataset, and WebVision dataset etc.
More experiments are shown in the Appendix.
Specifically, accuracy curve in Appendix D, experiments with synthetic label noise in Appendix F, sensitivity analysis for the number of clusters K and cluster methods in Appendix G, and contrast experiments for the sample selection module and the training module in Appendix H.
Experiment results on original CIFAR-N datasets.
We first evaluate the test accuracy on the original CIFAR-N datasets, as shown in Table 1.
Specifically, we select aggregate ( τ = 9 .
03% ), random 1 ( τ = 17 .
23% ), and Worst ( τ = 40 .
21% ) noisy label for CIFAR-10N and fine ( τ = 40 .
20% ) noisy label for CIFAR-100N.
We report the averaged test accuracy over the last 10 epochs.
In the original large dataset, our method reach state-of-the-art accuracy and outperforms slightly.
Experiment results on CIFAR-10 with different number of training samples.
As the nonuniform noise would have greater impact with less training samples, we conduct experiments on CIFAR-10N Worst dataset with different number of training samples (500 –50000), as shown in Table 2.
The samples are randomly sampled from the CIFAR-10N Worst dataset.
The categories keep balance and the noise rate holds in 40% ± 1% .
We report the averaged test accuracy over the lastepochs.
ClusterMix outperforms state-of-the-art methods with various number of training samples.
Notably, our method make greater improvement when training set goes smaller.
The results demonstrate the effectiveness of our approach to suppress the inconsistent data space problem after sample selection, as the problem has greater impact when the training set is smaller.
Experiment results on Clothing1M and WebVision datasets.
To validity the effectiveness of the proposed method on more general noisy datasets, experimental results on Clothing1M and (mini) WebVision datasets are shwon in Table 3.
For Clothing1M, we design two experiments with different training samples: Clothing1M-I : Training with all 1 million samples available.
Clothing1M-II : Training with randomly selected 5000 samples.
The two tasks evaluate the performance of our
[Table] [Table]
ELR+ 81. DivideMix 84.
SOP 80.
ClusterMix(ours) 87.
method with different training data size. Although our method does not achieve the state-of-the-art accuracy on the very large dataset (in Task I), it outperforms all baseline methods when the number of samples reduces to a smaller quantity (in Task II).
For (mini) WebVision dataset, we evaluate the models, which are trained on the mini WebVision training set, on WebVision validation set and ILSVRC12 validation set.
Our method performs well on the both validation sets.
Experiments on CIFAR-10 dataset with synthetic label noises.
To evaluate the performance under different skewness of the noise distribution, we conduct experiments on CIFAR-10 dataset with synthetic label noises. Table 4 shows the test accuracy for class-dependent label noise with varying skewness.
Empirical results verify the effectiveness of the proposed method on the nonuniform noise.
Besides, The results of symmetric and asymmetric noise are also available in Table 5, Appendix F.
C ONCLUSION
Due to the memorization effect of DNNs (Arpit et al., 2017), the small-loss trick would select simple patterns (which usually have lower noise rate) first but regard all samples in the hard regions as corrupt data, resulting in inconsistent data space.
Furthermore, the inconsistency problem would be harder when the training set is small and the label noise is heavy.
Therefore in this paper, we propose a novel ClusterMix algorithm to solve the inconsistent sample selection problem.
ClusterMix combines the cluster-dependent sample selection method with a semi-supervised learning mechanism to distinguish and leverage the corrupt labels.
Based on the idea of stratified sampling, our proposed cluster-dependent sample selection method would divide the dataset with variable criteria for samples in different clusters, which guarantees the sample selection from the entire sample space.
Experiment results demonstrate that our method can suppress the inconsistent sample selection problem effectively and our ClusterMix outperforms all baseline methods on the datasets with a small number of training samples.
Ethics Statement.
In this paper, our studies are not related to human subjects, practices to data set releases, potentially harmful insights, potential conflicts of interest and sponsorship, privacy and security issues, legal compliance, and research integrity issues.
In real Scenarios, discrimination/bias/fairness may also result in the non-uniform annotated noisy labels.
Due to the memorization effect of DNNs (Zhang et al., 2017), general training paradigms would also memorize the group bias of the annotators.
Our proposed method makes a step for the robust and unbiased training with the non-uniform label noise.
Reproducibility.
Experimental details are discussed in Section 4.1 and we will release the code upon acceptance.
A E XPERIMENT R ESULTS ON M OON D ATA OF OUR C LUSTER -D EPENDENT S AMPLE S ELECTION
As shown in Figure 1, existing methods cannot solve the classification problem of moon data on non-uniform label noise.
Figure 5 shows the classification result and sample selection result of our cluster-dependent sample selection.
Based on the idea of stratified sampling, we select clean samples from each cluster separately, using the small-loss trick and clean-label-dominate-the-cluster trick.
The clean-label-dominate-the-cluster trick means that there are more cleanly labeled data in each cluster.
[Figure]
B N OISE R ATE D ISTRIBUTION ON E ACH C LASS
As a supplementary of Figure 2, Figure 6 shows the noise rate varies not only on the entire dataset but also on each separate class.
The results in Figure 6 illustrates the reality of our cluster-dependent sample selection rather than class -dependent sample selection
[Figure]
C I LLUSTRATION OF THE PROPOSED SAMPLE SELECTION STRATEGY
Figure 7 illustrates the effectiveness of the proposed cluster-dependent sample selection strategy.
In the process of the small-loss samples selection, the proposed method adds another dimension and thus have variable criteria for different clusters.
The proposed method can select more precise and adequate clean samples and thus achieve better evaluation accuracy.
Besides, Figure 7b also shows the variability of the noise distribution inside each class.
[Figure]
D A CCURACY CURVE
Figure 8 shows the accuracy curve and standard deviation of the methods with different number of training samples.
Our method performs slightly high than the baseline methods with large number of training samples.
And the gap goes larger when the training samples are inadequate.
E E XAMPLE IMAGES OF CLUSTERS ON CIFAR-10 DATASET
Figure 9 shows several bird images of two clusters.
The images in the same cluster share similar patterns and thus have similar corrupt probabilities.
Conversely, the images in different clusters have far different patterns and have very different noise rate.
The visualization of the real clusters further verify the validity of the proposed non-uniform noise model.
[Figure]
F E XPERIMENTS WITH SYNTHETIC LABEL NOISE F. G ENERATION OF THE SYNTHETIC NOISE
The generation of symmetric and asymmetric label noise is strictly following Li et al.
Specifically, the symmetric label noise flips the labels in the uniform way, i.e. all the label share the same noise rate and flip to the other classes in a symmetric manner.
The asymmetric label noise also flips labels with the same probabilities but just flip to the specific class for each class.
The experiment results are shown in Table 5.
To further explore the methods with non-uniform noise, we synthesise a new class-dependent nonuniform noise on CIFAR-10 dataset.
Specifically, we randomly select five classes as the low-noiserate classes (noise rate is denoted as τ low ) and the other as the high-noise-rate ones (noise rate is denoted as τ high ).
Then the noisy labels are generated with corresponding classes noise rates.
The experiment results are shown in Table 4.
In this experiments, we vary the skewness of the classes noise rate while keep the average noise rate the same.
The experiment results are shown in Table 4.
F.
R ESULTS FOR THE SYMMETRIC AND ASYMMETRIC LABEL NOISE
Table 5 shows the experimental results on synthetic symmetric and asymmetric noise.
From the experimental results on various synthetic noises (Tables 4 and 5), our proposed method can perform well under various distributional scenarios, especially when the noise rate is non-uniform and the noise is heavy.
The empirical results verify the effectiveness of our method.
[Table]
G S ENSITIVITY A NALYSIS
In this section [Table] we investigate the hyper-parameter sensitivity for the number of clusters K and cluster methods, respectively.
For convenience [Table] all the experiments in this section are conducted on the CIFAR-10N Worst dataset with 5000 randomly selected training samples.
The 5000 selected samples are same with experiments in Table 2.
The test accuracy and training time are reported from the same NVIDIA GeForce RTX 3090 GPU.
Different number of clusters.
We first evaluate the test accuracy and training time for different number of clusters in our ClusterMix.
The results are shown in Table 6.
Both the test accuracy and training time increase with the increase of the number of clusters K , except K > 800 .
The experiment illustrate the effectiveness of stratified sampling.
However, the training time and number of samples limit the number of clusters not too much.
To balance the training effect and time, we prefer to select a intermediate value, which is 200 clusters (for 5000 samples) in the previous experiments.
[Table]
Different clustering methods.
Second, we analyse the sensitivity for the choices of different clustering methods.
The results are shown in Table 7.
Specifically, we compare the test accuracy and training time of four different clustering methods in our ClusterMix.
K-Means, K-Means++, and
Ward Hierarchical clustering require a specified number of cluster centers, which is set as 100 in the experiment.
In contrast, Density-Based Spatial Clustering of Applications with Noise (DBSCAN) can determine the number of clusters by itself.
The experiment results demonstrate that our method achieves similar test accuracy and training time among different clustering algorithms, except DBSCAN.
Therefore, we adopt K-Means as our clustering algorithm in all other experiments for convenience.
[Table]
H C ONTRAST EXPERIMENTS FOR THE SAMPLE SELECTION MODULE AND THE TRAINING MODULE
To evaluate the effectiveness and applicability of the proposed cluster-dependent sample selection method, we combine the basic small-loss trick/cluster-dependent sample selection with two different training mechanisms (i
[Table]
e. supervised learning and semi-supervised learning), as shown in Table 8.
The experimental results validate the effectiveness and applicability of the proposed method.
• Effectiveness.
Compared with basic small-loss trick, the proposed cluster-based sample selection strategy can improve the performance both in the supervised and semi-supervised mechanisms.
Besides, the proposed method plays a greater role when the training samples are inadequate.
• Applicability.
The proposed cluster-based sample selection strategy can be used to two label noise learning methods (one is supervised and the other is semi-supervised).
Both of methods can improve the performance over the basic small-loss trick.
[Table]