Estimating Noise Transition Matrix with Label Correlations for Noisy Multi-Label Learning
Abstract
In label-noise learning, the noise transition matrix , bridging the class posterior fornoisy and clean data, has been widely exploited to learn statistically consistentclassiﬁers.
The effectiveness of these algorithms relies heavily on estimating thetransition matrix.
Recently, the problem of label-noise learning in multi-label classiﬁcation has received increasing attention, and these consistent algorithms can beapplied in multi-label cases.
However, the estimation of transition matrices in noisymulti-label learning has not been studied and remains challenging, since most of theexisting estimators in noisy multi-class learning depend on the existence of anchorpoints and the accurate ﬁtting of noisy class posterior.
To address this problem, inthis paper, we ﬁrst study the identiﬁability problem of the class-dependent transitionmatrix in noisy multi-label learning, and then inspired by the identiﬁability results,we propose a new estimator by exploiting label correlations without both anchorpoints and accurate ﬁtting of noisy class posterior.
Speciﬁcally, we estimate theoccurrence probability of two noisy labels to get noisy label correlations.
Then, weperform sample selection to extract side information about clean label correlations,which is used to estimate the occurrence probability of one noisy label when acertain clean label appears.
By utilizing the mismatch of label correlations impliedin these occurrence probabilities, the transition matrix is identiﬁable , and can thenbe inferred by solving a simple bilinear decomposition problem.
Empirical resultsillustrate the effectiveness of our estimator to estimate the transition matrix withlabel correlations, leading to better classiﬁcation performance.
Introduction
In real-world scenarios, an instance is naturally associated with multiple labels, and these labelshave complex entangled correlations [5].
Recently, the problem of label-noise learning in multi-labelclassiﬁcation has received more and more attention [21, 27, 40], since it is time-consuming andexpensive to collect large-scale accurate labels and the noisy labels are much cheap and easy toacquire.
In the noisy multi-label setting, the multiple labels assigned to one example may be corruptedsimultaneously.
That is, any label for each class can be ﬂipped with its respective transition matrix ,which denotes the transition relationship from clean labels to noisy labels.
Transition matrix has been utilized to build a series of statistically consistent algorithms for noisymulti-class learning [25, 38].
The main advantage of these consistent algorithms is that they canguarantee to vanish the differences between the classiﬁers learned from noisy data and the optimalones from clean data by increasing the size of noisy examples [20, 26, 38, 31].
Fortunately, these statistically consistent algorithms for noisy multi-class learning can also be appliedin such noisy multi-label learning with a little modiﬁcation [40] (more details can be found in
Appendix A).
However, the effectiveness of these algorithms relies heavily on estimating the transition matrix.
Although the estimation of the transition matrix has been investigated in noisy multi-classlearning, the estimation of the transition matrix in noisy multi-label learning has not been studiedand remains challenging.
Speciﬁcally, a series of methods [20, 26, 44, 38, 18] has been proposedto estimate
the transition matrix for noisy multi-class learning.
Nevertheless, most of them assumethe existence of anchor points [20, 26, 44], which are deﬁned as the training examples belonging toa particular class surely.
However, the assumption is strong and hard to check when we only havenoisy data [38].
Also, the methods need to accurately ﬁt the noisy or intermediate class posterior ofanchor points, which are rather difﬁcult in multi-label cases, due to severe class imbalance [28].
In this paper, to address the problem of estimating the transition matrix in noisy multi-label learning,we consider utilizing label correlations among noisy multiple labels.
Speciﬁcally, some labelcorrelations that should not exist in practice are included in noisy multi-label learning.
For example,“ﬁsh” and “water” always co-occur, while “bird” and “sky” always co-occur.
But, due to label errors,there is a slight correlation between “ﬁsh” and “sky”, which is impractical.
In a high level, we canutilize the mismatch of label correlations to identify the transition matrix without both anchor pointsand accurate ﬁtting of noisy class posterior.
In more detail, we ﬁrst focus on the identiﬁability problem of the class-dependent transition matrixin noisy multi-label learning.
Accordingly, a new method that estimates the transition matrices byexploiting label correlations is proposed.
That is, motivated by the identiﬁability result that two noisylabels will not sufﬁce to identify the transition matrix in noisy multi-label learning, we utilize sampleselection to extract useful side information about clean label correlation from noisy data, to achievethe identiﬁability.
Afterward, we not only estimate the occurrence probability of two noisy labels onthe noisy data, but also of one noisy label when a certain clean label appears on the selected data.
By utilizing the mismatch of label correlations implied in these occurrence probabilities, we canprove the identiﬁability, and transform the problem of estimating the transition matrix using labelcorrelations into the problem of bilinear decomposition .
Finally, with easy frequency counting, wecan get a good estimation of the noise transition matrix.
Empirical results illustrate the effectiveness of the proposed estimator for estimating the transitionmatrix in noisy multi-label learning, and the consistent algorithms with our estimator can achievebetter classiﬁcation performance.
The rest of the paper is organized as follows.
In Section 2, we brieﬂy present the problem settingof label-noise learning in multi-label classiﬁcation.
In Section 3, we discuss the identiﬁabilityof transition matrices under such noisy multi-label setting, and introduce our estimation method.
Experimental results are provided in Section 4.
Finally, we conclude the paper in Section 5.
Problem Setting
In this section, we present the problem setting of label-noise learning in multi-label classiﬁcation.
Notations.
In what follows, scalars are in lowercase letters, vectors are in lowercase boldface letters,and matrices/variables are in uppercase letters.
Preliminaries.
Let D be the distribution of a pair of random variables ( X , Y ) , where X ∈ X ⊆ R ddenotes the variable of instances, Y =
{ Y 1 , Y 2 , ..., Y q } ∈ { 0 , 1 } q the variable of target with qpossible class labels.
Y j = 1 indicates instance X is associated with the class k ;
Y j = 0 , otherwise.
In multi-label learning, the goal is to learn a function from D which maps each unseen examplex ∈ X to proper labels y .
However, as discussed, Y is hard to be annotated precisely.
Beforebeing observed, their true labels are independently ﬂipped and what we can obtain are noisy trainingsamples { ( x i , ¯ y i ) } ni =1 , where ¯ y i denotes the noisy label.
Let ¯ D be the distribution of the noisyrandom variables ( X ,
¯ Y ) ∈
X × { 0 , 1 } q .
Let [ z ] = { 1 , . .
. , z }
Transition matrices.
The random variables ¯ Y j and Y j for the class j are related through a noisetransition matrix T j ∈ [0 , 1] 2 × 2 , j ∈ [ q ] .
Generally, the transition matrix depends on instances,i.e., T jik ( x ) =
P ( ¯ Y j = k | Y j
=
i, X = x ) .
Given only noisy examples, the instance-dependenttransition matrix is hard to learn without any additional assumption [38, 43].
In this paper, we study aspecial case by assuming that the transition matrix is class-dependent and instance-independent, i.e.,
P ( ¯ Y j = k | Y j
=
i, X = x ) =
P ( ¯ Y j = k | Y j = i )
=
T jik , which is focused by the vast majorityof current state-of-the-art label-noise learning algorithms in the single-label case [26, 46, 42, 23, 31].
Consistent algorithms.
The transition matrix bridges the class posterior probabilities for noisy andclean data, i.e., P ( ¯ Y = k
| X = x ) = (cid:80) 1 i =
T ik P ( Y = i
| X = x ) .
Thus, it has been exploitedto achieve many statistically consistent algorithms for noisy multi-class learning.
Speciﬁcally, ithas been utilized to build risk-consistent estimators via correcting loss functions [20, 26, 38], and todesign classiﬁer-consistent estimators via limiting hypotheses, e.g., [26, 6, 48].
Since the multi-labeltask can be decomposed into multiple conditionally independent binary classiﬁcation problems, wealso can apply these consistent methods in noisy multi-label learning [40].
In this paper, withoutloss of generality, we adopt a risk-consistent algorithm, i.e. Reweight [20, 38], to learn statisticallyconsistent classiﬁers with estimated transition matrices.
More details can be found in Appendix A.
Transition matrix estimation.
As inaccurate transition matrices will degenerate the performances ofthese consistent algorithms, a series of estimation methods [20, 37, 44, 49, 18] have been proposedfor noisy multi-class learning to efﬁciently identify the transition matrix.
However, most of themrequire anchor points [26, 44, 18], which is a strong assumption and hard to check in multi-label caseswhen only noisy data are provided [38].
Besides, severe class imbalance in multi-label learning [28]will make it difﬁcult to accurately approximate the noisy or intermediate class posterior of anchorpoints, which is crucial for these methods.
This motivates us to seek for a better estimator that can dowithout anchor points and avoid estimating noisy class posterior in noisy multi-label learning.
Estimating Transition Matrices using Label Correlations
In this section, we ﬁrst study the identiﬁability problem [22] of class-dependent transition matricesunder multi-label cases.
Furthermore, inspired by these results, we propose a new estimator toapproximate the transition matrix by utilizing the wealth of label correlations under the noisy multilabel setting.
It is worth pointing out that our estimator demands neither the existence of the anchorpoints nor accurate ﬁtting of noisy class posterior.
3.1 Identiﬁability of transition matrix
Recently, Liu et al.
[22] built identiﬁability of the noise transition matrix on the Kruskal’s identiﬁability results.
Inspired by them, with the complex correlations among class labels, we can getsome identiﬁability results of the class-dependent and instance-independent transition matrix in thenoisy multi-label setting.
First of all, we deﬁne the identiﬁability of the noise transition matrix T andpresent some reasonable assumptions.
Deﬁnition 1 (Identiﬁability of T ) .
For a class-dependent transition matrix T , denote the distributioninduced by T for on the observation space Ω as P T .
Identiﬁability requires that P TTT (cid:54) =
P TTT (cid:48) for
T (cid:54) =
T (cid:48) , up to label permutation.
Assumption 1.
P ( ¯ Y j = 0 | Y j = 1) + P ( ¯ Y j = 1 | Y j = 0) < 1 , j ∈ [ q ]
Assumption 1 means that the noisy label is agreed with the clean label on average, which is a standardcondition for analysis under the class-dependent transition matrix [25, 24].
Assumption 2.
P ( Y i =
| Y j = 0) (cid:54) =
P ( Y i =
| Y j = 1) , i, j ∈ [ q ] and i (cid:54) =
j .
Assumption 2 means that the multiple labels have correlations between each other, which is satisﬁedby the most of ( i, j ) pairs in the real-world dataset (See Appendix B).
When considering multi-label learning, the simplest case is having two class labels.
In this case, thefollowing result can be obtained:
Theorem 1.
Noisy labels { ¯ Y j , ¯ Y i } will not sufﬁce to identify T j .
This result tells us that we should utilize more information about Y j to achieve identiﬁability exceptfor the correlations of two noisy labels.
A natural idea is increasing the number of noisy labels, andactually, we can prove Theorem 2 based on the Kruskal’s identiﬁability result [16, 32]
Theorem 2.
If ¯ Y i and ¯ Y k are independent given Y j , noisy labels { ¯ Y j ,
¯ Y i , ¯ Y k } are sufﬁcient toidentify T j .
The assumption that ¯ Y i and ¯ Y k are independent given Y j can be satisﬁed in certain cases, e.g. theoccurrences of "blue" and "dolphin" may be independent given "sea" appearing or not.
Nevertheless,due to the complex correlations among labels, this assumption is hard to hold in most cases.
When the assumption can not hold, these label correlations are no more sufﬁcient to determine T j , as shownin Theorem 3 in the following.
Theorem 3.
If ¯ Y i and ¯ Y k are not independent given Y j , noisy labels { ¯ Y j ,
¯ Y
i , ¯ Y k } will not sufﬁceto identify T j .
The inspiration from Theorem 3 is that, although increasing the number of noisy labels can providemore information, it may also increase model complexity due to the entangled correlations, makingthe identiﬁability decrease.
Hence, we should know that how to improve the identiﬁability withoutincreasing model complexity.
A direct way is to reduce the number of unknown model parameters with side information.
Inspiredby this, we prove Theorem 4 by providing the transition relationship between noisy label for class iand clean label for class j via side information.
Theorem 4.
Noisy labels { ¯ Y j , ¯ Y i } and P ( ¯ Y i | Y j ) are sufﬁcient to identify T j .
In a high level, Theorem 4 theoretically guarantees that the identiﬁability of class-dependent transitionmatrix can be achieved by utilizing the mismatch of label correlations implied in the occurrenceprobabilities P ( ¯ Y i , ¯ Y j ) and P ( ¯ Y i | Y j ) .
The detailed proof of Theorem 1-4 is provided in Appendix C-F.
3.2 Our estimator
Our estimator is based on Theorem 4, where the side information is utilized to estimate P ( ¯ Y i | Y j ) .
Recently, the memorization effect [2] of deep networks have received much attention in learningwith noisy labels, which shows that deep networks will ﬁrst memorize the training data with cleanlabels and then those with noisy labels.
Prior works utilize this characteristic to develop the sampleselection methods [12, 9, 17, 13] for learning with noisy labels, where we select some samples D jswith more likely clean labels for each class j respectively in the early learning phase.
The sampleswe selected can serve as the side information about clean label correlations, through which we canachieve estimation via counting.
However, when implementing sample-selection-based methods, amajor concern is whether the sampling bias will lead to large estimation errors.
Generally speaking, according to the memorization effect , the sampling bias is selecting easy samplesfor class j , which usually means these samples has the easy-to-discriminate features for class j , andwe can reasonably assume that given Y j , the features about class j is biased, while the features aboutanother class i is unbiased, i.e.
[Equation]
where D js is the distribution of ( X , Y s ) ,
Y s = { ¯ Y 1 , ¯ Y 2 , ..., Y j , ..., ¯ Y q } , ¯ D js is the biased distributionof ( X , Y s ) , and X i is the part of X , which represents all information about class i appearing ornot.
When the assumption is satisﬁed, the sample selection will not lead to large estimation error onˆ P ( ¯ Y i | Y j ) , and it can converge to zero exponentially fast by counting [4].
In real-world scenarios,due to the complex label correlations, this assumption will not strictly hold, but it may be roughlymet as the classifying of simplest samples is not easily affected by the presence or absence of otherlabels.
In Section 4.1, our empirical results justify this by showing a little gap between the estimationerror of our estimator with the biased sample selection and an unbiased one.
Based on the above discussions, we can approximate P ( ¯ Y i , ¯ Y j ) and P ( ¯ Y i | Y j ) with a little errorby frequency counting, and utilize the mismatch of label correlations implied in these occurrenceprobabilities to estimate the transition matrix.
Hence, we propose to estimate transition matrices{ ˆ T j } qj =1 by following two stages:
In the ﬁrst stage, we utilize sample selection to obtain side information about clean label correlations.
We train a classiﬁer with the standard multi-label classiﬁcation loss on noisy training samples D t fora few epochs, and then perform sample selection to get selected "clean" set D js for each class label j .
Specially, we use a commonly used sample selection way [1, 17] in learning with noisy labels, whichextracts the subset of instances with small losses by modeling the distribution of per-sample loss forclass j with a Gaussian Mixture Model (GMM).
In the second stage, we perform co-occurrence estimation by frequency counting, and then estimatetransition matrix by solving a simple bilinear decomposition problem.
For class label j , we ﬁrstchoose another class label i and estimate P ( ¯ Y i , ¯ Y j ) and P ( ¯ Y i | Y j ) via counting, i.e.,
[Equation]
[Equation]
where I [ · ] is the indicator function which takes 1 if the identity index is true and 0 otherwise.
Then, these co-occurrence probabilities, which imply the mismatch of label correlations, can lead tofour equations involving T j :
P
[Equation]
[Equation]
¯ Y
[Equation]
[Equation] [Equation]
[Equation] [Equation]
[Equation]
P (cid:0) ¯ Y j = 0 , ¯ Y i = 1 (cid:1) =
P ( Y j = 0)
T j 00 P ( ¯ Y i =
| Y j = 0)
+ P ( Y j = 1) T j 10 P
[Equation]
¯ Y i =
| Y j = 1) ,
P (cid:0) ¯ Y j = 1 , ¯ Y i = 0 (cid:1) =
P ( Y j = 0) T j 01 P ( ¯ Y i =
| Y j = 0)
+ P ( Y j = 1) T j 11 P ( ¯ Y i =
| Y j = 1) ,
P (cid:0) ¯ Y j = 1 , ¯ Y
i
[Equation]
For simplicity, we denote
[Equation]
=
[Equation]
ρ − ρ + 1 − ρ + (cid:19) , and
[Equation]
Then the system of equations can be expressed as E =
( T j ) (cid:62) P M , i.e.
[Equation] [Equation]
Denote the estimation of E , P , T j and M as
[Equation]
As ˆ E and ˆ M can be derived from Eq.
(2) and Eq.
(3), the problem is hence equivalent to a bilineardecomposition problem:
[Equation]
By solving the above matrix equation, we can get
[Equation]
and the estimation of the transition matrix
[Equation]
Implementation of our estimator.
The pseudo code of our estimator is described in Algorithm 1.
Alittle difference from the above is that in order to make better use of correlations among labels, weperform R times co-occurrence estimation and bilinear decomposition for different classes i in thesecond stage to get R estimations, ˆ T j r , r = 1 , 2 , ..., R .
Finally, we estimating the transition matrix
T j by Eq.
[Equation]
where (cid:107) · (cid:107) 1 denotes (cid:96) 1 norm.
Require:
Noisy training samples D t , the number of classes q , the early warmup training epoch
E warm , clean probability threshold of sample selection τ , and repeated estimation times R .
Selection do 7: do 8:
Estimating P ( ¯ Y i , ¯ Y j ) by ˆ P ( ¯ Y i , ¯ Y j ) with Eq
(2) on D t P ( ¯ Y i | Y j ) by ˆ P ¯ Y Y
) with Eq 3) on
Solve a bilinear decomposition problem (Eq. (4)) to get estimation
T
Estimating transition matrix T j by ˆ T j which has the minimum error with Eq.
(7) from R estimations.
13: end for
The estimated
Experiments
Dataset We verify the effectiveness of the proposed method on three synthetic noisy multi-labeldatasets, i.e., Pascal-VOC2007 [7], Pascal-VOC2012 [8], and MS-COCO [19].
Pascal-VOC
[7]and Pascal-VOC2012 [8] datasets are two popular datasets for object recognition.
They both containimages from same 20 object classes, with an average of n
a = 1 . 5 labels per image.
Pascal-VOC2007contains a training set of 5,011 images and a test set of 4,952 images.
Pascal-VOC2012 consistsof 11,540 images as training set and 10,991 images as the test set [3].
As the labels of the test setin Pascal-VOC2012 are not publicly available, we use the test set in Pascal-VOC2007 for PascalVOC2012 evaluation.
MS-COCO [19] is a widely used multi-label dataset.
It contains 82,081images as the training set and 40,137 images as the test set and covers 80 object classes with anaverage of n a = 2 .
9 labels per image.
For these datasets, we corrupted the training sets manuallyaccording to true transition matrices { T j } qj =1 .
For convenience, we use the same true transitionmatrices for all classes, i.e. [Equation] =
[Equation]
,but do not divulge this informationfor algorithms.
we generate four different types of synthetic datasets by using different transitionmatrices: 1) ρ − = 0 , ρ + = ρ , which annotates some positive examples as negative examples, alsoknown as multi-label learning with missing labels [35, 34]; 2) ρ − = ρ, ρ + = 0 , which annotatessome negative examples as positive examples, also known as partial multi-label learning [39, 41];3) ρ − = ρ, ρ + = ρ , where positive samples and negative samples are mislabeled with the sameprobability ρ ; 4) ρ − = n a q − n a ρ, ρ + = ρ , where positive samples and negative samples are mislabeledwith the same number.
In the experiments, we test the algorithms on various ρ .
For all datasets,we leave out 10% of the noisy training examples as a noisy validation set.
We use mAP on noisyvalidation set as the criterion for model selection.
Implementation details For a fair comparison, we implement all methods with default parametersby PyTorch on NVIDIA RTX 3090.
We use a ResNet-50 network [10] pre-trained on ImageNet [29]for all datasets, and the optimizer is Adam optimizer [14] with β = (0 . 9 , 0 . 999) .
The batch size is128, the learning rate is 5e-5. The number of training epochs is 20 for Pascal-VOC2007/VOC2012,and 30 for MS-COCO. For the transition matrix estimation method, E warm is the same as the normaltraining epoch. For our estimator, we perform sample selection based on the average losses of 5epochs before a certain warmup epoch (10th epoch for Pascal-VOC2007/VOC2012, 15th epoch for
MS-COCO), R = q − 1 and τ = 0 . 5 in all experiments. All experiments are run at least three times with different random seeds, and we report the average and standard deviation value of results. Thebest results are in bold , and the second-best results are in blue.
Baselines For evaluating the effectiveness of estimating the transition matrix under multi-label cases,we compare the proposed method with the following methods: (1) T-estimator max [20, 26], whichestimates the transition matrix via the noisy class posterior probabilities for anchor points that havethe largest estimated noisy class posteriors.
(2) T-estimator 97% [20, 26], which selects the pointswith 97% largest estimated noisy class posteriors to be anchor points.
(3) Dual T-estimator max [44],which introduces an intermediate class to avoid directly estimating the noisy class posterior, andselects the points with the largest estimated intermediate class posteriors to be the anchor points.
Dual T-estimator 97% [44], which selects the points with the 97% largest estimated intermediateclass posteriors to be the anchor points.
Metrics We use the sum of estimation error for the transition matrices as the estimation evaluationmetric, i.e. (cid:80) qj =1 (cid:107) T j − ˆ T j (cid:107) 1 / (cid:107) T j (cid:107) 1 .
Results In Tab. 1, 2 and 3, we can see that for all cases on three datasets, the proposed estimationmethod leads to the smallest or second smallest estimator errors across various noise rates.
Notethat since the ﬁtting of noisy or intermediate class posterior is hard to be accurate in noisy multilabel learning, T-estimator and Dual T-estimator need to carefully tune a hyperparameter for betterestimation under different noise rates, and it’s very sensitive in some cases, e.g. MS-COCO datasetswith noise rates (0 . 1 , 0 . 1) .
In contrast, our method uses the same hyperparameters on one dataset toget good results for all cases, which reﬂects its robustness to various noise rates.
Besides, to studythe ablation of sampling bias, we also run our method with an unbias sample selection, named "ourestimator gold".
We can see that sample bias is the main factor that contributes to the error for ourestimator, but the little error gap between our estimator and our estimator gold shows it will not leadto large estimation error.
[Table]
[Table] [Table]
4.2 Comparison for classiﬁcation performance
Baselines We exploit 10 baselines: (1)Standard, which trains with a standard multi-label classiﬁcationloss.
(2)GCE [46], that proposes Generalized Cross-Entropy loss for robustness.
(3)CDR [36],that performs different update rules for two types of parameters to achieve robust learning.(4)AGCN [45], that adopts a Dynamic GCN to model the relation of content-aware class representations.
(5)CSRA [47], that generates class-speciﬁc features for every category by proposing aspatial attention score.
(6)WSIC [11] that proposes to use a small set with clean labels to learn a residual net for regularization in noisy multi-label learning, and we only provide noisy datasets to it for a
[Table]
[Table] [Table]
DualT 97%, which learns with Reweight algorithm using transition matrices estimated by Dual T-estimator 97% [44]. Note that Standard, AGCN and CSRA are designed for clean multi-label data,and GCE and CDR are designed for noisy multi-class learning.
274 Metrics Following conventional setting [7, 5, 28], we compute the mean average precision (mAP),overall F1-measure (OF1) and per-class F1-measure (CF1) as classiﬁcation evaluation metrics.
Foreach image, we assign a positive label if its prediction probability is greater than 0.5.
Results As shown in Tab. 4, 5 and 6, ﬁrst, we can ﬁnd those statistically consistent methods achievethe best or second-best results on all three metrics in the vast majority of cases, while other methodscan only achieve good results in some cases.
For example, on the Pascal-VOC2012 dataset with noiserates (0 . 0 , 0 . 6) , CSRA achieves the best result in the mAP metric with the help of its well-designednetwork, but its performance is far lower than those consistent methods on the OF1 and CF1 metrics,which shows the learned model can not approximate well the true class posterior P ( Y | X ) .
Note thatsince network structure and loss correction are compatible, the risk-consistent methods can also help
AGCN and CSRA perform better (shown in Appendix K).
Second, theoretically, the more accuratethe transition matrix is estimated, the more likely the consistent method is to achieve better resultsby increasing the size of noisy examples, and our experimental results verify this.
Among thoseconsistent methods, Rewight algorithm with our estimator (named "Reweight-Ours") obtains the mostbest or second-best results on the three metrics, which is due to the smaller error of our estimation.
Especially on the challenging and large-scale MS-COCO dataset, Reweight-Ours outperforms almostall state-of-the-art methods on the CF1 metric and signiﬁcantly surpasses other baselines by a largemargin in some cases.
For example, on the MS-COCO dataset with noise rates (0 . 6 , 0) , ReweightOurs achieves the best CF1 result (58.63 ± 1.30), while the suboptimal result is 56.70 ± 2.11, and theresult of Standard is only 7.07 ± 0.02.
In addition, the ablation studies about loss correction waysand base learning algorithms are provided in Appendix J and K, which shows that our estimator canachieve much better performance with the advanced frameworks.
Conclusion
In this paper, we study the estimation problem of the transition matrices in the noisy multi-label setting.
We prove some identiﬁability results of class-dependent transition matrices in such setting, inspiredby which we propose a new estimator to approximate the transition matrix.
The proposed estimatorutilizes the information of label correlations, and demands neither anchor points nor accurate ﬁttingof noisy class posterior.
Experiments in three popular multi-label datasets illustrate the effectiveness of the proposed estimator to accurately estimate transition matrices, and the consistent algorithmswith this estimator achieve better classiﬁcation performance.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section 2.
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Appendix G.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] SeeAppendix H.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 3.1.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix C,D,Eand F.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[Yes] We provide the source codes in the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section 4.
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes] We report the standard deviation value of results.
(d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Section 4.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 4.
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A Statistically Consistent Algorithms for Noisy Multi-label Learning
Many statistically consistent algorithms [20, 26, 38, 48] have been proposed for noisy multi-classlearning.
By decomposing the multi-label task into multiple conditionally independent binaryclassiﬁcation problems, we can apply these consistent algorithms to each binary classiﬁcationproblem for noisy multi-label learning.
Without loss of generality, we present applying a riskconsistent algorithm, i.e. Reweight [20, 38], in noisy multi-label learning here.
First of all, we decompose the task into q independent binary classiﬁcation problems given X , whichis a widely used assumption for deep multi-label learning [5, 28, 40] and the surrogate loss is as:
[Equation]
where f = ( f 1 , f 2 , ..., f q ) is the learnable q classiﬁcation functions, and (cid:96) is the base loss function.
In the deep learning commonunity, f is usually modeled by a deep nerual network with the outputsof q sigmoid functions, and (cid:96) is usually the binary cross entropy function.
Similar to the single-label case [20, 38], Reweight method employs the importance reweightingtechnique to rewrite the expected risk w.r.t.
clean data:
[Equation]
where D denotes the distribution for clean data, ¯ D for noisy data, ¯ (cid:96) j ( f j ( xxx ) , i ) =
P D ( Y j = i | XXX = xxx )
P ¯ D (¯ Y j
=
i
| XXX = xxx ) (cid:96) ( f j ( xxx ) , i ) , ¯ L ( fff ( xxx )
[Equation]
¯ yyy )
= (cid:80) qj =
[Equation]
¯
[Equation]
j ( f j ( xxx ) , ¯ Y j ) , and the third last equation holdsbecause label noise is assumed to be independent of instances.
In the paper, we have omitted thesubscript for P when no confusion is caused.
We use the output of the sigmoid function g j ( x ) to approximate P ( Y j = 1 | X = x ) , i.e, P ( Y j =1 | X = x ) ≈ ˆ P ( Y j = 1 | X = x ) =
g j ( x ) and P ( Y j = 0 | X =
x ) ≈ ˆ P
( Y j =
| X = x ) =1 − g j ( x ) .
Then, ˆ P ( ¯ Y j = k | X = x ) = (cid:80) 1 i =
T jik ˆ P ( Y j = i
| X = x ) is an approximation for
P ( ¯ Y j = k | X = x ) .
By employing Reweight algorithm, we build the risk-consistent estimator as:
[Equation]
where f j ( x ) =
I [ g j ( x ) > 0 . 5] , and I [ . ] is the indicator function which takes 1 if the identity index istrue and 0 otherwise; the subscript w denotes that the loss function is weighted.
B Validation of Assumption 2
In order to verify the assumption 2, we count the frequencies of Y i =
given Y 50 = 0 / 1 on MSCOCO training dataset, i.e. ˆ P ( Y i =
| Y 50 = 0) /
ˆ P ( Y i =
| Y 50 = 1) .
According to Hoeffding’sinequality [4], when the frequencies whose difference between given Y 50 = 0 and Y 50 = 1 is greaterthan 0.015, we have at least 95% conﬁdence to make sure P ( Y i =
| Y 50 = 0) (cid:54) =
P ( Y i =
| Y 50 =
[Table]
As shown in Tab. 7, class 50 have the correlation with another 46 classes with a high probability,which means they are very likely to hold Assumption 2, accounting for the majority of all 79 classes.
In the implementation of our estimator, we always assume class j and class i have a strong correlationat ﬁrst, and use them to estimate.
While, when a reasonable solution cannot be obtained, we willabandon class i and choose another class.
C Proof of Theorem
Lemma 1.
¯ Y i and ¯ Y j are independent given Y j .
Proof.
Since we assume that the transition matrix is class-dependent and instance-independent, ¯ Y jand any variable are independent given Y j .
Therefore, this lemma holds.
Lemma 2.
The product of two row-stochastic matrices is still a row-stochastic one.
Proof.
Let P and Q be row-stochastic matrices of the following form: P = (cid:18) 1 − p − p − p + 1 − p
+
(cid:19) and Q = (cid:18)
1 − q − q − q + 1 − q + (cid:19) .
Then the product of P and Q is P Q =(cid:18) 1 − q − + p − ( − 1 + q − + q + ) q − − p − ( − 1 + q − + q + ) q + − p
+ ( − 1 + q − + q + ) 1 − q + + p
+ ( − 1 + q − + q + ) (cid:19) .
It can be readily veriﬁed thatthe sum of each row of P Q is equal to 1 , meaning that P Q is a row-stochastic matrix.
Theorem 1.
Noisy labels { ¯ Y j , ¯ Y i } will not sufﬁce to identify T j .
Proof.
First, the information from ¯ Y j , ¯ Y i can be fully captured by the following four quantities:
P (cid:0) ¯ Y j = 0 , ¯ Y i = 0 (cid:1) , P (cid:0) ¯ Y j = 1 , ¯ Y i = 0 (cid:1) , P (cid:0) ¯ Y j = 0 , ¯ Y i = 1 (cid:1) , and P (cid:0)
¯ Y j = 1 , ¯ Y i = 1 (cid:1) .
According to Lemma 1, these four quantities can lead to four equations that depend on T j :
P (cid:0) ¯ Y j = 0 , ¯ Y i = 0 (cid:1) =
P ( Y j = 0)
T j 00 P ( ¯ Y i =
| Y j = 0)
+ P ( Y j = 1) T j 10 P ( ¯ Y i =
| Y j = 1)
P (cid:0) ¯ Y j = 0 , ¯ Y i = 1 (cid:1) =
P ( Y j = 0)
T j 00 P ( ¯ Y i =
| Y j = 0)
+ P ( Y j = 1) T j 10 P ( ¯ Y i = 1 | Y j = 1)
P (cid:0) ¯ Y j = 1 , ¯ Y i = 0 (cid:1) =
P ( Y j = 0) T j 01 P ( ¯ Y i =
| Y j = 0)
+ P ( Y j = 1) T j 11 P ( ¯ Y i =
| Y j = 1)
P (cid:0) ¯ Y j = 1 , ¯ Y i = 1 (cid:1) =
P ( Y j = 0) T j 01 P ( ¯ Y i =
| Y j = 0)
+ P ( Y j = 1) T j 11 P ( ¯ Y i =
| Y j = 1) .
For simplicity
[Equation]
we denote
[Equation]
and
[Equation]
Then
[Equation]
the system of equations can be expressed as E =
( T j ) [Equation] P M , i.e.
[Equation]
[Equation]
[Equation]
Assuming { T 0 , P 0 , M 0 } satisﬁes
[Equation]
Next, we will prove that by selecting proper parameters, a different solution of Eq.
(11) can be derived,which ruins the identiﬁability of T j .
Let A = (cid:18) 1 − a − a − a + 1 − a + (cid:19) and B = (cid:18) 1 − b − b − b + 1 − b
+
(cid:19) be invertible, row-stochastic matrices.
Based on the invertibility of A and B , Eq.
(12) can be rewrittenas E[-
-]
= ( AT 0 ) (cid:62) ( A (cid:62) )
− 1 P 0 B − 1 BM 0 .
According to Lemma 2, T 1 = AT 0 and M 1 = BM 0are row-stochastic matrices, which is consistent with the form of T j and M .
Last, denoting P 0 = (cid:18) 1 − p  p(cid:19) and P 1 = (cid:18) pp 01 pp(cid:19) , by letting P 1 =
( A (cid:62) )
− 1 P 0 B − 1 =(cid:34)(cid:18) 1 − a − a − a + 1 − a + (cid:19) (cid:62) (cid:35) − 1 (cid:18) 1 − p  p(cid:19) (cid:18) 1 − b − b − b + 1 − b + (cid:19) =
(cid:18) pp 01 pp(cid:19) be in the form of P , i.e., solving the following equations:
[Equation]
01 = 0 , p 10 = 0 , we can get a + = b − (1 − p 0 )
b − − pand b
+ = a − (1 − p 0 ) a − − p 0 .
It means that when we have a solution{ T 0 , P 0 , M 0 } of Eq.
(11), we can get another different solution { T 1 , P 1 , M 1 } by setting appropriate values of b − and a − .
Hence, T j is unidentiﬁable in this situation.
D Proof of Theorem
To make the proof clear, following [22], we reproduce the Kruskal’s identiﬁability result here.
Thesetup of Kruskal’s identiﬁability result is as follows: suppose that there is an unobserved variable
Z that takes values in { 0 , 1 , ..., K − 1 } .
Z has a non-degenerate prior P ( Z = i ) > 0 .
Instead ofobserving Z , we observe a set of conditionally independent variables (cid:8) O ( t ) (cid:9) N t =1 .
Each O ( t ) has aﬁnite state space with cardinality κ t .
Let M ( t ) be a matrix of size K × κ t , which j -th row is simply 536 (cid:2)
P (cid:0) O ( t ) = 1 | Z = j (cid:1) , . .
, P (cid:0) O ( t ) = κ t | Z = j (cid:1)(cid:3) .
The previous works [16, 32] have proved thefollowing Theorem 5.
Deﬁnition 2 (Kruskal rank [22]) .
For a matrix M , the Kruskal rank of M is the largest number Isuch that every set of I rows of M are independent.
The symbol is Kr( M )
=
I .
Theorem 5 (Kruskal’s identiﬁability result [16, 32]) .
The model parameters are uniquely identiﬁable, up to label swapping, if
[Equation]
We can prove Theorem 2 with the following lemmas:
Lemma 3.
¯ Y i and Y j are independent given Y i .
Proof.
Since we assume that the transition matrix is class-dependent and instance-independent, ¯ Y iand any variable are independent given Y i .
Therefore, this lemma holds.
Lemma 4.
If Y j ∈ { 0 , 1 } corresponds to Z , { ¯ Y j , ¯ Y
i , ¯ Y k } correspond to the observations(cid:8) O ( t ) (cid:9) 3 t =1 , then Kr (cid:16) M ( t ) (cid:17)
= 2 , t ∈ [3] .
Proof.
As M (1) =
T j is a row-stochastic matrix, according to Assumption 1, every set of 2 rows ofit are independent, thus its Kruskal rank is 2.
Therefore, according to Deﬁnition 2, Kr (cid:0) T j (cid:1) = 2 .
The ( p, q ) entry of M (2) is
[Equation]
13) where the last equation holds because of Lemma.
Denote M (cid:48) (
[Equation]
= (cid:18) P ( Y
[Equation]
= [Equation] P [Equation]
P
[Equation]
) (cid:19) .
Then Eq.
(13) can be rewrittenas the following matrix form:
[Equation]
) where T i denotes the transition matrix of class i .
According to Assumption 1, Kr (cid:0) T i (cid:1) = 2 .
As
M (cid:48) (2) is a row-stochastic matrix, according to Assumption 2, every set of 2 rows of it are independent.
Hence, according to Deﬁnition 2, Kr (cid:16) M (cid:48) (2) (cid:17)
= 2 .
Since T i and M (cid:48) (2) are full-rank matrices, based on Eq.
(13), we can get the Kruskal rank of M (2)as Kr (cid:16) M (2) (cid:17)
= 2 .
Similarly, Kr (cid:16) M (3) (cid:17) = 2 .
Theorem 2.
If ¯ Y i and ¯ Y k are independent given Y j , noisy labels { ¯ Y j ,
¯ Y i , ¯ Y k } are sufﬁcient toidentify T j .
Proof.
According to Lemma 1 and that ¯ Y i and ¯ Y k are independent given Y j , we can relate ourmulti-label noise setting to the setup of Kruscal’s identiﬁability scenario:
Y j ∈ { 0 , 1 } corresponds tothe unobserved hidden variable Z ; P ( Y j = i ) corresponds to the prior of this hidden variable; Noisylabels { ¯ Y j , ¯ Y
i , ¯ Y k } correspond to the observations (cid:8) O ( t ) (cid:9) 3 t =1 .
κ t is then simply the cardinality ofthe noisy label space, i.e. κ t =
K = 2 ; Each O ( t ) has a corresponding observation matrix M ( t ) , and
M ( t ) vk = P (cid:0) O ( t ) =
k | Y j = v (cid:1) .
Now we can get the following result about identiﬁability of T .
According to Lemma 4, the Kruskal ranks satisfy
[Equation]
N =
[Equation]
Calling Theorem 5 proves the uniqueness of M ( t ) .
As M (1) =
T j , then T j is identiﬁable.
E Proof of Theorem 3
Lemma 5.
¯ Y i , ¯ Y j and ¯ Y k are independent given Y j and Y i .
Proof.
Since we assume that the transition matrix is class-dependent and instance-independent, ¯ Y jand any variable are independent given Y j , and ¯ Y i and any variable are independent given Y i .
Therefore, this lemma holds.
Lemma 6.
If M 4 × 2 = P ( ¯ Y i | Y j , Y i ) , the ﬁrst two rows and last two rows of M are identicalrespectively, i.e. M 0 p = M 1 p and M 2 p =
M 3 p , p = 0 , 1 .
Proof.
M =
[Equation]
where the second equation holds because ¯ Y i is only dependent on Y i . Accordingly, M 0 p = M 1 p and
M 2 p =
M 3 p , p = 0 , 1 .
Theorem 3.
If ¯ Y i and ¯ Y k are not independent given Y j , noisy labels { ¯ Y j ,
¯ Y
i , ¯ Y k } will not sufﬁceto identify T j .
Proof.
The likelihood of noisy labels can be formulated as a third-order tensor L , where the ( p, q, m )entry is [Equation] According to Lemma 5, Eq. (15) can be expanded by Bayes Rule:
[Equation] where we omit the value of Y i , Y j in the last eqution for simplicity. Let M (1)4 × 2 = P ( ¯ Y j
| Y i , Y j ) ,
M (2) 4 × 2 = P ( ¯ Y i | Y i , Y j ) , M (3)4 × 2 = P ( ¯ Y k
| Y
i , Y j ) and Λ 4 × 1 = P ( Y i , Y j ) , and the Eq. (16) canbe expressed as L pqm = (cid:80) 3 {+v =
 M (2) vp M (1) vq+}
M (3) vm Λ v .
Let { A 0 , B 0 , C 0 , Λ 0 } be a solution of { M (1) , M (2) , M (3) , Λ } , which means it fulﬁls the likelihood equations (Eq.
(16)), i.e.
[Equation]
Note that as stated in Lemma. 6, the ﬁrst two rows and last two rows of B 0 are identical respectively. Next, we will show that a different solution can be constructed by simply switching the corresponding rows in A 0 , C 0 and Λ 0 , which is consistent with the result in [15, 33] when Kr( M (2) ) = 1 .
By letting
A 1 =   A A A
A
A
A
A A  , C 1 =   C
C
C
C
C
C C C  , and Λ 1 =   ΛΛΛΛ  , then the Eq. (17) is equivalent to
[Equation]
18) where the second equation holds because the ﬁrst two rows and last two rows of B 0 are identicalrespectively, i.e. B 00 p = B 01 p and B 02 p = B 03 p . Note that A 1 , C 1 and Λ 1 are consistent with theform of M (1) , M (3) and Λ respectively.
Then, according to Eq.
(18), it can be readily observed thatthe { A 1 , B 0 , C 1 , Λ 1 } is a new solution of { M (1) , M (2) , M (3) , Λ } , hence the uniqueness is notguaranteed under this circumstance.
As M (1) = (cid:18) T j T j (cid:19) is not unique, then T j is unidentiﬁable.
F Proof of Theorem 4
Theorem 4.
Noisy labels { ¯ Y j , ¯ Y i } and P ( ¯ Y i | Y j ) are sufﬁcient to identify T j .
Proof.
The proof of Theorem 4 is much similar to that of Theorem 1, we can get a system of equations expressed as E =
( T j ) (cid:62) P M .
The difference lies in that in Theorem 4, the matrix M which is parameterized by P ( ¯ Y i | Y j ) is given.
Since M is invertible (Similar to Lemma. 4), the problem can be converted to a simple bilinear decomposition problem:
[Equation]
i.e.
[Equation]
Solving the above bilinear decomposition problem, the unique solution can be obtained as:
[Equation]
Substituting Eq.
(20) into Eq.
(19), then right multiplying P − 1 on both side of the equation, the matrix T j can be derived as:
[Equation]
which indicates that T j is identiﬁable given label correlation P ( ¯ Y i | Y j ) .
G Limitations of Our Work
Our work still has certain limitations, including: 1) This work exploits the memorization effect in deep learning to perform sample selection for estimating probabilities of one noisy label when another clean label appears, while the memorization effect has not been found in other traditional machine learning methods, and therefore, our estimation
can not applied to such learning methods.
This work estimates occurrence probabilities using frequency counting.
Although this estimation error will converge to zero exponentially fast [4], when the number of one label appearing is too small, e.g. less than 50, the estimation of the transition matrix for this class label is still difﬁcult to be accurate.
H Broader Impact
In this era of deep learning, datasets are becoming bigger and bigger.
However, it is much expensiveto obtain large-scale dataset with high-quality annotated labels.
Alternatively, people can collectlabels based on non-expert annotators or automated labeling methods, e.g. web search and user tags,especially for non-proﬁt research institutions and startups.
However, these cheap annotation methodsare likely to introduce noisy labels to the collected datasets, which may pose serious threats to thesupervised learning algorithm designed for clean data.
Label-noise learning, therefore, becomes amore and more important topic recently.
Existing label-noise learning methods typically focus on the single-label case by assuming that onlyone label is corrupted.
In real applications, an example is usually associated with multiple classlabels, which could be corrupted simultaneously with their respective different probabilities.
Theproposed method is to estimate the transition matrix using label correlations in noisy multi-labeldatasets.
The transition matrix is essential to building the statistically consistent label-noise learningalgorithms.
We have shown that our method usually leads to a better estimation compared to thecurrent estimator and can improves the classiﬁcation performance of statistically consistent labelnoise learning applications, which should has a positive impact on science, society, and economy.
Hence, our approach can reduce the need for accurate labeling, and potentially, it may have a negativeimpact on the salary of label workers.
I Ablation study about the sample selection threshold
The sample selection we adopted [30, 17] is to estimate this clean probability of samples by modelingsample loss values with a GMM model using the Expectation-Maximization algorithm. If the cleansample can be distinguished according to loss values, and its estimated probability is accurate, thebest threshold will be about 0.5. Hence, τ = 0 .
5 is a typical value in related works [30, 17], andwe follow this practice in our experiments.
In this section, we conduct the ablation study about thesample selection threshold experiments on noisy VOC2007 datasets.
As shown in Fig. ??
and Fig. ??
,τ = 0 .
5 is a good choice both according to mAP scores on the noisy validation set and according tomAP scores on the clean test dataset.
[Figure]
J Comparison with different loss correction ways
Many statistically consistent algorithms [20, 26, 38] consist of a two-step training procedure.
The ﬁrststep estimates the transition matrix and the second step builds statistically consistent algorithms viamodifying loss functions.
Our proposed estimator can be seamlessly embedded into their frameworks.
In this section, we compare classiﬁcation performance of applying our estimation method to fourdifferent loss correction ways: Reweight [20] (named Reweight-Ours), Backward [25, 26, 40] (named
Backward-Ours), Forward [26] (named Forward-Ours) and T-Reversion [38] (named Reversion-Ours)on Pascal-VOC2007 dataset.
As shown in Tab. 8, T-Reversion with our estimation achieves the best performance in most cases, which may be because it can further tune the transition matrices viaclassiﬁcation learning.
[Table]
K Comparison with different base learning algorithms
Recently, many advanced multi-label learning algorithms [45, 47, 45, 3], which designed exquisitenetworks for multi-label learning
[Table]
have been proposed [Table] and they perform well on the clean dataset.
Astheir loss functions are similar to Eq.
(8), we can also apply reweight method with our estimator to theirframeworks.
In this section, we compare the classiﬁcation performance of applying Reweight methodwith our estimator to three different base learning algorithms: Standard (named "Reweight-Ours"),
AGCN [45] (named "AGCN-R-Ours") and CSRA [47] (named "CSRA-R-Ours") on Pascal-VOC2007dataset.
Besides, to show the importance of the consistent algorithms, we also present the performanceof base learning algorithms here.
As shown in Tab. 9, the consistent algorithms with our estimator canhelp base learning algorithms perform much better, especially on the OF1 and CF1 metrics.
Besides,
AGCN-R-Ours and CSRA-R-Ours perform better than Reweight-Ours in some cases, which meansthat the consistent algorithms with our estimator can work well with more advanced network
[Table]
[Table]