Estimating Noise Transition Matrix with Label Correlations for Noisy Multi-Label Learning
Abstract
In label-noise learning, the noise transition matrix , bridging the class posterior for noisy and clean data, has been widely exploited to learn statistically consistent classiﬁers.
The effectiveness of these algorithms relies heavily on estimating the transition matrix.
Recently, the problem of label-noise learning in multi-label classiﬁcation has received increasing attention, and these consistent algorithms can be applied in multi-label cases.
However, the estimation of transition matrices in noisy multi-label learning has not been studied and remains challenging, since most of the existing estimators in noisy multi-class learning depend on the existence of anchor points and the accurate ﬁtting of noisy class posterior.
To address this problem, in this paper, we ﬁrst study the identiﬁability problem of the class-dependent transition matrix in noisy multi-label learning, and then inspired by the identiﬁability results, we propose a new estimator by exploiting label correlations without neither anchor points nor accurate ﬁtting of noisy class posterior.
Speciﬁcally, we estimate the occurrence probability of two noisy labels to get noisy label correlations.
Then, we perform sample selection to further extract information that implies clean label correlations, which is used to estimate the occurrence probability of one noisy label when a certain clean label appears.
By utilizing the mismatch of label correlations implied in these occurrence probabilities, the transition matrix is identiﬁable , and can then be acquired by solving a simple bilinear decomposition problem.
Empirical results demonstrate the effectiveness of our estimator to estimate the transition matrix with label correlations, leading to better classiﬁcation performance.
Introduction
In real-world scenarios, an instance is naturally associated with multiple labels, and these labels have complex entangled correlations [7].
Recently, the problem of label-noise learning in multi-label classiﬁcation has received more and more attention [29, 34, 54, 50, 46, 47], since it is time-consuming and expensive to collect large-scale accurate labels and the noisy labels are much cheaper and easier to acquire.
In the setting of noisy multi-label learning , the multiple labels assigned to an instance may be corrupted simultaneously.
That is to say, any label for each class can be ﬂipped with its respective transition matrix that denotes the transition relationship from clean labels to noisy labels.
Transition matrix has been utilized to build a series of statistically consistent algorithms for noisy multi-class learning [32, 52, 48, 8].
The main advantage of these consistent algorithms is that they can guarantee to vanish the differences between the classiﬁers learned from noisy data and the optimal ones from clean data by increasing the size of noisy examples [28, 33, 52, 39].
Fortunately, these statistically consistent algorithms for noisy multi-class learning can also be applied in such noisy multi-label learning with a little modiﬁcation [54] (more details can be found in
Appendix B).
However, the effectiveness of these algorithms heavily relies on estimating the transition matrix.
Although the estimation of the transition matrix has been investigated in noisy multi-class learning, the estimation of the transition matrix in noisy multi-label learning has not been studied and remains challenging.
Speciﬁcally, a series of methods [28, 33, 57, 52, 26] has been proposed to estimate the transition matrix for noisy multi-class learning.
Most of them assume the existence of anchor points [28, 33, 57] that are deﬁned as the training examples belonging to a particular clean class surely.
Nevertheless, the assumption is strong and hard to check when we only have noisy data [52].
Also, the methods need to accurately ﬁt the noisy or intermediate class posterior of anchor points, which are rather difﬁcult in multi-label cases, due to severe positive-negative imbalance [36].
In this paper, to address the problem of estimating the transition matrix in noisy multi-label learning, we consider utilizing label correlations among noisy multiple labels.
Speciﬁcally, some label correlations that should not exist in practice are included in noisy multi-label learning.
For example, “ﬁsh” and “water” always co-occur, while “bird” and “sky” always co-occur.
But, due to label errors, there is a slight correlation between “ﬁsh” and “sky”, which is impractical.
At a high level, we can utilize the mismatch of label correlations to identify the transition matrix without neither anchor points nor accurate ﬁtting of noisy class posterior.
In more detail, we ﬁrst focus on the identiﬁability problem of the class-dependent transition matrix in noisy multi-label learning.
Accordingly, a new method that estimates the transition matrix by exploiting label correlations is proposed.
That is, motivated by the identiﬁability result that the label correlation of two noisy labels can not sufﬁce to identify the transition matrix in noisy multi-label learning, we utilize sample selection to extract useful information from noisy data, which implies clean label correlations to achieve the identiﬁability.
Afterward, we not only estimate the occurrence probability of two noisy labels in noisy data, but also of one noisy label when a certain clean label appears in selected data.
By utilizing the mismatch of label correlations implied in these occurrence probabilities, we can prove the identiﬁability, and transform the problem of estimating the transition matrix using label correlations into a problem of bilinear decomposition .
Finally, with easy frequency counting, we can get a good estimation of the noise transition matrix.
Empirical results illustrate the effectiveness of the proposed estimator for estimating the transition matrix in noisy multi-label learning, and the consistent algorithms with our estimator can achieve better classiﬁcation performance.
The rest of the paper is organized as follows.
In Section 2, we brieﬂy present the problem setting of label-noise learning in multi-label classiﬁcation.
In Section 3, we discuss the identiﬁability of the transition matrix under such a noisy multi-label setting, and introduce our estimation method.
Experimental results are provided in Section 4.
The limitations of this work are discussed in Section 5.
Finally, we conclude the paper in Section 6.
Problem Setting
In this section, we introduce the problem setting of label-noise learning in multi-label classiﬁcation.
In what follows, scalars are in lowercase letters, vectors are in lowercase boldface letters, and matrices/variables are in uppercase letters.
For simplicity, let [ q ] = { 1 , . . .
, q } .
Preliminaries.
Let D be the distribution of a pair of random variables ( X , Y ) , where X ∈ X
⊆
R
d denotes the variable of instances, and Y =
{ Y 1 , Y 2 , ..., Y q } ∈ { 0 , 1 } q denotes the variable of targets with q possible class labels.
As for Y , Y j = 1 indicates that the instance X is associated with the class j ;
Y j = 0 , otherwise.
In multi-label learning, the goal is to learn a function from D which maps each unseen instance x ∈ X to proper labels y .
However, as discussed, Y is hard to be annotated precisely.
Before being observed, their true labels are independently ﬂipped and what we can obtain are noisy training examples
D t
= { ( x i , ¯ y i ) } n i =1 , where ¯ y i denotes noisy labels.
Let ¯ D be the distribution of the noisy random variables ( X ,
¯ Y ) ∈
X × { 0 , 1 } q .
In noisy multi-label learning, our goal is to infer proper labels for each unseen instance by only using the noisy training examples.
Noise transition matrix.
The random variables ¯ Y j and Y j for the class j are related through a noise transition matrix T j ∈ [0 , 1] 2 × 2 , j ∈ [ q ] .
Generally, the transition matrix depends on instances, i.e., T jik ( x ) =
P ( ¯ Y j = k | Y j
=
i, X = x ) .
Nevertheless, given only noisy examples, the instance-dependent transition matrix is non-identiﬁable without any additional assumption [52, 56].
For example, both P ( ¯ Y j = k | X = x ) = (cid:80) 1 i =
T jik P
( Y j = i
| X = x ) and P ( ¯ Y j = k
| X = x ) = (cid:80) 1 i =
T (cid:48) jik P (cid:48)
( Y j = i | X = x ) are valid, when T (cid:48)
jik ( X = x ) =
T jik ( X = x ) P ( Y j = i
| X = x ) /P
(cid:48)
( ¯ Y j = i
| X = x ) .
Therefore, in this paper, we assume that the transition matrix is classdependent and instance-independent [33], i.e., P ( ¯ Y j = k | Y j
= i, X = x ) =
P ( ¯ Y j = k | Y j = i )
=
T jik .
The deﬁnition of the class-dependent label noise can be found in Appendix A, where we further discuss its differences with the class-dependent label noise in multi-class cases.
Consistent algorithms.
The transition matrix bridges the class posterior probabilities for noisy and clean data, i.e., P ( ¯ Y = k
| X = x ) = (cid:80) 1 i =
T ik P ( Y = i
| X = x ) .
Thus, it has been exploited to achieve many statistically consistent algorithms in noisy multi-class learning.
Speciﬁcally, it has been utilized to build risk-consistent estimators via correcting loss functions [28, 33, 52], and to design classiﬁer-consistent estimators via limiting hypotheses, e.g., [33, 9, 62].
Since the multi-label task can be decomposed into multiple conditionally independent binary classiﬁcation problems, we also can apply these consistent methods in noisy multi-label learning [54].
In this paper, without loss of generality, we adopt a risk-consistent algorithm, i.e., Reweight [28, 52], to learn statistically consistent classiﬁers with estimated transition matrices.
More details can be found in Appendix B.
Transition matrix estimation.
As inaccurate transition matrices will degenerate the performances of these consistent algorithms, a series of estimation methods [28, 51, 57, 63, 26] have been proposed for noisy multi-class learning to efﬁciently identify the transition matrix.
However, most of them require the assumption of anchor points [33, 57, 26], which is strong and hard to check in multi-label cases when only noisy data are provided [52].
Besides, severe positive-negative imbalance in multi-label learning [36] will make it difﬁcult to accurately approximate the noisy or intermediate class posterior of anchor points, which is crucial for these methods.
This motivates us to seek for a better estimator that can do without anchor points and avoid estimating noisy posterior in noisy multi-label learning.
Estimating Transition Matrices using Label Correlations
In this section, we ﬁrst study the identiﬁability problem [30] of class-dependent transition matrices in multi-label cases.
Furthermore, inspired by these results, we propose a new estimator to approximate the transition matrix by utilizing label correlations.
It is worth pointing out that our estimator demands neither the existence of anchor points nor accurate ﬁtting of noisy class posterior.
3.1 Identiﬁability of transition matrix
Recently, Liu et al.
[30] built identiﬁability of the noise transition matrix on the Kruskal’s identiﬁability results.
Inspired by them, with the complex correlations among class labels, we can get some identiﬁability results of the class-dependent transition matrix in noisy multi-label learning.
Following [30], to deﬁne identiﬁability, we denote an observation space by Ω .
For a general parametric space Θ , denote the distribution (probability density function) induced by the parameter θ ∈ Θ on the observation space Ω as P θ [2].
The identiﬁability for a general parametric space is deﬁned as follows:
Deﬁnition 1 (Identiﬁability [30]) .
The parameter θ is identiﬁable if P θ (cid:54)
=
P θ (cid:48) , ∀ θ (cid:54)
= θ (cid:48) .
For the class-dependent transition matrix T j for class j , the identiﬁability can be deﬁned as the following, when T j is a part of θ .
Deﬁnition 2 (Identiﬁability of T j ) .
T j is identiﬁable if P θ (cid:54) =
P θ (cid:48) for θ (cid:54)
= θ (cid:48) , up to the label permutation of class j .
Here, the label permutation of class j means swapping 1 and 0 class values in class j , and the rows of T j will also swap.
Note that Ω does not necessarily include all observable variables.
We use an example to better understand it.
For example, let Ω :=
{ ¯ Y j , X } , θ := { T j , P ( Y j | X ) } , and P θ :=
P ( ¯ Y j | X ) , the identiﬁability of T j can be achieved with the anchor point assumption [33, 57, 26].
To use notations without confusion, for the identiﬁability of T j using label correlations, we let Ω := { ¯ Y j , ¯ Y − j } , where − j means other classes having correlations with class j , θ :=
{ T j , P ( Y j ) , P ( ¯ Y − j | Y j ) } , and P θ := P ( ¯ Y j , ¯ Y − j ) .
Also, Ω does not necessarily include all noisy class labels.
[-As P ( ¯ Y j = k ) is known and P ( ¯ Y j = k ) = (cid:80) 1 i =
-]
We need Ω to provide useful information to achieve the identiﬁability.
The exploration of an effective Ω and necessary conditions is one of the focuses of the paper.
Assumption 1.
P ( ¯ Y j = 0 | Y j = 1) + P ( ¯ Y j = 1 | Y j = 0) < 1 , j ∈ [ q ] .
Assumption 1 means that the noisy label is agreed with the clean label on average, which is a standard condition for analysis under the class-dependent transition matrix [32, 31].
Assumption 2.
P ( Y i =
| Y j = 0) (cid:54) =
P ( Y i =
| Y j = 1) , i, j ∈ [ q ] and i (cid:54) =
j .
Assumption 2 means that the multiple labels have correlations between each other, which is satisﬁed by the most of ( i, j ) pairs in the real-world dataset (see Appendix C).
When considering multi-label learning, the simplest case is having two class labels.
In this case, the following theoretical results can be obtained.
Theorem 1.
Two noisy labels { ¯ Y j , ¯ Y i } will not sufﬁce to identify T j .
This result tells us that the label correlations of two noisy labels can not offer enough information to achieve the identiﬁability of T j .
We provide Theorem 2 based on the Kruskal’s identiﬁability result [21, 40].
Theorem 2.
If ¯ Y i and ¯ Y k are independent given Y j , three noisy labels { ¯ Y j ,
¯ Y i , ¯ Y k } are sufﬁcient to identify T j .
The assumption that ¯ Y i and ¯ Y k are independent given Y j can be satisﬁed in certain cases, e.g., the occurrences of “blue” and “dolphin” may be independent given “sea” appearing or not.
Nevertheless, due to the complex correlations among labels, this assumption is hard to hold in most cases.
When the assumption can not hold, these label correlations are no more sufﬁcient to determine T j , as shown in Theorem 3 in the following.
Theorem 3.
If ¯ Y i and ¯ Y k are not independent given Y j , three noisy labels { ¯ Y j ,
¯ Y
i , ¯ Y k } will not sufﬁce to identify T j .
The inspiration from Theorem 3 is that, the increase of the number of noisy labels may make the identiﬁability decrease due to the entangled correlations (see Appendix G and I).
To handle this problem, we prove Theorem 4 by assuming the transition relationship between noisy label for class i and clean label for class j is known.
Theorem 4.
If P ( ¯ Y i | Y j ) is known, two noisy labels { ¯ Y j , ¯ Y i } are sufﬁcient to identify T j .
Theorem 4 theoretically guarantees that the identiﬁability of the class-dependent transition matrix can be achieved by utilizing the occurrence probabilities P ( ¯ Y i , ¯ Y j ) and P ( ¯ Y i | Y j ) .
Note that
P ( ¯ Y i , ¯ Y j ) can represent noisy label correlations, and P ( ¯ Y i | Y j )
=
P ( ¯ Y i
| Y i )
P
( Y i | Y j ) , which can imply clean label correlations.
At a high level, the mismatch of label correlations implied in the occurrence probabilities can achieve the identiﬁability.
The detailed proof of Theorem 1-4 is provided in Appendix E-H.
3.2 Our estimator
Our estimator is based on Theorem 4, which needs extra information to estimate P ( ¯ Y i | Y j ) .
Recently, the memorization effect [4] of deep networks has received much attention in learning with noisy labels, which shows that deep networks will ﬁrst memorize the training data with clean labels and then those with noisy labels.
Prior works utilize this characteristic to develop sample selection methods [16, 13, 22, 17, 25], where we select some examples with more likely clean labels for each class j respectively in the early learning phase.
The selected examples can serve as the useful extra information that implies clean label correlations, through which we can achieve estimation via counting.
However, when implementing sample-selection-based methods, a major concern is whether the sampling bias will lead to large estimation errors.
Generally speaking, according to the memorization effect , the sampling bias is about selecting easy examples for class j , which usually means these examples have easy-to-discriminate features.
We can reasonably assume that given Y j , the distribution of the features about class j is biased, while the distribution of the features about another class i is unbiased, i.e.,
[Equation]
(1) where D js is the distribution of ( X , Y s ) ,
Y s = { ¯ Y 1 , ¯ Y 2 , ..., Y j , ..., ¯ Y q } , ¯ D js is the biased distribution of ( X , Y s ) , and X i is the part of X , which represents all information about class i appearing or not.
When the assumption is satisﬁed, the sample selection will not lead to large estimation errors on
P ( ¯ Y i | Y j ) , and it can converge to zero exponentially fast by counting [6].
In real-world scenarios, due to the complex label correlations, this assumption will not strictly hold.
While, it may be roughly met when class labels i and j do not share the major discriminative features.
Intuitively speaking, the classifying of simplest examples for one label is not easily affected by the presence or absence of other signiﬁcantly different labels.
Also, since most label pairs from typical real-world multi-label datasets are signiﬁcantly different (see Appendix D), the assumption can be roughly hold for those label pairs in typical cases.
In Section 4.1, our empirical results justify this by showing a little gap between the estimation error of our estimator with a biased sample selection and an unbiased one.
Based on the above discussions, we can approximate P ( ¯ Y i , ¯ Y j ) and P ( ¯ Y i | Y j ) with frequency counting, and utilize the mismatch of label correlations implied in these occurrence probabilities to estimate the transition matrix.
In this work, we propose to estimate transition matrices { T j } qj =1 by following two stages.
In the ﬁrst stage, we utilize sample selection to obtain useful information that implies clean label correlations.
Speciﬁcally, we train a classiﬁer with the standard multi-label classiﬁcation loss on noisy training examples D t for a few epochs, and then perform sample selection to get a selected clean set D js for each class label j .
Specially, we use a commonly used sample selection way [3, 22] in learning with noisy labels, which extracts the subset of examples with small losses by modeling the distribution of losses for class j with a Gaussian mixture model (GMM).
In the second stage, we perform co-occurrence estimation by frequency counting, and then estimate the transition matrix by solving a simple bilinear decomposition problem.
For class label j , we ﬁrst choose another class label i and estimate P ( ¯ Y i , ¯ Y j ) and P ( ¯ Y i | Y j ) via counting, i.e., ˆ P
[Equation]
where I [ · ] is the indicator function which takes 1 if the identity index is true and 0 otherwise.
Then, these co-occurrence probabilities, which imply the mismatch of label correlations, can lead to four equations involving T j :
P
[Equation]
¯ Y j = 0 , ¯ Y i = 0 (cid:1) =
P ( Y j = 0)
T j 00 P ( ¯ Y i =
| Y j = 0)
+ P ( Y j = 1) T j 10 P ( ¯ Y i =
| Y j = 1) ,
P (cid:0) ¯ Y j = 0 , ¯ Y i = 1 (cid:1) =
P ( Y j = 0)
T j 00 P ( ¯ Y i =
| Y j = 0)
+ P ( Y j = 1) T j 10 P ( ¯ Y i = 1 | Y j = 1) ,
P (cid:0) ¯ Y j = 1 , ¯ Y i = 0 (cid:1) =
P ( Y j = 0) T j 01 P ( ¯ Y i =
| Y j = 0)
+ P ( Y j = 1) T j 11 P ( ¯ Y i =
| Y j = 1) ,
P (cid:0) ¯ Y j = 1 , ¯ Y i = 1 (cid:1) =
P ( Y j = 0) T j 01 P ( ¯ Y i =
| Y j = 0)
+ P ( Y j = 1) T j 11 P ( ¯ Y i =
| Y j = 1) .
For simplicity, we denote
E
[Equation]
( ¯ Y j = 0 , ¯ Y i = 1)
P ( ¯ Y j = 1 , ¯ Y i = 0) P ( ¯ Y j = 1 , ¯ Y i = 1) (cid:19) =
(cid:18) ee 01 ee(cid:19) ,
P = (cid:18) P ( Y j = 0)
P ( Y j = 1) (cid:19) =
(cid:18) 1 − p p (cid:19) ,
T j = (cid:18) P ( ¯ Y j =
| Y j = 0) P ( ¯ Y j = 1 | Y j = 0)
P ( ¯ Y j = 0 | Y j = 1) P ( ¯ Y j = 1 | Y j = 1) (cid:19) =
(cid:18) 1 − ρ − ρ − ρ + 1 − ρ + (cid:19) , and
M = (cid:18) P ( ¯ Y i =
| Y j = 0) P ( ¯ Y i =
| Y j = 0)
P ( ¯ Y i =
| Y j = 1) P ( ¯ Y i =
| Y j = 1) (cid:19) =
(cid:18) 1 − ρ (cid:48)− ρ (cid:48)− ρ (cid:48) + 1 − ρ (cid:48)
+ (cid:19) .
Then the system of equations can be expressed as E =
( T j ) (cid:62) P M , i.e., (cid:18) ee 01 ee(cid:19)
= (cid:18) 1 − ρ − ρ − ρ + 1 − ρ + (cid:19) (cid:62) (cid:18) 1 − p p (cid:19) (cid:18) 1 − ρ (cid:48) − ρ (cid:48)
− ρ (cid:48)
+ 1 − ρ (cid:48)
+ (cid:19) .
Denote the estimation of E , P , T j , and M as ˆ E = (cid:18) ˆ eˆ eˆ eˆ e
[Equation]
As ˆ E and ˆ M can be derived from Eq.
(2) and Eq.
(3), the problem is hence equivalent to a bilinear decomposition problem:
[Equation]
By solving the above matrix equation, we can get
[Equation]
and the estimation of the transition matrix
[Equation]
Implementation of our estimator.
The pseudo code of our estimator is described in Algorithm 1.
A little difference from the above is that in order to make better use of correlations among labels, we perform R times co-occurrence estimation and bilinear decomposition for different classes i in the second stage to get R estimations, ˆ T j r , r = 1 , 2 , ..., R .
Finally, we estimate the transition matrix T j by Eq.
[Equation]
where (cid:107) · (cid:107) 1 denotes (cid:96) 1 norm.
Require:
Noisy training examples D t , the number of classes q , the early warmup training epoch
E warm , the threshold of sample selection τ , and repeated estimation times R .
..., q do 7: for r = 1 , 2 , ..., R do 8:
Choose another class label i .
Estimate P ( ¯ Y i , ¯ Y j ) by ˆ P ( ¯ Y
i , ¯ Y j ) with Eq.
(2) on D t , and P ( ¯ Y i | Y j ) by ˆ P ( ¯ Y i | Y j ) with Eq.
(3) on D js .
Solve a bilinear decomposition problem (Eq. (4)) to get a estimation ˆ T
j r by Eq.
11: end for 12:
Estimate T j by ˆ T j which has the minimum error with Eq.
(7) from R estimations.
13: end for
Ensure: The estimated transition matrices { ˆ T j } qj =1 .
Experiments
Dataset We verify the effectiveness of the proposed method on three synthetic noisy multilabel datasets, i.e., Pascal-VOC2007 [11], Pascal-VOC2012 [12], and MS-COCO [27].
PascalVOC2007 [11] and Pascal-VOC2012 [12] datasets are two popular datasets for object recognition.
They both contain images from the same 20 object classes, with an average of n a = 1 .
5 labels per image.
Pascal-VOC2007 contains a training set of 5,011 images and a test set of 4,952 images.
Pascal-VOC2012 consists of 11,540 images as the training set and 10,991 images as the test set [5].
As the labels of the test set in Pascal-VOC2012 are not publicly available, we use the test set in
Pascal-VOC2007 for Pascal-VOC2012 evaluation.
MS-COCO [27] is a widely used multi-label dataset.
It contains 82,081 images as the training set and 40,137 images as the test set and coversobject classes with an average of n a = 2 .
9 labels per image.
For these datasets, we corrupted the training sets manually according to true transition matrices { T j } qj =1 .
For convenience, we use the same true transition matrices for all classes, i.e., T j = T = (cid:18) 1 − ρ − ρ −
[Equation]
+ 1 − ρ + (cid:19) , but do not divulge this information for algorithms.
We generate four different types of synthetic datasets by using different transition matrices: 1) ρ − = 0 , ρ + = ρ , which annotates some positive examples as negative examples, also known as multi-label learning with missing labels [45, 44]; 2) ρ − = ρ, ρ + = 0 , which annotates some negative examples as positive examples, also known as partial multi-label learning [53, 55]; 3) ρ − = ρ, ρ + = ρ , where positive examples and negative examples are mislabeled with the same probability ρ ; 4) ρ − = n a q − n a ρ, ρ + = ρ , where positive examples and negative examples are mislabeled with the same number.
In the experiments, we test the algorithms on various ρ .
For all datasets, we leave out 10% of the noisy training examples as a noisy validation set.
We use mAP on noisy validation set as the criterion for model selection.
Implementation details For a fair comparison, we implement all methods with default parameters by PyTorch on NVIDIA RTX 3090.
We use a ResNet-50 network [14] pre-trained on ImageNet [37] for all datasets, and the optimizer is Adam optimizer [18] with β = (0 . 9 , 0 . 999) .
The batch size is 128, the learning rate is 5e-5.
The number of training epochs is 20 for Pascal-VOC2007/VOC2012, and 30 for MS-COCO.
For the transition matrix estimation method, E warm is the same as the normal training epoch.
For our estimator, we perform sample selection based on the average losses ofepochs before a certain warmup epoch (10th epoch for Pascal-VOC2007/VOC2012, 15th epoch for MS-COCO), R = q − 1 and τ = 0 .
5 in all experiments.
All experiments are run at least three times with different random seeds, and we report the average and standard deviation values of results.
4.1 Comparison for estimating transition matrices
Baselines For evaluating the effectiveness of estimating the transition matrix under multi-label cases, we compare the proposed method with the following methods: (1) T-estimator max [28, 33], which estimates the transition matrix via the noisy class posterior probabilities for anchor points that have the largest estimated noisy class posteriors.
(2) T-estimator 97% [28, 33], which selects the points with 97% largest estimated noisy class posteriors to be anchor points.
(3) Dual T-estimator max [57], which introduces an intermediate class to avoid directly estimating the noisy class posterior, and selects the points with the largest estimated intermediate class posteriors to be the anchor points.
Dual T-estimator 97% [57], which selects the points with the 97% largest estimated intermediate class posteriors to be the anchor points.
Metrics We use the sum of estimation error for the transition matrices as the estimation evaluation metric, i.e.,
[Equation]
Results In Tab. 1, 2 and 3, we can see that for all cases on three datasets, the proposed estimation method leads to the smallest or second-smallest estimator errors across various noise rates.
Note that since the ﬁtting of noisy or intermediate class posterior is hard to be accurate in noisy multilabel learning, T-estimator and Dual T-estimator need to carefully tune a hyperparameter for better estimation under different noise rates, and it’s very sensitive in some cases, e.g., MS-COCO datasets with noise rates (0 . 1 , 0 . 1) .
In contrast, our method uses the same hyperparameters on one dataset to get good results for all cases, which reﬂects its robustness to various noise rates.
Besides, to study the ablation of sampling bias, we also run our method with an unbiased sample selection, named “our estimator gold”.
We can see that sample bias is the main factor that contributes to the error for our estimator, but the little error gap between our estimator and our estimator gold shows it will not lead to large estimation errors.
[Table]
[Table]
on MS-COCO dataset.
[Table]
4.2 Comparison for classiﬁcation performance Baselines
We exploit 10 baselines: (1) Standard, which trains with a standard multi-label classiﬁcation loss.
(2) GCE [60], which proposes a Generalized Cross-Entropy loss for robustness.
CDR [49], which performs different update rules for two types of parameters to achieve robust learning.
(4) AGCN [58], which adopts a dynamic GCN to model the relation of content-aware class representations.
(5) CSRA [61], which generates class-speciﬁc features for every category by proposing a spatial attention score.
(6) WSIC [15] that proposes to use a small set with clean labels to learn a residual net for regularization in noisy multi-label learning, and we only provide noisy datasets to it for a fair comparison.
(7) Reweight-T max, which learns with a reweighting algorithm using transition matrices estimated by T-estimator max [33].
(8) Reweight-T 97%, which learns with a reweighting algorithm using transition matrices estimated by T-estimator 97% [33].
Reweight-DualT max, which learns with a reweighting algorithm using transition matrices estimated by Dual T-estimator max [57].
(10) Reweight-DualT 97%, which learns with a reweighting algorithm using transition matrices estimated by Dual T-estimator 97% [57].
Note that Standard, AGCN, and
CSRA are designed for clean multi-label data, and GCE and CDR are designed for noisy multi-class learning.
Metrics Following conventional setting [11, 7, 36, 35], we compute the mean average precision (mAP), overall F1-measure (OF1) and per-class F1-measure (CF1) as classiﬁcation evaluation metrics.
For each image, we assign a positive label if its prediction probability is greater than 0.5.
Results As shown in Tab. 4, 5 and 6, ﬁrst, we can ﬁnd those statistically consistent methods achieve the best or second-best results on all three metrics in the vast majority of cases, while other methods can only achieve good results in some cases.
For example, on the Pascal-VOC2012 dataset with noise rates (0 . 0 , 0 . 6) , CSRA achieves the best result in the mAP metric with the help of its well-designed network, but its performance is far lower than those consistent methods on the OF1 and CF1 metrics, which shows the learned model can not approximate well the true class posterior P ( Y | X ) .
Note that since network structure and loss correction are compatible, the risk-consistent methods can also help AGCN and CSRA perform better (shown in Appendix M).
Second, theoretically, the more accurate the transition matrix is estimated, the more likely the consistent method is to achieve better results by increasing the size of noisy examples, and our experimental results verify this.
Among those consistent methods, the reweighting algorithm with our estimator (named "Reweight-Ours") obtains the best or second-best results on the three metrics, which is due to the smaller error of our estimation.
Especially on the challenging and large-scale MS-COCO dataset, Reweight-Ours outperforms almost all state-of-the-art methods on the CF1 metric and signiﬁcantly surpasses other baselines by a large margin in some cases.
For example, on the MS-COCO dataset with noise rates (0 . 6 , 0) , Reweight-Ours achieves the best CF1 result (58.63 ± 1.30), while the suboptimal result is 56.70 ± 2.11, and the result of Standard is only 7.07 ± 0.02.
In addition, the ablation studies about loss correction ways and base learning algorithms are provided in Appendix L and M, which shows that our estimator can achieve much better performance with the advanced frameworks.
Besides, although our method is based on the assumption of class-dependent label noise, the experiments with two types of instance-dependent label noise are provided in Appendix O.
[Table]
[Table] [Table]
Limitations
This work still has certain limitations, including: 1) This work exploits the memorization effect [4] in deep learning to perform sample selection, while the memorization effect has not been found in other traditional machine learning methods, and therefore, the proposed estimator can not be applied to such learning methods.
This work estimates occurrence probabilities using frequency counting.
Although this estimation error will converge to zero exponentially fast [6], when the number of one label appearing is too small, e.g., less than 50, the estimation of the transition matrix for this class label is still difﬁcult to be accurate.
3) Since our work assumes label noise is class-dependent but instance-independent, when this assumption does not hold, the estimation is not guaranteed.
The discussions about the relaxation of instance-independent assumption can be found in Appendix N, which reveals its applicability in certain typical instance-dependent cases.
Conclusion
In this paper, we study the estimation problem of the transition matrices in the noisy multi-label setting.
We prove some identiﬁability results of class-dependent transition matrices in such a setting, inspired by which we propose a new estimator to approximate the transition matrix.
The proposed estimator utilizes the information of label correlations, and demands neither anchor points nor accurate ﬁtting of noisy class posterior.
Experiments on three popular multi-label datasets illustrate the effectiveness of the proposed estimator to accurately estimate transition matrices, and the consistent algorithms with this estimator achieve better classiﬁcation performance.
Acknowledgements
Yibing Zhan was partially supported by the Major Science and Technology Innovation 2030 "
New
Generation Artiﬁcial Intelligence" Key Project (No. 2021ZD0111700) and the National Natural
Science Foundation of China (No. 62002090).
Shiming Ge was partially supported by grants from the
Beijing Natural Science Foundation (L192040), and National Natural Science Foundation of China (
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or [N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section 2.
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Appendix Q. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 3.1.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix E,F,G and H. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We provide the source codes at https://github.com/ShikunLi/Estimating_T_For_Noisy_Mutli-Labels.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section 4.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] We report the standard deviation value of results.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Section 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 4.
(b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Deﬁnition of Class-dependent Label Noise
In this paper, the class-dependent multi-label noise for class label Y j means that the ﬂip probabilities of ¯ Y j are only dependent on the value of class label Y j , i.e., Y j = 0 or Y j = 1 .
And the corresponding class-dependent transition matrix represents P ( ¯ Y j | Y j ) .
The differences between this deﬁnition and the class-dependent label noise in the single-label cases are as follows:
First, the class-dependent label noise in the single-label cases represents the ﬂip probability from class i to class j ( i and j are two different classes), while in this paper, the class-dependent label noise for class label Y j is only dependent on Y j = 0 or Y j = 1 , which is independent on another class label Y
i , i.e., P (cid:0) ¯ Y j | Y j , Y i (cid:1) =
P (cid:0) ¯ Y j | Y j (cid:1) .
Second, the class-dependent label noise in the single-label cases can be modeled by a C × C transition matrix, bridging the transition from clean single label to noisy single label, while in this paper, the class-dependent label noise for class label Y j can be modeled by a 2 × 2 transition matrix, bridging the transition from clean label Y j to noisy label ¯ Y j .
Third, the deﬁnition of class-dependent multi-label noise in this paper can be extended to instancedependent multi-label noise, where the ﬂip probabilities of ¯ Y j are dependent on class label Y j and instance feature X .
Such instance-dependent label model can sufﬁciently model various multilabel noise cases such as missing multi-labels [45, 44], partial multi-labels [53, 55], pair-wise label noise [13, 23, 24], and PMD label noise [59].
While the instance-dependent label noise in the single-label cases can not simultaneously model such complex multi-label noise.
B Statistically Consistent Algorithms for Noisy Multi-label Learning
Many statistically consistent algorithms [28, 33, 52, 62] have been proposed for noisy multi-class learning.
By decomposing the multi-label task into multiple conditionally independent binary classiﬁcation problems, we can apply these consistent algorithms to each binary classiﬁcation problem for noisy multi-label learning.
Without loss of generality, we present applying a riskconsistent algorithm, i.e., Reweight [28, 52], in noisy multi-label learning here.
First of all, we decompose the task into q independent binary classiﬁcation problems given X , which is a widely used assumption for deep multi-label learning [7, 36, 54] and the surrogate loss is as:
[Equation]
where f = ( f 1 , f 2 , ..., f q ) is the learnable q classiﬁcation functions, and (cid:96) is the base loss function.
In the deep learning commonunity, f is usually modeled by a deep nerual network with the outputs of q sigmoid functions, and (cid:96) is usually the binary cross entropy function.
Similar to the single-label case [28, 52], Reweight method employs the importance reweighting technique to rewrite the expected risk w.r.t.
clean data:
[Equation]
where D denotes the distribution for clean data, ¯ D for noisy data, ¯ (cid:96) j ( f j ( xxx ) , i ) =
P D ( Y j = i | XXX = xxx )
P ¯ D (¯ Y j
=
i
| XXX = xxx ) (cid:96) ( f j ( xxx ) , i ) , ¯ L ( fff ( xxx ) , ¯ yyy ) =
(cid:80) qj =1 ¯ (cid:96) j
( f j ( xxx ) , ¯ Y j ) , and the third last equation holds because label noise is assumed to be independent of instances.
In the paper, we have omitted the subscript for P when no confusion is caused.
We use the output of the sigmoid function g j ( x ) to approximate P ( Y j = 1 | X = x ) , i.e, P ( Y j = 1 | X = x ) ≈ ˆ P ( Y j = 1 | X = x ) =
g j ( x ) and P ( Y j = 0 | X =
x ) ≈ ˆ P
( Y j =
| X = x ) = 1 − g j ( x ) .
Then, ˆ P ( ¯ Y j = k | X = x ) = (cid:80) 1 i =
T jik ˆ P ( Y j = i
| X = x ) is an approximation for
P ( ¯ Y j = k | X = x ) .
By employing Reweight algorithm, we build the risk-consistent estimator as:
[Equation]
where f j ( x ) =
I [ g j ( x ) > 0 . 5] , and I [ . ] is the indicator function which takes 1 if the identity index is true and 0 otherwise; the subscript w denotes that the loss function is weighted.
C Validation of Assumption
[Table]
In order to verify the assumption 2, we count the frequencies of Y i =
given Y 50 = 0 / 1 on MSCOCO training dataset, i.e., ˆ P ( Y i =
| Y 50 = 0) /
ˆ P ( Y i =
| Y 50 = 1) .
According to Hoeffding’s inequality [6], when the frequencies whose difference between given Y 50 = 0 and Y 50 = 1 is greater than 0.015, we have at least 95% conﬁdence to make sure P ( Y i =
| Y 50 = 0) (cid:54) =
P ( Y i =
| Y 50 = 1) .
As shown in Tab. 7, class 50 have the correlation with another 46 classes with a high probability, which means they are very likely to hold Assumption 2, accounting for the majority of all 79 classes.
In the implementation of our estimator, we always assume class j and class i have a strong correlation at ﬁrst, and use them to estimate.
While, when a reasonable solution cannot be obtained, we will abandon class i and choose another class.
D Discussion about Signiﬁcantly Different Label Pairs in Real-world Datasets
Although the real-world scenarios have complex label correlations, we claim signiﬁcantly different class label pairs usually account for the majority of all label pairs in the typical multi-label datasets, e.g., MS-COCO [27] and OpenImages [19] datasets.
It is because among a large number of classes, most label pairs belong to signiﬁcantly different superclasses.
For example, in MS-COCO datasets, there are 80 classes, which belong to 10 signiﬁcantly different superclasses (outdoor, food, indoor, appliance, sports, person, animal, vehicle, furniture, accessory, electronic, kitchen).
In OpenImages dataset, there are 19,957 class labels, and also have signiﬁcantly different superclasses, such as Toy,
Budilding, Medical equipment, Clothing, Insect and so on.
A part of these superclasses can be seen in [1], which clearly shows most of the label pairs are signiﬁcantly different and do not share the major discriminant features.
E Proof of Theorem
Lemma 1.
¯ Y i and ¯ Y j are independent given Y j .
Proof.
Since we assume that the transition matrix is class-dependent and instance-independent, ¯ Y j and any variable are independent given Y j .
Therefore, this lemma holds.
Lemma 2.
The product of two row-stochastic matrices is still a row-stochastic one.
Proof.
Let P and Q be row-stochastic matrices of the following form: P =
[Equation]
and Q
[Equation]
Then the product of P and Q is P Q =
[Equation]
It can be readily veriﬁed that the sum of each row of P Q is equal to 1 , meaning that P Q is a row-stochastic matrix.
Theorem 1.
Two noisy labels { ¯ Y j , ¯ Y i } will not sufﬁce to identify T j .
Proof.
First, the information from ¯ Y j , ¯ Y i can be fully captured by the following four quantities:
P (cid:0) ¯ Y j = 0 , ¯ Y i = 0 (cid:1) , P (cid:0) ¯ Y j = 1 , ¯ Y i = 0 (cid:1) , P (cid:0) ¯ Y j = 0 , ¯ Y i = 1 (cid:1) , and P (cid:0)
¯ Y j = 1 , ¯ Y i = 1 (cid:1) .
According to Lemma 1, these four quantities can lead to four equations that depend on T j :
P (cid:0)
[Equation]
, ¯
[Equation]
=
[Equation]
=
[Equation]
| Y j = 0) + P ( Y j = 1) T j 10 P ( ¯ Y i =
| Y j = 1)
P (cid:0) ¯ Y j = 0 , ¯ Y i = 1 (cid:1) =
P ( Y j = 0)
T j 00 P
[Equation]
¯ Y
i
[Equation] [Equation]
Y j
[Equation] [Equation]
P
[Equation]
Y j = 1)
[Equation] [Equation]
Y i =
| Y j = 1)
P (cid:0) ¯ Y j = 1 , ¯ Y i = 0 (cid:1) =
P ( Y j = 0) T j 01 P ( ¯ Y i =
| Y j = 0)
+ P ( Y j = 1)
[Equation]
j 11 P ( ¯ Y i =
| Y j = 1)
P (cid:0) ¯ Y j = 1 , ¯ Y i = 1 (cid:1)
[Equation]
[Equation] [Equation]
j
[Equation] [Equation]
i
[Equation] [Equation]
Y j = 0)
+ P ( Y j = 1) T j 11 P ( ¯ Y i =
| Y j = 1) .
For simplicity, we denote
E =
[Equation]
=
[Equation] [Equation]
[Equation]
) (cid:19)
= (cid:18)
[Equation]
ρ − ρ − ρ + 1 − ρ + (cid:19) , and
M = (cid:18)
[Equation]
Then, the system of equations can be expressed as E =
( T j ) (cid:62) P M , i.e.,
[Equation]
Assuming { T 0 , P 0 , M 0 } satisﬁes
[Equation]
Next, we will prove that by selecting proper parameters, a different solution of Eq.
(11) can be derived, which ruins the identiﬁability of T j .
Let A = (cid:18) 1 − a − a − a + 1 − a + (cid:19) and B = (cid:18) 1 − b − b − b + 1 − b
+
(cid:19) be invertible, row-stochastic matrices.
Based on the invertibility of A and B , Eq.
(12) can be rewritten as E =
( AT 0 ) (cid:62) ( A (cid:62) )
− 1 P 0 B − 1 BM 0 .
According to Lemma 2, T 1 =
AT 0 and M 1 =
BMare row-stochastic matrices, which is consistent with the form of T j and M .
Last, denoting P 0 = (cid:18) 1 − p  p(cid:19) and P 1 = (cid:18) pp 01 pp(cid:19) , by letting P 1 =
( A (cid:62) )
− 1 P 0 B − 1 =
(cid:34)(cid:18) 1 − a − a − a
+
[Equation]
+
[Equation]
be in the form of
P , i.e., solving the following equations:
[Equation]
we can get a + = b − (1 − p 0 )
b − − pand b
+ = a − (1 − p 0 ) a − − p 0 .
It means that when we have a solution { T 0 , P 0 , M 0 } of Eq.
(11), we can get another different solution { T 1 , P 1 , M 1 } by setting appropriate values of b − and a − .
Hence, T j is unidentiﬁable in this situation.
F Proof of Theorem
To make the proof clear, following [30], we reproduce the Kruskal’s identiﬁability result here.
The setup of Kruskal’s identiﬁability result is as follows: suppose that there is an unobserved variable
Z that takes values in { 0 , 1 , ..., K − 1 } .
Z has a non-degenerate prior P ( Z = i ) > 0 .
Instead of observing Z , we observe a set of conditionally independent variables (cid:8) O ( t ) (cid:9) N t =1 .
Each O ( t ) has a ﬁnite state space with cardinality κ t .
Let M ( t ) be a matrix of size K × κ t , which j -th row is simply (cid:2)
P (cid:0) O ( t ) = 1 | Z = j (cid:1) , . .
, P (cid:0) O ( t ) = κ t | Z = j (cid:1)(cid:3) .
The previous works [21, 40] have proved the following Theorem 5.
Deﬁnition 3 (Kruskal rank [30]) .
For a matrix M , the Kruskal rank of M is the largest number I such that every set of I rows of M are independent.
The symbol is Kr( M )
=
I .
Theorem 5 (Kruskal’s identiﬁability result [21, 40]) .
The model parameters are uniquely identiﬁable, up to label permutation, if
[Equation]
We can prove Theorem 2 with the following lemmas:
Lemma 3.
¯ Y i and Y j are independent given Y i .
Proof.
Since we assume that the transition matrix is class-dependent and instance-independent, ¯ Y i and any variable are independent given Y i .
Therefore, this lemma holds.
Lemma 4.
If Y j ∈ { 0 , 1 } corresponds to Z , { ¯ Y j , ¯ Y
i , ¯ Y k } correspond to the observations (cid:8) O ( t ) (cid:9) 3 t =1 , then Kr (cid:16) M ( t ) (cid:17)
= 2 , t ∈ [3] .
Proof.
As M (1) =
T j is a row-stochastic matrix, according to Assumption 1, every set of 2 rows of it are independent, thus its Kruskal rank is 2.
Therefore, according to Deﬁnition 3, Kr (cid:0) T j (cid:1) = 2 .
The ( p, q ) entry of M (2) is
[Equation]
where the last equation holds because of Lemma.
Denote M (cid:48)
[Equation]
=
[Equation]
Then Eq.
(13) can be rewritten as the following matrix form:
[Equation]
where T i denotes the transition matrix of class i .
According to Assumption 1, Kr (cid:0) T i (cid:1) = 2 .
As
M (cid:48) (2) is a row-stochastic matrix, according to Assumption 2, every set of 2 rows of it are independent.
Hence, according to Deﬁnition 3, Kr (cid:16) M (cid:48) (2) (cid:17)
= 2 .
Since T i and M (cid:48) (2) are full-rank matrices, based on Eq.
(13), we can get the Kruskal rank of M (2) as Kr (cid:16) M (2) (cid:17)
= 2 .
Similarly, Kr (cid:16) M (3) (cid:17) = 2 .
Theorem 2.
If ¯ Y i and ¯ Y k are independent given Y j , three noisy labels { ¯ Y j ,
¯ Y i , ¯ Y k } are sufﬁcient to identify T j .
Proof.
According to Lemma 1 and that ¯ Y i and ¯ Y k are independent given Y j , we can relate our multi-label noise setting to the setup of Kruscal’s identiﬁability scenario:
Y j ∈ { 0 , 1 } corresponds to the unobserved hidden variable Z ; P ( Y j = i ) corresponds to the prior of this hidden variable; Noisy labels { ¯ Y j , ¯ Y
i , ¯ Y k } correspond to the observations (cid:8) O ( t ) (cid:9) 3 t =1 .
κ t is then simply the cardinality of the noisy label space, i.e., κ t =
K = 2 ; Each O ( t ) has a corresponding observation matrix M ( t ) , and M ( t ) vk =
P (cid:0) O ( t ) = k
| Y j = v (cid:1) .
Now we can get the following result about identiﬁability of T .
According to Lemma 4, the Kruskal ranks satisfy
[Equation]
Calling Theorem 5 proves the uniqueness of M ( t ) .
As M (1) =
T j , then T j is identiﬁable.
G Proof of Theorem
Lemma 5.
¯ Y i , ¯ Y j and ¯ Y k are independent given Y j and Y i .
Proof.
Since we assume that the transition matrix is class-dependent and instance-independent, ¯ Y j and any variable are independent given Y j , and ¯ Y i and any variable are independent given Y i .
Therefore, this lemma holds.
Lemma 6.
If M 4 × 2 = P ( ¯ Y i
| Y j , Y i ) , the ﬁrst two rows and last two rows of M are identical respectively, i.e., M 0 p =
M 1 p and M 2 p =
M 3 p , p = 0 , 1 .
Proof.
M =  
P ( ¯ Y i =
| Y j = 0 , Y i = 0) P ( ¯ Y i =
| Y j = 0 , Y i = 0)
P ( ¯ Y i =
| Y j = 1 , Y i = 0) P ( ¯ Y i = 1 | Y j = 1 , Y i = 0)
P ( ¯ Y i =
| Y j = 0 , Y i = 1) P ( ¯ Y i =
| Y j = 0 , Y i = 1)
P ( ¯ Y i =
| Y j = 1 , Y i = 1) P ( ¯ Y i =
| Y j = 1 , Y i = 1) 
 =  
P ( ¯ Y i =
| Y i = 0)
P ( ¯ Y i = 1 | Y i = 0)
P ( ¯
[Equation]
where the second equation holds because ¯ Y i is only dependent on Y i .
Accordingly, M 0 p =
M 1 p and
M 2 p =
M 3 p , p = 0 , 1 .
Theorem 3.
If ¯ Y i and ¯ Y k are not independent given Y j , three noisy labels { ¯ Y j ,
¯ Y
i , ¯ Y k } will not sufﬁce to identify T j .
Proof.
The likelihood of noisy labels can be formulated as a third-order tensor L , where the ( p, q, m ) entry is
[Equation]
According to Lemma 5, Eq.
(15) can be expanded by Bayes Rule:
[Equation]
where we omit the value of Y i , Y j in the last equation for simplicity.
Let M (1)4 × 2 = P ( ¯ Y
j
| Y i , Y j ) ,
M
[Equation]
2)4 × 2 = P ( ¯ Y i
| Y
i , Y j ) , M (3)4 ×
2 = P ( ¯ Y k
| Y i , sY j ) and Λ 4 × 1 = P ( Y i , Y j ) , and the Eq.
(16) can be expressed as L pqm
[Equation]
Let { A 0 , B 0 , C 0 , Λ 0 } be a solution of { M (1) , M (2) , M (3) , Λ } , which means it fulﬁls the likelihood equations (Eq.
(16)), i.e.,
[Equation] [Equation]
Note that as stated in Lemma.
6, the ﬁrst two rows and last two rows of B 0 are identical respectively.
Next, we will show that a different solution can be constructed by simply switching the corresponding rows in A 0 , C 0 and Λ 0 , which is consistent with the result in [20, 42] when Kr( M (2) )
= 1 .
By letting
A
[Equation]
=
[Equation]
and Λ 1 = 
[Equation]
then the Eq.
(17) is equivalent to
[Equation] [Equation]
where the second equation holds because the ﬁrst two rows and last two rows of B 0 are identical respectively, i.e., B 00 p
=
B 01 p and B 02 p =
B 03 p .
Note that A 1 , C 1 and Λ 1 are consistent with the form of M (1) , M (3) and Λ respectively.
Then, according to Eq.
(18), it can be readily observed that the { A 1 , B 0 , C 1 , Λ 1 } is a new solution of { M (1) , M (2) , M (3) , Λ } , hence the uniqueness is not guaranteed under this circumstance.
As M (
[Table]
= (cid:18) T
[Table]
is not unique, then T j is unidentiﬁable.
Note that according to Deﬁnition 2, in the situation of Theorem 3, the model parameter θ :=
{ T j , P ( Y j ) , P ( ¯ Y i , ¯ Y k | Y j ) } .
For convenience, in the proof of Theorem 3, we use θ 1 :=
{ T j , P ( Y j , Y i ) , P ( ¯ Y i | Y j , Y i ) , P ( ¯ Y k | Y j , Y i ) } as the model parameter.
It is easy to prove that the above different solutions of θ 1 can lead to two different solutions of θ .
H Proof of Theorem
Theorem 4.
If P ( ¯ Y i | Y j ) is known, two noisy labels { ¯ Y j , ¯ Y i } are sufﬁcient to identify T j .
Proof.
The proof of Theorem 4 is much similar to that of Theorem 1, we can get a system of equations expressed as E =
( T j ) (cid:62) P M .
The difference lies in that in Theorem 4, the matrix M which is parameterized by P ( ¯ Y i | Y j ) is given.
Since M is invertible (Similar to Lemma. 4), the problem can be converted to a simple bilinear decomposition problem:
[Equation]
Solving the above bilinear decomposition problem, the unique solution can be obtained as:
[Equation]
Substituting Eq.
(20) into Eq.
(19), then right multiplying P − 1 on both side of the equation, the matrix T j can be derived as:
[Equation]
which indicates that T j is identiﬁable given label correlation P ( ¯ Y i | Y j ) .
I Summary of the Inspirations from the proof of Theorem 1-4.
From the proof of Theorem 1, we can know that the label correlations of two noisy labels { ¯ Y j , ¯ Y i } can not offer enough information to achieve the identiﬁability of T j .
From the proof of Theorem 2, three noisy labels { ¯ Y j , ¯ Y
i , ¯ Y k } can provide more information than two noisy labels { ¯ Y j , ¯ Y i , ¯ Y k } to achieve the identiﬁability when ¯ Y i and ¯ Y k are independent given
Y j .
Note that when satisfying the condition, the parameter P ( ¯ Y i , ¯ Y k | Y j ) is reduced to P ( ¯ Y i | Y j ) and P ( ¯ Y k | Y j ) .
From the proof of Theorem 3, due to the entangled correlations, when ¯ Y i and ¯ Y k are not independent given Y j , modelling three noisy labels { ¯ Y j ,
¯ Y i , ¯ Y k } will increase too many model parameters, making the identiﬁability decrease from the situation in Theorem 2.
From the proof of Theorem 4, by reducing the unknown model parameters via some extra information, the label correlations of two noisy labels { ¯ Y j , ¯ Y i } can achieve the identiﬁability.
The summary of Ω , θ , P θ , condition and the identiﬁability in Theorem 1-4 is shown in Tab. 8.
[Table]
J Ablation Study about Sample Selection Threshold
The sample selection we adopted [38, 22] is to estimate this clean probability of examples by modeling loss values with a GMM model using the Expectation-Maximization algorithm.
If the clean sample can be distinguished according to loss values, and its estimated probability is accurate, the best threshold will be about 0.5.
Hence, τ = 0 .
5 is a typical value in related works [38, 22], and we follow this practice in our experiments.
In this section, we conduct the ablation study about the sample selection threshold experiments on noisy VOC2007 datasets.
As shown in Fig.
1, τ = 0 .
5 is a good choice both according to mAP scores on the noisy validation set and according to mAP scores on the clean test dataset.
[Figure]
K Ablation Study about Label Correlations
According to the results in Appendix C, we divide all labels on MS-COCO dataset into two categories: one is very likely to has a correlation with class label 50 (the label belonging to this category is termed as "label A") and the other is less likely to has a correlation with class label 50 (the label belonging to this category is termed as "label B").
Tab.
9 shows the mean estimation error of transition matrices for class label 50 by using the correlations of label 50 and label A (or B) in various cases.
The results represent using the correlations with label A, the estimation error of transition matrices T 49 will be smaller, showing stronger label correlations can lead to better estimation in our approach.
[Table]
L Comparison with Different Loss Correction Ways
Many statistically consistent algorithms [28, 33, 52] consist of a two-step training procedure.
The ﬁrst step estimates the transition matrix, and the second step builds statistically consistent algorithms via modifying loss functions.
Our proposed estimator can be seamlessly embedded into their frameworks.
In this section, we compare classiﬁcation performance of applying our estimation method to four different loss correction ways: Reweight [28] (named Reweight-Ours), Backward [32, 33, 54] (named
Backward-Ours), Forward [33] (named Forward-Ours) and T-Revision [52] (named Revision-Ours) on Pascal-VOC2007 dataset.
As shown in Tab. 10, T-Revision with our estimation achieves the best performance in most cases, which may be because it can further tune the transition matrices via classiﬁcation learning.
M Comparison with Different Base Learning Algorithms
Recently, many advanced multi-label learning algorithms [58, 61, 58, 5], which designed exquisite networks for multi-label learning, have been proposed, and they perform well on the clean dataset.
As their loss functions are similar to Eq.
(8), we can also apply reweight method with our estimator to their frameworks.
In this section, we compare the classiﬁcation performance of applying Reweight method with our estimator to three different base learning algorithms: Standard (named "Reweight-Ours"),
AGCN [58] (named "AGCN-R-Ours") and CSRA [61] (named "CSRA-R-Ours") on Pascal-VOC dataset.
Besides, to show the importance of the consistent algorithms, we also present the performance of base learning algorithms here.
As shown in Tab. 11, the consistent algorithms with our estimator can help base learning algorithms perform much better, especially on the OF1 and CF1 metrics.
[Table] [Table]
Besides, AGCN-R-Ours and CSRA-R-Ours perform better than Reweight-Ours in some cases, which means that the consistent algorithms with our estimator can work well with more advanced network.
N Discussion about Relaxation of Instance-independent Assumption
In this work, we assume that label noise is class-dependent but instance-independent.
While, in the real-world scenarios, label noise is instance-dependent.
Actually, this instance-independent assumption can be roughly relaxed to the assumption that the label noise of one class label is dependent on the label correlations with a few classes, and independent on the label correlations with most classes, which means most label pairs ( i, j ) s meet P (cid:0)
¯ Y j | Y j , Y i (cid:1) =
P (cid:0) ¯ Y j | Y j (cid:1) , P (cid:0)
¯ Y i | Y j , Y i (cid:1) =
P (cid:0) ¯ Y i | Y i (cid:1) .
With such labels, the system of equations involving T j in Section 3.2 holds, and our approach also works well.
This relaxed assumption can be nearly satisﬁed in many real-world scenes, because generally speaking, the multi-label label noises for class j are usually dependent on confusing features for itself, and the majority of classes will not share the same confusing features with class j .
This claim agrees with the discussion about signiﬁcantly different label pairs in Appendix D, and some research works about real-world label noise [43, 41] also show that noisy labels usually ﬂips to some similar class labels in the real-world scene.
For example, In CIFAR-100N, which is a re-annotated version of the CIFAR-100 with real-world human annotations, most classes are more likely to be mislabeled into less than four ﬁne classes [43].
In ANIMAL-10N, the label noise mainly happens between ﬁve pairs of confusing animals [41].
Besides, the experiments in Appendix O also verify the effectiveness of our approach in two typical instance-dependent multi-label noise cases.
O Experiments on Instance-dependent Label Noise
We perform the experiments with two types of instance-dependent label noise: pair-wise label noise [13] and PMD label noise [59] on Pascal-VOC2007 and MS-COCO datasets to illustrate the applicability in realistic scenarios.
For pair-wise label noise, one class label is mistaken as another class label with a certain probability.
For PMD label noise, data near the decision boundary are harder to distinguish and more likely to be mislabeled.
Both of them are much realistic.
The estimation error between the estimated transition matrices and P (cid:0) ¯ Y j | Y j (cid:1) can be seen in Tab.
12 and 13, and comparison for classiﬁcation performance can be seen in Tab. 14 and 15.
Note that as class labels in both datasets are unbalanced, in order to prevent class change, we only test with pair-wise label noise less than 20%, and do not ﬂip the labels into classes with few positive examples.
The results show the proposed estimator can achieve the smaller estimation errors on such instance-dependent cases, leading to better classiﬁcation performance.
[Table] [Table]
[Table]
P The Wilcoxon Signed-Ranks Test for Reweight-Ours against Baselines
Wilcoxon signed-ranks test [10] is employed to show whether Reweight-Ours has a signiﬁcant performance than other comparing approaches.
Note that the performance of these methods used for the Wilcoxon signed-ranks test is from both class-dependent label noise cases and instance-dependent label-noise cases.
As shown in Tab. 16, Reweight-Ours outperforms other baselines on both OF1 and
CF1 metrics at 0.1 signiﬁcance level.
[Table] [Table]
Q Broader Impact
In this era of deep learning, datasets are becoming bigger and bigger.
However, it is much expensive to obtain large-scale dataset with high-quality annotated labels.
Alternatively, people can collect labels based on non-expert annotators or automated labeling methods, e
[Table]
g. web search and user tags, especially for non-proﬁt research institutions and startups.
However, these cheap annotation methods are likely to introduce noisy labels to the collected datasets, which may pose serious threats to the supervised learning algorithm designed for clean data.
Label-noise learning, therefore, becomes a more and more important topic recently.
Existing label-noise learning methods typically focus on the single-label case by assuming that only one label is corrupted.
In real applications, an example is usually associated with multiple class labels, which could be corrupted simultaneously with their respective different probabilities.
The proposed method is to estimate the transition matrix using label correlations in noisy multi-label datasets.
The transition matrix is essential to building the statistically consistent label-noise learning algorithms.
We have shown that our method usually leads to a better estimation compared to the current estimator and can improve the classiﬁcation performance of statistically consistent label-noise learning applications, which should has a positive impact on science, society, and economy.
Hence, our approach can reduce the need for accurate labeling, and potentially, it may have a negative impact on the salary of label workers.