Estimating Noise Transition Matrix with Label Correlations for Noisy Multi-Label Learning
Abstract
In label-noise learning, the noise transition matrix , bridging the class posterior for noisy and clean data, has been widely exploited to learn statistically consistent classiﬁers.
The effectiveness of these algorithms relies heavily on estimating the transition matrix.
Recently, the problem of label-noise learning in multi-label classiﬁcation has received increasing attention, and these consistent algorithms can be applied in multi-label cases.
However, the estimation of transition matrices in noisy multi-label learning has not been studied and remains challenging, since most of the existing estimators in noisy multi-class learning depend on the existence of anchor points and the accurate ﬁtting of noisy class posterior.
To address this problem, in this paper, we ﬁrst study the identiﬁability problem of the class-dependent transition matrix in noisy multi-label learning, and then inspired by the identiﬁability results, we propose a new estimator by exploiting label correlations without neither anchor points nor accurate ﬁtting of noisy class posterior.
Speciﬁcally, we estimate the occurrence probability of two noisy labels to get noisy label correlations.
Then, we perform sample selection to further extract information that implies clean label correlations, which is used to estimate the occurrence probability of one noisy label when a certain clean label appears.
By utilizing the mismatch of label correlations implied in these occurrence probabilities, the transition matrix is identiﬁable , and can then be acquired by solving a simple bilinear decomposition problem.
Empirical results demonstrate the effectiveness of our estimator to estimate the transition matrix with label correlations, leading to better classiﬁcation performance.
Source codes are available at https://github.com/tmllab/multi-label-noise-learning.
Introduction
In real-world scenarios, an instance is naturally associated with multiple labels, and these labels have complex entangled correlations [7].
Recently, the problem of label-noise learning in multi-label classiﬁcation has received more and more attention [29, 34, 54, 50, 46, 47], since it is time-consuming and expensive to collect large-scale accurate labels and the noisy labels are much cheaper and easier to acquire.
In the setting of noisy multi-label learning , the multiple labels assigned to an instance may be corrupted simultaneously.
That is to say, any label for each class can be ﬂipped with its respective transition matrix that denotes the transition relationship from clean labels to noisy labels.
Transition matrix has been utilized to build a series of statistically consistent algorithms for noisy multi-class learning [32, 52, 48, 8].
The main advantage of these consistent algorithms is that they can guarantee to vanish the differences between the classiﬁers learned from noisy data and the optimal ones from clean data by increasing the size of noisy examples [28, 33, 52, 39].
Fortunately, these statistically consistent algorithms for noisy multi-class learning can also be applied in such noisy multi-label learning with a little modiﬁcation [54] (more details can be found in
Appendix B).
However, the effectiveness of these algorithms heavily relies on estimating the transition matrix.
Although the estimation of the transition matrix has been investigated in noisy multi-class learning, the estimation of the transition matrix in noisy multi-label learning has not been studied and remains challenging.
Speciﬁcally, a series of methods [28, 33, 57, 52, 26] has been proposed to estimate the transition matrix for noisy multi-class learning.
Most of them assume the existence of anchor points [28, 33, 57] that are deﬁned as the training examples belonging to a particular clean class surely.
Nevertheless, the assumption is strong and hard to check when we only have noisy data [52].
Also, the methods need to accurately ﬁt the noisy or intermediate class posterior of anchor points, which are rather difﬁcult in multi-label cases, due to severe positive-negative imbalance [36].
In this paper, to address the problem of estimating the transition matrix in noisy multi-label learning, we consider utilizing label correlations among noisy multiple labels.
Speciﬁcally, some label correlations that should not exist in practice are included in noisy multi-label learning.
For example, “ﬁsh” and “water” always co-occur, while “bird” and “sky” always co-occur.
But, due to label errors, there is a slight correlation between “ﬁsh” and “sky”, which is impractical.
At a high level, we can utilize the mismatch of label correlations to identify the transition matrix without neither anchor points nor accurate ﬁtting of noisy class posterior.
In more detail, we ﬁrst focus on the identiﬁability problem of the class-dependent transition matrix in noisy multi-label learning.
Accordingly, a new method that estimates the transition matrix by exploiting label correlations is proposed.
That is, motivated by the identiﬁability result that the label correlation of two noisy labels can not sufﬁce to identify the transition matrix in noisy multi-label learning, we utilize sample selection to extract useful information from noisy data, which implies clean label correlations to achieve the identiﬁability.
Afterward, we not only estimate the occurrence probability of two noisy labels in noisy data, but also of one noisy label when a certain clean label appears in selected data.
By utilizing the mismatch of label correlations implied in these occurrence probabilities, we can prove the identiﬁability, and transform the problem of estimating the transition matrix using label correlations into a problem of bilinear decomposition .
Finally, with easy frequency counting, we can get a good estimation of the noise transition matrix.
Empirical results illustrate the effectiveness of the proposed estimator for estimating the transition matrix in noisy multi-label learning, and the consistent algorithms with our estimator can achieve better classiﬁcation performance.
The rest of the paper is organized as follows.
In Section 2, we brieﬂy present the problem setting of label-noise learning in multi-label classiﬁcation.
In Section 3, we discuss the identiﬁability of the transition matrix under such a noisy multi-label setting, and introduce our estimation method.
Experimental results are provided in Section 4.
The limitations of this work are discussed in Section 5.
Finally, we conclude the paper in Section 6.
Problem Setting
In this section, we introduce the problem setting of label-noise learning in multi-label classiﬁcation.
In what follows, scalars are in lowercase letters, vectors are in lowercase boldface letters, and matrices/variables are in uppercase letters.
For simplicity, let [ q ] = { 1 , . . .
, q } .
Preliminaries.
Let D be the distribution of a pair of random variables ( X , Y ) , where X ∈ X
⊆
R
d denotes the variable of instances, and Y =
{ Y 1 , Y 2 , ..., Y q } ∈ { 0 , 1 } q denotes the variable of targets with q possible class labels.
As for Y , Y j = 1 indicates that the instance X is associated with the class j ;
Y j = 0 , otherwise.
In multi-label learning, the goal is to learn a function from D which maps each unseen instance x ∈ X to proper labels y .
However, as discussed, Y is hard to be annotated precisely.
Before being observed, their true labels are independently ﬂipped and what we can obtain are noisy training examples
D t
= { ( x i , ¯ y i ) } n i =1 , where ¯ y i denotes noisy labels.
Let ¯ D be the distribution of the noisy random variables ( X ,
¯ Y ) ∈
X × { 0 , 1 } q .
In noisy multi-label learning, our goal is to infer proper labels for each unseen instance by only using the noisy training examples.
Noise transition matrix.
The random variables ¯ Y j and Y j for the class j are related through a noise transition matrix T j ∈ [0 , 1] 2 × 2 , j ∈ [ q ] .
Generally, the transition matrix depends on instances, i.e., T jik ( x ) =
P ( ¯ Y j = k | Y j
=
i, X = x ) .
Nevertheless, given only noisy examples, the instance-dependent transition matrix is non-identiﬁable without any additional assumption [52, 56].
For example, both P ( ¯ Y j = k | X = x ) = (cid:80) 1 i =
T jik P
( Y j = i
| X = x ) and P ( ¯ Y j = k
| X = x ) = (cid:80) 1 i =
T (cid:48) jik P (cid:48)
( Y j = i | X = x ) are valid, when T (cid:48)
jik ( X = x ) =
T jik ( X = x ) P ( Y j = i
| X = x ) /P
(cid:48)
( ¯ Y j = i
| X = x ) .
Therefore, in this paper, we assume that the transition matrix is classdependent and instance-independent [33], i.e., P ( ¯ Y j = k | Y j
= i, X = x ) =
P ( ¯ Y j = k | Y j = i )
=
T jik .
The deﬁnition of the class-dependent label noise can be found in Appendix A, where we further discuss its differences with the class-dependent label noise in multi-class cases.
Consistent algorithms.
The transition matrix bridges the class posterior probabilities for noisy and clean data, i.e., P ( ¯ Y = k
| X = x ) = (cid:80) 1 i =
T ik P ( Y = i
| X = x ) .
Thus, it has been exploited to achieve many statistically consistent algorithms in noisy multi-class learning.
Speciﬁcally, it has been utilized to build risk-consistent estimators via correcting loss functions [28, 33, 52], and to design classiﬁer-consistent estimators via limiting hypotheses, e.g., [33, 9, 62].
Since the multi-label task can be decomposed into multiple conditionally independent binary classiﬁcation problems, we also can apply these consistent methods in noisy multi-label learning [54].
In this paper, without loss of generality, we adopt a risk-consistent algorithm, i.e., Reweight [28, 52], to learn statistically consistent classiﬁers with estimated transition matrices.
More details can be found in Appendix B.
Transition matrix estimation.
As inaccurate transition matrices will degenerate the performances of these consistent algorithms, a series of estimation methods [28, 51, 57, 63, 26] have been proposed for noisy multi-class learning to efﬁciently identify the transition matrix.
However, most of them require the assumption of anchor points [33, 57, 26], which is strong and hard to check in multi-label cases when only noisy data are provided [52].
Besides, severe positive-negative imbalance in multi-label learning [36] will make it difﬁcult to accurately approximate the noisy or intermediate class posterior of anchor points, which is crucial for these methods.
This motivates us to seek for a better estimator that can do without anchor points and avoid estimating noisy posterior in noisy multi-label learning.
Estimating Transition Matrices using Label Correlations
In this section, we ﬁrst study the identiﬁability problem [30] of class-dependent transition matrices in multi-label cases.
Furthermore, inspired by these results, we propose a new estimator to approximate the transition matrix by utilizing label correlations.
It is worth pointing out that our estimator demands neither the existence of anchor points nor accurate ﬁtting of noisy class posterior.
3.1 Identiﬁability of transition matrix
Recently, Liu et al.
[30] built identiﬁability of the noise transition matrix on the Kruskal’s identiﬁability results.
Inspired by them, with the complex correlations among class labels, we can get some identiﬁability results of the class-dependent transition matrix in noisy multi-label learning.
Following [30], to deﬁne identiﬁability, we denote an observation space by Ω .
For a general parametric space Θ , denote the distribution (probability density function) induced by the parameter θ ∈ Θ on the observation space Ω as P θ [2].
The identiﬁability for a general parametric space is deﬁned as follows:
Deﬁnition 1 (Identiﬁability [30]) .
The parameter θ is identiﬁable if P θ (cid:54)
=
P θ (cid:48) , ∀ θ (cid:54)
= θ (cid:48) .
For the class-dependent transition matrix T j for class j , the identiﬁability can be deﬁned as the following, when T j is a part of θ .
Deﬁnition 2 (Identiﬁability of T j ) .
T j is identiﬁable if P θ (cid:54) =
P θ (cid:48) for θ (cid:54)
= θ (cid:48) , up to the label permutation of class j .
Here, the label permutation of class j means swapping 1 and 0 class values in class j , and the rows of T j will also swap.
Note that Ω does not necessarily include all observable variables.
We use an example to better understand it.
For example, let Ω :=
{ ¯ Y j , X } , θ := { T j , P ( Y j | X ) } , and P θ :=
P ( ¯ Y j | X ) , the identiﬁability of T j can be achieved with the anchor point assumption [33, 57, 26].
To use notations without confusion, for the identiﬁability of T j using label correlations, we let Ω := { ¯ Y j , ¯ Y − j } , where − j means other classes having correlations with class j , θ :=
{ T j , P ( Y j ) , P ( ¯ Y − j | Y j ) } , and P θ := P ( ¯ Y j , ¯ Y − j ) .
Also, Ω does not necessarily include all noisy class labels.
We need Ω to provide useful information to achieve the identiﬁability.
The exploration of an effective Ω and necessary conditions is one of the focuses of the paper.
Assumption 1.
P ( ¯ Y j = 0 | Y j = 1) + P ( ¯ Y j = 1 | Y j = 0) < 1 , j ∈ [ q ] .
Assumption 1 means that the noisy label is agreed with the clean label on average, which is a standard condition for analysis under the class-dependent transition matrix [32, 31].
Assumption 2.
P ( Y i =
| Y j = 0) (cid:54) =
P ( Y i =
| Y j = 1) , i, j ∈ [ q ] and i (cid:54) =
j .
Assumption 2 means that the multiple labels have correlations between each other, which is satisﬁed by the most of ( i, j ) pairs in the real-world dataset (see Appendix C).
When considering multi-label learning, the simplest case is having two class labels.
In this case, the following theoretical results can be obtained.
Theorem 1.
Two noisy labels { ¯ Y j , ¯ Y i } will not sufﬁce to identify T j .
This result tells us that the label correlations of two noisy labels can not offer enough information to achieve the identiﬁability of T j .
We provide Theorem 2 based on the Kruskal’s identiﬁability result [21, 40].
Theorem 2.
If ¯ Y i and ¯ Y k are independent given Y j , three noisy labels { ¯ Y j ,
¯ Y i , ¯ Y k } are sufﬁcient to identify T j .
The assumption that ¯ Y i and ¯ Y k are independent given Y j can be satisﬁed in certain cases, e.g., the occurrences of “blue” and “dolphin” may be independent given “sea” appearing or not.
Nevertheless, due to the complex correlations among labels, this assumption is hard to hold in most cases.
When the assumption can not hold, these label correlations are no more sufﬁcient to determine T j , as shown in Theorem 3 in the following.
Theorem 3.
If ¯ Y i and ¯ Y k are not independent given Y j , three noisy labels { ¯ Y j ,
¯ Y
i , ¯ Y k } will not sufﬁce to identify T j .
The inspiration from Theorem 3 is that, the increase of the number of noisy labels may make the identiﬁability decrease due to the entangled correlations (see Appendix G and I).
To handle this problem, we prove Theorem 4 by assuming the transition relationship between noisy label for class i and clean label for class j is known.
Theorem 4.
If P ( ¯ Y i | Y j ) is known, two noisy labels { ¯ Y j , ¯ Y i } are sufﬁcient to identify T j .
Theorem 4 theoretically guarantees that the identiﬁability of the class-dependent transition matrix can be achieved by utilizing the occurrence probabilities P ( ¯ Y i , ¯ Y j ) and P ( ¯ Y i | Y j ) .
Note that
P ( ¯ Y i , ¯ Y j ) can represent noisy label correlations, and P ( ¯ Y i | Y j )
=
P ( ¯ Y i
| Y i )
P
( Y i | Y j ) , which can imply clean label correlations.
At a high level, the mismatch of label correlations implied in the occurrence probabilities can achieve the identiﬁability.
The detailed proof of Theorem 1-4 is provided in Appendix E-H.
3.2 Our estimator
Our estimator is based on Theorem 4, which needs extra information to estimate P ( ¯ Y i | Y j ) .
Recently, the memorization effect [4] of deep networks has received much attention in learning with noisy labels, which shows that deep networks will ﬁrst memorize the training data with clean labels and then those with noisy labels.
Prior works utilize this characteristic to develop sample selection methods [16, 13, 22, 17, 25], where we select some examples with more likely clean labels for each class j respectively in the early learning phase.
The selected examples can serve as the useful extra information that implies clean label correlations, through which we can achieve estimation via counting.
However, when implementing sample-selection-based methods, a major concern is whether the sampling bias will lead to large estimation errors.
Generally speaking, according to the memorization effect , the sampling bias is about selecting easy examples for class j , which usually means these examples have easy-to-discriminate features.
We can reasonably assume that given Y j , the distribution of the features about class j is biased, while the distribution of the features about another class i is unbiased, i.e.,
[Equation]
(1) where D js is the distribution of ( X , Y s ) ,
Y s = { ¯ Y 1 , ¯ Y 2 , ..., Y j , ..., ¯ Y q } , ¯ D js is the biased distribution of ( X , Y s ) , and X i is the part of X , which represents all information about class i appearing or not.
When the assumption is satisﬁed, the sample selection will not lead to large estimation errors on
P ( ¯ Y i | Y j ) , and it can converge to zero exponentially fast by counting [6].
In real-world scenarios, due to the complex label correlations, this assumption will not strictly hold.
While, it may be roughly met when class labels i and j do not share the major discriminative features.
Intuitively speaking, the classifying of simplest examples for one label is not easily affected by the presence or absence of other signiﬁcantly different labels.
Also, since most label pairs from typical real-world multi-label datasets are signiﬁcantly different (see Appendix D), the assumption can be roughly hold for those label pairs in typical cases.
In Section 4.1, our empirical results justify this by showing a little gap between the estimation error of our estimator with a biased sample selection and an unbiased one.
Based on the above discussions, we can approximate P ( ¯ Y i , ¯ Y j ) and P ( ¯ Y i | Y j ) with frequency counting, and utilize the mismatch of label correlations implied in these occurrence probabilities to estimate the transition matrix.
In this work, we propose to estimate transition matrices { T j } qj =1 by following two stages.
In the ﬁrst stage, we utilize sample selection to obtain useful information that implies clean label correlations.
Speciﬁcally, we train a classiﬁer with the standard multi-label classiﬁcation loss on noisy training examples D t for a few epochs, and then perform sample selection to get a selected clean set D js for each class label j .
Specially, we use a commonly used sample selection way [3, 22] in learning with noisy labels, which extracts the subset of examples with small losses by modeling the distribution of losses for class j with a Gaussian mixture model (GMM).
In the second stage, we perform co-occurrence estimation by frequency counting, and then estimate the transition matrix by solving a simple bilinear decomposition problem.
For class label j , we ﬁrst choose another class label i and estimate P ( ¯ Y i , ¯ Y j ) and P ( ¯ Y i | Y j ) via counting, i.e., ˆ P
[Equation]
where I [ · ] is the indicator function which takes 1 if the identity index is true and 0 otherwise.
Then, these co-occurrence probabilities, which imply the mismatch of label correlations, can lead to four equations involving T j :
P
[Equation]
¯ Y j = 0 , ¯ Y i = 0 (cid:1) =
P ( Y j = 0)
T j 00 P ( ¯ Y i =
| Y j = 0)
+ P ( Y j = 1) T j 10 P ( ¯ Y i =
| Y j = 1) ,
P (cid:0) ¯ Y j = 0 , ¯ Y i = 1 (cid:1) =
P ( Y j = 0)
T j 00 P ( ¯ Y i =
| Y j = 0)
+ P ( Y j = 1) T j 10 P ( ¯ Y i = 1 | Y j = 1) ,
P (cid:0) ¯ Y j = 1 , ¯ Y i = 0 (cid:1) =
P ( Y j = 0) T j 01 P ( ¯ Y i =
| Y j = 0)
+ P ( Y j = 1) T j 11 P ( ¯ Y i =
| Y j = 1) ,
P (cid:0) ¯ Y j = 1 , ¯ Y i = 1 (cid:1) =
P ( Y j = 0) T j 01 P ( ¯ Y i =
| Y j = 0)
+ P ( Y j = 1) T j 11 P ( ¯ Y i =
| Y j = 1) .
For simplicity, we denote
E
[Equation]
( ¯ Y j = 0 , ¯ Y i = 1)
P ( ¯ Y j = 1 , ¯ Y i = 0) P ( ¯ Y j = 1 , ¯ Y i = 1) (cid:19) =
(cid:18) ee 01 ee(cid:19) ,
P = (cid:18) P ( Y j = 0)
P ( Y j = 1) (cid:19) =
(cid:18) 1 − p p (cid:19) ,
T j = (cid:18) P ( ¯ Y j =
| Y j = 0) P ( ¯ Y j = 1 | Y j = 0)
P ( ¯ Y j = 0 | Y j = 1) P ( ¯ Y j = 1 | Y j = 1) (cid:19) =
(cid:18) 1 − ρ − ρ − ρ + 1 − ρ + (cid:19) , and
M = (cid:18) P ( ¯ Y i =
| Y j = 0) P ( ¯ Y i =
| Y j = 0)
P ( ¯ Y i =
| Y j = 1) P ( ¯ Y i =
| Y j = 1) (cid:19) =
(cid:18) 1 − ρ (cid:48)− ρ (cid:48)− ρ (cid:48) + 1 − ρ (cid:48)
+ (cid:19) .
Then the system of equations can be expressed as E =
( T j ) (cid:62) P M , i.e., (cid:18) ee 01 ee(cid:19)
= (cid:18) 1 − ρ − ρ − ρ + 1 − ρ + (cid:19) (cid:62) (cid:18) 1 − p p (cid:19) (cid:18) 1 − ρ (cid:48) − ρ (cid:48)
− ρ (cid:48)
+ 1 − ρ (cid:48)
+ (cid:19) .
Denote the estimation of E , P , T j , and M as ˆ E = (cid:18) ˆ eˆ eˆ eˆ e
[Equation]
As ˆ E and ˆ M can be derived from Eq.
(2) and Eq.
(3), the problem is hence equivalent to a bilinear decomposition problem:
[Equation]
By solving the above matrix equation, we can get
[Equation]
and the estimation of the transition matrix
[Equation]
Implementation of our estimator.
The pseudo code of our estimator is described in Algorithm 1.
A little difference from the above is that in order to make better use of correlations among labels, we perform R times co-occurrence estimation and bilinear decomposition for different classes i in the second stage to get R estimations, ˆ T j r , r = 1 , 2 , ..., R .
Finally, we estimate the transition matrix T j by Eq.
[Equation]
where (cid:107) · (cid:107) 1 denotes (cid:96) 1 norm.
Require:
Noisy training examples D t , the number of classes q , the early warmup training epoch
E warm , the threshold of sample selection τ , and repeated estimation times R .
, ..., q do 7: for r = 1 , 2 , ..., R do 8:
Choose another class label i .
Estimate P ( ¯ Y i , ¯ Y j ) by ˆ P ( ¯ Y
i , ¯ Y j ) with Eq.
(2) on D t , and P ( ¯ Y i | Y j ) by ˆ P ( ¯ Y i | Y j ) with Eq.
(3) on D js .
Solve a bilinear decomposition problem (Eq. (4)) to get a estimation ˆ T
j r by Eq.
11: end for 12:
Estimate T j by ˆ T j which has the minimum error with Eq.
(7) from R estimations.
13: end for
Ensure: The estimated transition matrices { ˆ T j } qj =1 .
Experiments
Dataset We verify the effectiveness of the proposed method on three synthetic noisy multilabel datasets, i.e., Pascal-VOC2007 [11], Pascal-VOC2012 [12], and MS-COCO [27].
PascalVOC2007 [11] and Pascal-VOC2012 [12] datasets are two popular datasets for object recognition.
They both contain images from the same 20 object classes, with an average of n a = 1 .
5 labels per image.
Pascal-VOC2007 contains a training set of 5,011 images and a test set of 4,952 images.
Pascal-VOC2012 consists of 11,540 images as the training set and 10,991 images as the test set [5].
As the labels of the test set in Pascal-VOC2012 are not publicly available, we use the test set in
Pascal-VOC2007 for Pascal-VOC2012 evaluation.
MS-COCO [27] is a widely used multi-label dataset.
It contains 82,081 images as the training set and 40,137 images as the test set and coversobject classes with an average of n a = 2 .
9 labels per image.
For these datasets, we corrupted the training sets manually according to true transition matrices { T j } qj =1 .
For convenience, we use the same true transition matrices for all classes, i.e., T j = T = (cid:18) 1 − ρ − ρ
[Equation]
1 − ρ + (cid:19) , but do not divulge this information for algorithms.
We generate four different types of synthetic datasets by using different transition matrices: 1) ρ − = 0 , ρ + = ρ , which annotates some positive examples as negative examples, also known as multi-label learning with missing labels [45, 44]; 2) ρ − = ρ, ρ + = 0 , which annotates some negative examples as positive examples, also known as partial multi-label learning [53, 55]; 3) ρ − = ρ, ρ + = ρ , where positive examples and negative examples are mislabeled with the same probability ρ ; 4) ρ − = n a q − n a ρ, ρ + = ρ , where positive examples and negative examples are mislabeled with the same number.
In the experiments, we test the algorithms on various ρ .
For all datasets, we leave out 10% of the noisy training examples as a noisy validation set.
We use mAP on noisy validation set as the criterion for model selection.
Implementation details For a fair comparison, we implement all methods with default parameters by PyTorch on NVIDIA RTX 3090.
We use a ResNet-50 network [14] pre-trained on ImageNet [37] for all datasets, and the optimizer is Adam optimizer [18] with β = (0 . 9 , 0 . 999) .
The batch size is 128, the learning rate is 5e-5.
The number of training epochs is 20 for Pascal-VOC2007/VOC2012, and 30 for MS-COCO.
For the transition matrix estimation method, E warm is the same as the normal training epoch.
For our estimator, we perform sample selection based on the average losses ofepochs before a certain warmup epoch (10th epoch for Pascal-VOC2007/VOC2012, 15th epoch for MS-COCO), R = q − 1 and τ = 0 .
5 in all experiments.
All experiments are run at least three times with different random seeds, and we report the average and standard deviation values of results. The
4.1 Comparison for estimating transition matrices
Baselines For evaluating the effectiveness of estimating the transition matrix under multi-label cases, we compare the proposed method with the following methods: (1) T-estimator max [28, 33], which estimates the transition matrix via the noisy class posterior probabilities for anchor points that have the largest estimated noisy class posteriors.
(2) T-estimator 97% [28, 33], which selects the points with 97% largest estimated noisy class posteriors to be anchor points.
(3) Dual T-estimator max [57], which introduces an intermediate class to avoid directly estimating the noisy class posterior, and selects the points with the largest estimated intermediate class posteriors to be the anchor points.
Dual T-estimator 97% [57], which selects the points with the 97% largest estimated intermediate class posteriors to be the anchor points.
Metrics We use the sum of estimation error for the transition matrices as the estimation evaluation metric, i.e.,
[Equation]
Results In Tab. 1, 2 and 3, we can see that for all cases on three datasets, the proposed estimation method leads to the smallest or second-smallest estimator errors across various noise rates.
Note that since the ﬁtting of noisy or intermediate class posterior is hard to be accurate in noisy multilabel learning, T-estimator and Dual T-estimator need to carefully tune a hyperparameter for better estimation under different noise rates, and it’s very sensitive in some cases, e.g., MS-COCO datasets with noise rates (0 . 1 , 0 . 1) .
In contrast, our method uses the same hyperparameters on one dataset to get good results for all cases, which reﬂects its robustness to various noise rates.
Besides, to study the ablation of sampling bias, we also run our method with an unbiased sample selection, named “our estimator gold”.
We can see that sample bias is the main factor that contributes to the error for our estimator, but the little error gap between our estimator and our estimator gold shows it will not lead to large estimation errors.
[Table]
[Table]
on MS-COCO dataset.
[Table]
4.2 Comparison for classiﬁcation performance Baselines
We exploit 10 baselines: (1) Standard, which trains with a standard multi-label classiﬁcation loss.
(2) GCE [60], which proposes a Generalized Cross-Entropy loss for robustness.
CDR [49], which performs different update rules for two types of parameters to achieve robust learning.
(4) AGCN [58], which adopts a dynamic GCN to model the relation of content-aware class representations.
(5) CSRA [61], which generates class-speciﬁc features for every category by proposing a spatial attention score.
(6) WSIC [15] that proposes to use a small set with clean labels to learn a residual net for regularization in noisy multi-label learning, and we only provide noisy datasets to it for a fair comparison.
(7) Reweight-T max, which learns with a reweighting algorithm using transition matrices estimated by T-estimator max [33].
(8) Reweight-T 97%, which learns with a reweighting algorithm using transition matrices estimated by T-estimator 97% [33].
Reweight-DualT max, which learns with a reweighting algorithm using transition matrices estimated by Dual T-estimator max [57].
(10) Reweight-DualT 97%, which learns with a reweighting algorithm using transition matrices estimated by Dual T-estimator 97% [57].
Note that Standard, AGCN, and
CSRA are designed for clean multi-label data, and GCE and CDR are designed for noisy multi-class learning.
Metrics Following conventional setting [11, 7, 36, 35], we compute the mean average precision (mAP), overall F1-measure (OF1) and per-class F1-measure (CF1) as classiﬁcation evaluation metrics.
For each image, we assign a positive label if its prediction probability is greater than 0.5.
Results As shown in Tab. 4, 5 and 6, ﬁrst, we can ﬁnd those statistically consistent methods achieve the best or second-best results on all three metrics in the vast majority of cases, while other methods can only achieve good results in some cases.
For example, on the Pascal-VOC2012 dataset with noise rates (0 . 0 , 0 . 6) , CSRA achieves the best result in the mAP metric with the help of its well-designed network, but its performance is far lower than those consistent methods on the OF1 and CF1 metrics, which shows the learned model can not approximate well the true class posterior P ( Y | X ) .
Note that since network structure and loss correction are compatible, the risk-consistent methods can also help AGCN and CSRA perform better (shown in Appendix M).
Second, theoretically, the more accurate the transition matrix is estimated, the more likely the consistent method is to achieve better results by increasing the size of noisy examples, and our experimental results verify this.
Among those consistent methods, the reweighting algorithm with our estimator (named "Reweight-Ours") obtains the best or second-best results on the three metrics, which is due to the smaller error of our estimation.
Especially on the challenging and large-scale MS-COCO dataset, Reweight-Ours outperforms almost all state-of-the-art methods on the CF1 metric and signiﬁcantly surpasses other baselines by a large margin in some cases.
For example, on the MS-COCO dataset with noise rates (0 . 6 , 0) , Reweight-Ours achieves the best CF1 result (58.63 ± 1.30), while the suboptimal result is 56.70 ± 2.11, and the result of Standard is only 7.07 ± 0.02.
In addition, the ablation studies about loss correction ways and base learning algorithms are provided in Appendix L and M, which shows that our estimator can achieve much better performance with the advanced frameworks.
Besides, although our method is based on the assumption of class-dependent label noise, the experiments with two types of instance-dependent label noise are provided in Appendix O.
[Table]
[Table] [Table]
Limitations
This work still has certain limitations, including: 1) This work exploits the memorization effect [4] in deep learning to perform sample selection, while the memorization effect has not been found in other traditional machine learning methods, and therefore, the proposed estimator can not be applied to such learning methods.
This work estimates occurrence probabilities using frequency counting.
Although this estimation error will converge to zero exponentially fast [6], when the number of one label appearing is too small, e.g., less than 50, the estimation of the transition matrix for this class label is still difﬁcult to be accurate.
3) Since our work assumes label noise is class-dependent but instance-independent, when this assumption does not hold, the estimation is not guaranteed.
The discussions about the relaxation of instance-independent assumption can be found in Appendix N, which reveals its applicability in certain typical instance-dependent cases.
Conclusion
In this paper, we study the estimation problem of the transition matrices in the noisy multi-label setting.
We prove some identiﬁability results of class-dependent transition matrices in such a setting, inspired by which we propose a new estimator to approximate the transition matrix.
The proposed estimator utilizes the information of label correlations, and demands neither anchor points nor accurate ﬁtting of noisy class posterior.
Experiments on three popular multi-label datasets illustrate the effectiveness of the proposed estimator to accurately estimate transition matrices, and the consistent algorithms with this estimator achieve better classiﬁcation performance.
Acknowledgements
Yibing Zhan was partially supported by the Major Science and Technology Innovation 2030 "
New
Generation Artiﬁcial Intelligence" Key Project (No. 2021ZD0111700) and the National Natural
Science Foundation of China (No. 62002090).
Shiming Ge was partially supported by grants from the
Beijing Natural Science Foundation (L192040), and National Natural Science Foundation of China (
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or [N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section 2.
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Appendix Q. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 3.1.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix E-H. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We provide the source codes at https://github.com/tmllab/multi-label-noise-learning.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section 4.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] We report the standard deviation value of results.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Section 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 4.
(b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]