A N I NFORMATION F USION A PPROACH TO L EARNING WITH I NSTANCE -D EPENDENT L ABEL N OISE
Anonymous authors
Paper under double-blind review
A BSTRACT
Instance-dependent label noise (IDN) widely exists in real-world datasets and usually misleads the training of deep neural networks.
Noise transition matrix (NTM) (i.e., the probability that clean labels ﬂip into noisy labels) is used to characterize the label noise and can be adopted to bridge the gap between clean and noisy underlying data distribution.
However, most of instances are long-tail, i.e., the number of appearance for each instance is usually limited, which leads to the gap between underlying distribution and empirical distribution.
Therefore, the genuine problem caused by IDN is empirical , instead of underlying, data distribution mismatch during training.
To directly tackle empirical distribution mismatch problem, we propose posterior transition matrix (PTM) to posteriorly model label noise given limited observed noisy labels achieving statistically consistent classiﬁers .
Note that even if the instance is corrupted by the same NTM, the intrinsic randomness incurs to different noisy labels, and thus requires different correction methods.
Motivated by this observation, we propose an I nformation
F usion (IF) approach to ﬁne-tune the NTM based on estimated PTM.
Speciﬁcally, we adopt the noisy labels and model predicted probability to estimate PTM and then correct the NTM in forward propagation .
Empirical evaluations on synthetic and real-world datasets demonstrate that our method is superior to the state-of-the-art approaches, and achieves more stable training for instance-dependent label noise.
I NTRODUCTION
Data labeling annotated from human efforts, such as crowdsourcing (Yan et al., 2014; Chen et al., 2017) and online queries (Divvala et al., 2014), may be heavily noisy in practice.
To make it worse, the label noise stemmed from the human annotations is often instance-dependent.
For example, the images closing to the decision boundary are usually prone to be mislabeled (Zhang et al., 2021;
Zhu et al., 2021b).
On the other hand, the remarkable success of deep neural networks (DNNs) on supervised learning tasks heavily relies on the expressive power and a large number of data with accurate labels.
Unfortunately, deep neural networks memorizes noisy labels leading to poor generalization (Zhang et al., 2017).
It is challenging to learn with practical instance-dependent label noise (IDN) due to the hidden and complicated label noise properties (Zhu et al., 2021a; Liu, 2021).
The methods of dealing with noisy labels falls in two lines, including heuristically identifying noisy samples or statistical label noise modeling.
The training of deep neural network often learns the clean labels ﬁrst
(Arpit et al., 2017) and memorizes the noisy labels gradually, which is recognized as the memorization effect.
Based on the general memorization effects, the heuristical methods are all designed via following the anomaly detection strategy: identifying the noisy sample based on different behaviors (e.g., loss values) between clean and noisy samples during training.
The typical methods contain sample selection (Yu et al., 2019; Han et al., 2018b), reweight samples (Cheng et al., 2021; Jiang et al., 2018; Ren et al., 2018), label correction (Ma et al., 2018; Tanaka et al., 2018), and regularization (Han et al., 2018a).
Although these algorithms empirically work well, without modeling the label noise explicitly, the reliability cannot be guaranteed.
Another line of works relies on noise transition matrix (NTM) to model the label noise statistically (Xia et al., 2019; 2020; Patrini et al., 2017).
Namely, the NTM quantiﬁes the probabilities that clean labels ﬂip into noisy labels.
Although the NTM-based methods possess theoretical guarantee, under
IDN, the NTM estimation for each instance is pretty challenging.
To ease the estimation, some unrealistic assumptions have to be posed the on NTM, including instance-independent transition matrix (Liu & Guo, 2020; Wei & Liu, 2021; Li et al., 2021), symmetric transition matrix (Menon et al., 2018), upper bounded noise rate (Cheng et al., 2020), and part-dependent label noise (Xia et al., 2020).
However, under the complex IDN, the empirical noise distribution is highly different with underlying noise distribution.
For example, in Figure 1, the underlying and empirical noisy distribution for long-tail instance is different since the empirical noisy label is either the same or different with clean label.
Additionally, observed noisy label provide inductive bias toward label corruption.
In other word, the genuine problem causing from IDN is the empirical , instead of underlying, clean and noisy distribution mismatch problem.
To mitigate empirical distribution mismatch problem, we propose the posterior transition matrix (PTM), posteriorly modeling label noise given the observed noisy labels.
We adopt PTM to provably bridge the gap between clean and noisy underlying data distribution, and anchor points (i.e., data points that belong to a speciﬁc class almost surely) empirical distribution simultaneously.
We also provide an easy-to-compute PTM estimation under speciﬁc condition of low label noise.
To extend the applicability to a wide range of label noise, motivated by Kalman ﬁltering (Kalman, 1960), we proposed the information fusion (IF) method linearly combining the estimated noise and PTM, which achieves lower transition matrix estimation error.
We empirically show that proposed method IF can achieve higher accuracy and more stable training.
The main contributions of this work are summarized below.
• We propose a new concept, named PTM, achieving consistent classiﬁer for underlying distribution and anchor point empirical distribution mismatch simultaneously.
• We propose the simple PTM estimation method based on observed noisy label under low label noise.
To extend the applicability, we propose IF method, linearly combining the estimated NTM and PTM, to achieve lower estimation error with theoretical and experimental justiﬁcation.
• We experimentally demonstrate the superiority of proposed IF method achieving higher accuracy and more stable training under synthetic and real-world label noise. Clean label
Empirical
Noisy label Noise TM C
a s e0 -- 1 -- ----
Posterior TM
C a s e
Clean Distribution
Noisy Distribution 1 Underlying  .
Noise TM
[Figure]
Posterior
[Figure]
P RELIMINARIES
Throughout this paper, we adopt uppercase letters to denote random variables, and lowercase letters to denote particular realization of the random variables.
We target the c -class classiﬁcation problem with instance-dependent (label) noise (IDN).
Let P ( X, Y ) be the underlying clean distribution of the random variables ( X, Y ) , where X and Y represent the instance and clean label.
However, in real-world scenario, the clean label is corrupted and thus only noisy label, denoted as ˜ Y , is observed.
Given a set of N training instances denoted by ˜ D :
= { ( x n , ˜ y n ) }
Nn =1 , where x n is the instance vector of the n -th sample, ˜ y n ∈ [ c ] := { 1 , · · · , c } is the corresponding observed but noisy label, our goal is to predict the clean label y n for any given instance x n .
Data distribution We assume that the unobserved clean samples ( x n , y n ) and available noisy samples ( x n , ˜ y n ) are drawn from unknown underlying clean distribution P ( X, Y ) and underlying noisy distribution P ( X, ˜ Y ) , respectively.
For the noisy samples data ˜ D , we may approximate underlying noisy distribution by the empirical noisy distribution , i.e., ˆ P ˜ D
( X, ˜ Y )
= 1 N (cid:80) Nn
=1 δ ( X = x n , ˜ Y =
˜
y n ) , where δ ( X = x n , ˜ Y =
˜
y n ) is a Dirac mass centered at ( x n , ˜ y n ) .
Since the clean data sample is unobserved, we deﬁne posterior empirical clean distribution ˆ P ( X, Y | ˜ D ) as the inference posterior empirical clean distribution from noisy samples data ˜ D .
Based on Bayes’s rule, the posterior empirical clean distribution is given by ˆ P ( X, Y | ˜ D )
= 1 N (cid:80) Nn
=1 (cid:80) cy n =1 δ ( X = x n , Y = ˜
y n ) P ( Y =
y n | ˜ Y =
˜
y n , X = x n ) .
Noise transition matrix T ( x ) To describe the corruption process of the label, the NTM T ( x ) ∈
R c × c is deﬁned as T i,j ( x ) =
P ( ˜ Y =
j | Y =
i, X = x ) , which represents the transition probability of the instance x labeling from clean i -th class to noisy j -th class.
The underlying noisy label ˜
Y probability given the instance x satisﬁes:
[Equation]
The NTM T ( x ) hence bridges the gap between the underlying clean labels probability P ( Y | X = x )
= (cid:104) P
( Y = 1 | X = x ) , · · · , P ( Y = c | X = x ) (cid:105) (cid:62) and underlying noisy labels probability
P ( ˜Y | X = x ) = (cid:104) P ( ˜ Y = 1 | X = x ) , · · · , P ( ˜ Y = c
| X = x ) (cid:105) (cid:62) given instance x , i.e., P ( ˜Y | X = x ) =
T ( x ) (cid:62) P ( Y | X = x ) .
Loss Correction method Let function f ( · ) represents a neural network and f ( x ) denotes the c -dimensional output probability for instance x , where the i th index of the output
y f,i ( x ) represents the predicted probability for class i .
The common approach is to minimize the cross-entropy (CE) loss l ( f ( x ) , y ) :
= − log( y f,y ( x )) to force the output y f,y ( x ) approximate 1 .
However, the label noise may mislead deep learning model.
The existing NTM based methods ﬁrst estimate NTM T ( x ) and then adopt it to correct the loss function.
For example, in the forward correction procedure (Patrini et al., 2017), the estimated NTM is adopted to corrupt the output predicted probability f ( x ) , i.e., the corrupted predicted probability is ˜ f ( x )
=
T ( x ) (cid:62) f ( x ) , and then compares it with the noisy label ˜ y .
Suppose T ( x ) is non-singular and the loss function is proper and composite, the forward loss correction can achieve consistent classiﬁer , i.e., the optimal classiﬁer for the corrected loss with respect to the underlying noisy distribution is the same as that for the CE loss with respect to the underlying clean distribution:
[Equation]
L EARNING WITH I NFORMATION F USION
Even though the existing loss correction methods could achieve consistent classiﬁer theoretically, their performances are still undesirable in practice.
Since the deep learning model is often trained on the empirical distribution with limited samples, the NTM cannot correctly bridge the empirical clean distribution and empirical noisy distribution, i.e., ˆ P ( ˜ Y =
j | X = x ) (cid:54) =
(cid:80) ci
=1 T ij ( x )ˆ P ˜ D
( Y = i
| X = x ) .
In other words, the NTM-based method may not work well in practice since due to the empirical clean and noisy distribution mismatch problem in IDN.
Aiming to utilize the observed noisy label, we propose a concept, named posterior transition matrix (PTM), to describe the transition probability given the observed noisy labels (formally deﬁned in
Section 3.1).
The motivation for PTM stems from a simple thought experiment as shown in Figure 1, which shows that PTM can model the empirical label noise better than NTM.
Figure 2 illustrates the overview framework of IF, including NTM and PTM estimations (Section 3.2), information fusion via linear combination (Section 3.3), and posterior loss function (Section 3.1).
[Figure]
3. T HE L OSS C ORRECTION M ETHOD
The main goal is to train a c -class neural network classiﬁer f ( x , ω ) to predict the clean label probability P ( Y | X ) .
Since only the noisy labels are observed, there is a gap between the clean and noisy label, describing via NTM (Goldberger & Ben-Reuven, 2017).
Motivated by the observed noisy labels (posterior information), we deﬁne the PTM W ( x ) to describe the posterior clean label probability given noisy label, where W i,j ( x ) =
P ( Y = i | ˜ Y = j, X = x ) .
We provide the relationship between the PTM W ( x ) and NTM T ( x ) via Bayes’ rulei.e.,
[Equation]
Notice that only the summation of any column is 1 for PTM W ( x ) , while the summation of any rows are 1 for NTM T ( x ) .
Subsequently, we provide posterior reweight loss correction method via NTM.
Deﬁnition 1 (Posterior reweight loss) Assume the model prediction is f ( x ) for noisy sample ( x , ˜ y ) and W ( x ) is the NTM associated with the noisy sample.
The posterior reweight loss is deﬁned as
[Equation]
We next analyze the property of the posterior reweight loss and provide the theoretical justiﬁcation.
Speciﬁcally, we analyze the expected risk R P ( X, ˜ Y )
( f ) =
E ( x , ˜ y ) ∼ P ( X, ˜ Y )
[ l p − rew (cid:16) ˜
y, f ( x ) (cid:17) ] and empirical risk
ˆ R ˆ D ( f )
= 1 N (cid:80) Nn
=1 l p − rew (cid:16) ˜
y n , f ( x n ) (cid:17) under noisy samples dataset.
Theorem 3.1 (Statistically Consistent Classiﬁer)
The posterior reweight loss can achieve unbiased classiﬁer for underlying distribution and empirical distribution.
(i) For underlying distribution, the expected risk satisﬁes
[Equation]
(ii)
For the anchor points samples ( x ap , ˜ y ap ) with underlying clean probability P ( Y = y ap |
X = x ap ) , the empirical risk satisﬁes l p − rew (cid:0)
˜
y ap , f ( x ap ) (cid:1)
=
l ( y ap , f ( x ap )) .
For the empirical distribution, the empirical risk satisﬁes
[Equation]
We also show that the PTM can be adopted in the forward manner.
The deﬁnition of posterior reweight loss is given as follows,
Deﬁnition 2 (Posterior forward loss) Assume the model prediction is f ( x ) for noisy sample ( x , ˜ y ) and W ( x ) is the NTM associated with the noisy sample.
The posterior forward loss is deﬁned as
[Equation]
We next analyze the property of the posterior forward loss and provide the theoretical justiﬁcation.
Lemma 3.2 The posterior forward loss is not larger than posterior reweighted loss l
[Equation]
Theorem 3.3 (Statistically Consistent Classiﬁer) Suppose the loss function l ( y, f ) is convex with respect to f (the convex condition is commonly satisﬁes, e.g., cross entropy loss), and the minimum expected risk R P ( X,Y ) ( f ) can achieve 0 , then the posterior forward loss can achieve unbiased classiﬁer for underlying distribution,
[Equation]
Compare the posterior forward loss with forward loss, it is seen that both of them correct the output of the neural network with statistically consistent risk guarantee, while the difference falls in the different transition matrix in the loss correction.
In addition, the statistically consistent risk for forward loss and posterior forward loss both require the accurate transition matrix estimation.
However, the label/PTM estimation is pretty challenging since the label noise is instance-dependent in real world.
Such observation motivates us to combine the NTM T ( x ) and PTM W ( x ) to correct the loss.
Motivated by Kalman ﬁltering(Kalman, 1960), a famous estimation method combining the prior knowledge and measurement information, we propose an information approach to tackle instance-dependent label noise via loss correction.
Speciﬁcally, we ﬁrstly claim that there are prior knowledge information and measurement information on the label noise, where prior knowledge information and measurement information represent the estimated NTM and PTM. [Figure]
PTM E STIMATION
The posterior loss correction requires knowing PTM W ( x ) given any instance x .
However, the PTM is unknown and needs to be estimated.
In this subsection, we provide a simple PTM estimation method under the condition that the neural network output f ( X ) probability can well approximate the underlying probability P ( Y | X ) , i.e., f ( x ) ≈ W
( X = x )ˆ P ( ˜Y | X = x ) .
Intuitively, this condition is strong since neural network is prone to overﬁt noisy data.
To achieve this condition, the warm-up training strategy and iterative PTM estimation are adopted.
Warm-up training to make "good" neural network output approximating the empirical clean distribution since the network ﬁts clean samples in the beginning.
Subsequently, we iteratively estimate the PTM for loss correction during training.
For the case that the number of appearance for instance x is large, the empirical noisy distribution convergences to the underlying noisy distribution, i.e., ˆ P ( ˜Y | X = x ) −→ P ( ˜Y | X = x ) .
For the long-tail instance with only one appearance ( x , ˜ y ) , the empirical noisy labels distribution is always one-hot.
Thus, this condition requires that the output of neural network f ( x ) should approximate to the underlying posterior clean probability, which is quite strong.
Under this condition, we propose a PTM estimation method based on the noisy labels and the model predicted probability.
For the long-tail instance with only one appearance ( x , ˜ y ) , the observed noisy labels distribution is the one-hot.
Compared with the NTM estimation (Xia et al., 2019; 2020), the proposed estimation method does not require the anchor point assumption that data points belong to a speciﬁc class almost surely.
Speciﬁcally, the PTM can be estimated as follows,
[Equation]
For the case that the number of appearance is larger than one, the PTM estimation is ill-posed since the number of unknown variable c 2 is larger the number of constraint c .
Thus, we choose the PTM with minimum Frobenius norm, which is consistent with that for the case of only one appearance instance.
Speciﬁcally, we have the following theorem:
Theorem 3.4 (Posterior transition matrix estimation) We aim to ﬁnd the solution with minimal norm solution, i.e.,
[Equation]
Then, we have the optimal solution as follows,
[Equation]
Theorem 3.4 provides the PTM estimation for general empirical noisy labels distribution.
For the case of only one appearance instance x , the empirical noisy distribution satisﬁes || ˆ P ( ˜ Y | x ) || 2 = 1 , and achieve consistent estimated PTM in equation (9).
3. I NFORMATION F USION
Subsection 3.2 introduces the PTM estimation methods based on the observed noisy label, the condition that neural network approximates clean label is quite strong even though the warm-up strategy and iterative estimation are adopted. Thus PTM estimation error could be large for large
IDN rate.
To further reduce the estimation error, motivated by Kalman ﬁltering, we propose the information fusion (IF) to obtain more accurate transition matrix estimation via weighted average
PTM estimation and NTM estimation with intuitive and theoretical justiﬁcation.
1 Intuitively, for each instance, the estimated NTM and PTM may have different estimation accuracy.
Via adaptively and linearly combine these two matrices, it is possible to obtain a more accurate transition matrix estimation.
Speciﬁcally, we quantify the estimation uncertainty and lower uncertainty incurs higher weight.
In this way, more accurate estimated transition matrix can be provided.
Before the illustration of information fusion algorithm, we need to quantify the transition matrix estimation error without ground-truth.
Suppose the (posterior) forward loss is adopted.
For the estimated NTM ˆ T ( x ) , the corrupted model predicted probability ˜ f
( x ) =
ˆ T (cid:62) ( x ) f ( x ) .
Suppose the corrupted model predicted probability is accurate, the noisy label
˜
Y satisﬁes c -dimension Bernoulli distribution with parameter ˜ f
( x ) , i.e., ˜ Y ∼ Bernoulli (cid:0) ˆ T (cid:62) ( x ) f ( x ) (cid:1) , where Bernoulli ( · ) represents multi-dimension Bernoulli distribution.
Furthermore, we deﬁne the uncertainty σ ˆ T for transition matrix ˆ T ( x ) as the trace of the covariance matrix for ˜ Y , i.e., σ ˆ T ( x ) =
Tr ( K ˜ Y ( x )) , where Tr ( · ) means the trace of matrix,
K ˜ Y ( x ) represents the covariance matrix of random variable ˜ Y .
For the multi-dimension Bernoulli distribution, we have the covariance matrix as follows,
[Equation]
Taking trace operation on the covariance matrix, we have the uncertainty for NTM as σ ˆ T ( x ) =
||
ˆ T (cid:62) ( x ) f ( x ) ||
−
||
ˆ T (cid:62) ( x ) f ( x ) || 2 .
Similarly, the uncertainty for the PTM estimation is given by σ ˆ W ( x ) =
|| ˆ W (cid:62) ( x ) f ( x ) ||
−
||
ˆ W (cid:62) ( x ) f ( x ) || 2 .
Once the uncertainty for estimated transition matrix has been established, we further integrate the two estimated transition matrices into Kalman transition matrix, deﬁned as W km ( x ) , via weighted average operation.
Mathematically, Kalman transition matrix is given by
[Equation]
(10) where the Kalman gain λ ( x ) is carefully selected to minimize the estimation error.
Suppose the true
PTM is W ∗ ( x ) , where W ∗ ij ( x )
=
if y = i and ˜ y
=
j , otherwise 0 .
Subsequently, we deﬁne the reconstruction error for NTM and PTM as e T ( x ) =
W ∗ ( x ) − ˆ T ( x ) 2 and e W ( x ) =
W ∗ ( x ) − ˆ W ( x ) , respectively.
The parameter λ ( x ) is determined via minimization of the mean square reconstruction error.
We theoretically justify the superiority of IF for a general scenario.
Theorem 3.5 (Theoretical justiﬁcation of IF)
Suppose the correlation coefﬁcient between reconstruction error e T ( x ) and e W ( x ) is deﬁned as cov ( x ) ∈ [ − 1 , 1] and the trace of covariance matrix for error e ˆ T ( x ) and e ˆ W can be quantiﬁed by σ ˆ T ( x ) and σ ˆ W ( x ) , i.e., Tr ( e ˆ T ( x ) e (cid:62) ˆ T ( x ))
=
σ ˆ T ( x ) and Tr ( e ˆ W ( x ) e (cid:62) ˆ W ( x ))
=
σ ˆ W ( x ) .
For the Kalman transition matrix W km , we deﬁne the reconstruction error as L mse ( λ ( x )) =
Tr (cid:16) ( W ∗ ( x ) − W km ( x ))
( W ∗ ( x ) − W km ( x )) (cid:62) (cid:17) , then the optimal kalman gain to minimize the reconstruction error is given by
[Equation]
and the minimum reconstruction error satisﬁes L mse ( λ ∗ ( x )) ≤ min( σ ˆ T ( x ) , σ ˆ W ( x )) .
Theorem 3.5 implies that the Kalman transition matrix W km ( x ) provably achieve lower estimation error for general e T ( x ) and e W ( x ) , either correlated or independent.
E XPERIMENTS
In this section, we empirically evaluate the effectiveness and stability of information fusion on synthetic and real-world noisy datasets.
We aim to answer two questions as follows.
Q1 :
Compared with the state-of-the-art transition matrix based methods, can information fusion achieve higher accuracy?
Q2 :
Can information fusion leads to more accurate and robust transition matrix estimation and more stable training?
4. E XPERIMENTAL S ETTINGS
Datasets We verify the superiority of IF on the three manually corrupted datasets, i.e., F-MNIST,
SVHN, CIFAR-10, and one real-world noisy dataset Clothing1M.
The ﬁrst three datasets contain clean data and we manually corrupted the label of training dataset following from (Xia et al., 2020).
More details about this instance-dependent label noise generation approach can be found in Appendix.
IDN- τ means that the controlled noise rate is τ .
All experiments on those datasets with synthetic instance-dependent label noise are repeated ﬁve times.
The real-world dataset Clothing1M has 1 M images with real-world noisy labels and 10 k images with clean labels for testing.
In experiments, we leave out 10% of the noisy training samples as a noisy validation set for model selection.
Baselines We compare the proposed IF method with (i) CE, which trains the standard deep network with the cross entropy loss on noisy datasets.
(ii) DMI, a novel information-theoretic robust loss function for instance-independent label noise (Xu et al., 2019).
(iii) Forward
(Patrini et al., 2017),
Reweight (Liu & Tao, 2015), and T-Revision (Xia et al., 2019).
(iv) Part-dependent transition matrix (PTD)
(Xia et al., 2020).
(v) instance-level forward correction (ILFC) (Berthon et al., 2021).
These approaches utilize the NTM to correct the loss function.
Implementations We choose the standard neural network and optimizer.
We use a ResNetnetwork for F-MNIST, a ResNet-34 network for SVHN and CIFAR-10.
For the optimization, we ﬁrst use SGD with 0 .
9 momentum, 10 − 4 weight decay, 128 batch size, 50 epochs and an initial learning rate of 10 − 2 to initialize the network.
Then, we adopt Adam optimizer and 5 × 10 − 7 learning rate to learn the NTM following PTD (Xia et al., 2020).
Once the NTM is obtained, we retrain the neural network.
In the forward propagation, Kalman transition matrix is obtained and then adopted to correction the loss function.
In addition, the revision trick (Xia et al., 2019) is also adopted for our proposed IF method, which introduces the slack variables ∆ W for the estimated Kalman transition matrix.
Speciﬁcally, the slack variable ∆ W is initialized as all zero elements in the experiments and can be optimized during the training.
Aiming to guarantee valid transition matrix, We ﬁrst project their negative entries of W km ( x ) + ∆ W to zero and then normalize the summation of each row as 1 .
Following (Xia et al., 2020), we do not use any data augmentation technique in the experiments.
For Clothing1M, we use a ResNet-50 pre-trained model on ImageNet.
We do not use any clean data to learn the transition matrices and classiﬁers.
After the NTM ˆ T ( x ) is obtained according to (Xia et al., 2020), we use SGD with 0 .
9 momentum, 10 − 3 weight decay, 32 batch size, and run with 10 −learning rate and 10 epochs.
For the evaluation, we adopt the test accuracy of the epoch with best accuracy on validation dataset. [Table] [Table]
E XPERIMENTAL R ESULTS
Accuracy on synthetic and real-world noisy datasets To answer Q1 , Figrue 3 shows the estimation error of PTD, PTM and IF methods with noise rate IDN- 10% to IDN- 40% , where the estimation error is deﬁned as the average difference of the transition probability err = 1 N (cid:80) Nn =1 (1 − ˆ W y n ˆ y n ) . It is seen that IF can achieve the lowest estimation error over all noise ratio. 3 Table 1 reports the classiﬁcation accuracy on the SVHN , CIFAR-10 and F-MNIST with noise rate IDN- 10% to IDN- 50% , where the highest accuracy is bold faced.
It is seen that information fusion achieves better performance compared with all baselines across the three datasets and ﬁve noise rate.
For example, in CIFAR-10 , the improvements over the strongest baseline is 1 .
22% , 9 . 89% for noise rate
IDN- 10% to IDN- 50% , respectively.
The higher accuracy implies that the posterior information is more important in higher noise rate.
In addition, the methods IF-F and IF-F-V is comparable across the three datasets and noise rate, which demonstrates that the revision on the transition matrix is not signiﬁcant beneﬁcial compared with the method PTD.
This observation validates the effectiveness of posterior information on transition matrix modiﬁcation.
For the real-world dataset Clothing1M, IF outperforms the baselines as shown in Table 2.
Our IF method achieves 0 .
5% improvements over the strongest baseline early-learning regularization (ELR) (Liu et al., 2020). CE Forward Reweight
T
[Table]
Revision
PTD
[Table]
PTD
[Table]
ELR
IF
[Table]
IF
[Table]
[Table]
[Table]
Transition matrix estimation evaluation during We study transition matrix estimation error in the training datat for IF and PTM to answer Q2 .
Figures 4 shows the training accuracy and estimation error of each epoch on CIFAR10 dataset with noise rate IDN- 10% to IDN- 40% , respectively.
The shadow area represents the standard deviation.
It is seen that, on all IDN rate, IF can achieve higher training accuracy and lower estimation error compared with IF.
Stability on synthetic noisy datasets We study the training stability via inspecting the test accuracy over epochs to answer Q2 .
Figures 5 shows the test accuracy with respect to epochs on CIFAR dataset with noise rate IDN- 10% to IDN- 40% , respectively.
The shadow area represents the standard deviation of the accuracy on the same epoch across ﬁve experiments.
We can clearly observe that, on both low-level and high level noise, IF shows higher accuracy in a more stable way since the posterior information of each instance provides reliable guideness on the loss correction.
C ONCLUSIONS
This paper presents PTM to characterize label noise given observed noisy label, and a simple but effective learning paradigm named information fusion, which trains deep neural networks under instance-dependent label noise.
The main goal of PTM is to characterize the label noise via limited observed noisy labels for each instance, which can bridge the empirical noisy and posterior empirical clean distribution.
Aiming to effectively combine NTM and PTM, we propose PTM to characterize label noise given observed noisy label.
Furthermore, we proposed an information method integrating both transition matrix to correct the loss function.
Experiments on synthetic noisy and real-world dataset show that IF can achieve higher accuracy and more stable training compared with state-of-theart methods, especially for high noise rate.
In the future, we can extend to incorporate more prior knowledge of the transition matrix, e g., sparsity or low rank, into the end-to-end learning algorithm.
A T HEORETICAL ANALYSIS A. P ROOF OF T HEOREM 3.
Based on the deﬁnition of posterior reweight loss
[Equation]
we have l p − rew (cid:16) ˜
y, f ( x ) (cid:17) =
(cid:80) cy
=1 P ( Y = y | ˜ Y =
˜
y, X = x ) l ( y, f ( x )) .
For noisy underlying distribution, the expected risk can be calculated as follows,
[Equation]
which implies that posterior reweight loss over noisy underlying distribution equals to cross entropy loss over clean underlying distribution and thus posterior reweight loss induces to statistically consistent classiﬁer.
For the anchor points samples ( x ap , ˜ y ap ) with underlying clean probability P ( Y = y ap | X
= x ap ) , based on equation (2), the PTM value satisﬁes W y ap , ˜
y
ap ( x ) = 1 and W k, ˜ y
ap
= 0 for any k (cid:54) =
y ap , which implies that empirical distribution mismatch on anchor points samples can be completely mitigated.
As for empirical distribution, the empirical risk satisﬁes
[Equation]
[Equation]
For noisy empirical distribution, the expected risk satisﬁes
[Equation]
[Equation]
which implies that posterior reweight loss over noisy empiricial distribution equals to cross entropy loss over posterior clean empirical distribution.
A. P ROOF OF L EMMA 3.
Note that the loss function l ( y, f ) is convex with respect to f and (cid:80) ci =1 W i, ˜ y ( x ) =
(cid:80) ci =1 P
( Y = i | ˜ Y =
˜
y, X = x ) = 1 for any ˜ y and x , according to Jensen’s inequality, we have
[Equation]
which shows the relation between posterior reweight loss and posterior forward loss.
A. P ROOF OF T HEOREM 3.
Based on Lemma 3.2 and Theorem 3.
[Equation]
we have the inequality on the expected risk as follows, E
[Equation]
Deﬁne the optimal classiﬁer f ∗ to minimize expected risk for underlying clean distribution and cross entropy loss as
[Equation]
then R P ( X,Y ) ( f ∗ ) = 0 and for l p − rew (cid:0) ˜
y, f ∗ ( x ) (cid:1) =
for any x .
Note that the expected risk for posterior forward loss is non-negative and not larger than that for posterior reweight loss, we also have l p − fw (cid:0)
˜
y, f ∗ ( x ) (cid:1) =
for any x .
In addition, the difference of expected risk for two different classiﬁers satisﬁes
[Equation] [Equation]
which implies that the classiﬁer f ∗ can also minimize expected risk for underlying noisy distribution and posterior reweight loss.
This completes the proof.
A. P ROOF OF T HEOREM 3.
Aiming to standardize the constraint, we vectorize W ( x ) as c 2 × 1 dimension w vec ( x ) :=
[ W 11 ( x ) , · · · , W c 1 ( x ) , · · · , W 1 c ( x ) , · · · , W cc ( x )] (cid:62) and deﬁne the coefﬁcient matrix A ( x ) as follows,
[Equation]

[Equation]
where P (cid:62) ( ˜Y | x ) =
[ P ( ˜ Y = 1 | x ) , · · · , P ( ˜ Y =
c | x )] 1 × c .
Therefore, the constraint W (cid:62) ( x )ˆ P ( ˜ Y | x ) = f
( x ) can be transformed as A ( x ) w vec ( x ) =
f
( x ) , which is a least-norm problem with undetermined equations constraint.
Note that matrix
A
( x ) is full column rank, we can prove that the least-norm solutions is
[Equation]
Considering any possible solution w vec ( x ) satisfying the constraint A ( x ) w vec ( x ) =
f
( x ) , then
A
[Equation]
= 0 and [ w vec ( x ) − w ∗ vec ( x )]
[Equation]
vec
[Equation]
which implies that the vectors w vec ( x ) − w ∗ vec ( x ) and w ∗ vec ( x ) are perpendicular.
Thus, the Lnorm of w vec ( x ) satisﬁes
[Equation]
[Equation]
which shows that the equation (19) is the optimal solution.
Via direct calculation, it is easy to obtain that A ( x ) A (cid:62) ( x ) = (cid:80) cj =
k ˆ P 2 ( ˜ Y
= k | x ) I c × c .
Therefore, the element of PTM satisﬁes
[Equation]
21) i.e., W ∗ [Equation] x ) = [Equation]
P [Equation] ˜ Y | x
[Equation]
|| [Equation] P (˜ Y | x ) || 22 and completes the proof.
A. P ROOF OF T HEOREM 3.
Considering the general reconstruction error relationship, the trace of the covariance matrix for reconstruction error satisﬁes Tr ( e ˆ T ( x ) e (cid:62) ˆ W ( x ))
=
cov ( x ) (cid:112) σ T ( x ) σ W ( x ) .
For Kalman transition matrix, the reconstruction deviation satisﬁes W ∗ ( x ) − W km ( x ) =
(1 − λ )
e T ( x ) + λe W ( x ) .
For simplicity, we remove the instance x to abbreviate the notations.
Therefore, the reconstruction error is given by
[Equation]
For such quadratic function
[Equation]
it is easy to obtain the optimal parameter via the local minimal ∂ L [Equation] [Equation] and thus the optimal weight is given by λ ∗ =
[Equation]
Plugging in the optimal weight into the reconstruction error, we obtain the minimal reconstruction error is given by [Equation] Aiming to investigate the relation between minimal reconstruction error for IF, PTM and PTD, we adopt variable substitution t = σ T + σ W − 2 cov √ σ T σ W ≥ 0 to further simplify minimal reconstruction error L mse ( λ ∗ ) , then we have
[Equation]
where the inequality holds according to arithmetic mean-geometric mean inequality.
Therefore, the linear combination can provably achieve lower estimation error for general e T ( x ) and e W ( x ) , either correlated or independent.
B R EPRODUCIBILITY B. D ATASET S TATISTICS
For fairly comparison with previous work, we performance the image classiﬁcation task on the three manually corrupted datasets, i.e., F-MNIST, SVHN, CIFAR-10, and one real-world noisy dataset
Clothing1M.
They have been widely adopted to study label noise problem.
The detailed statistics are listed in Table 3.
[Table]
B. R UNNING E NVIRONMENT
All baselines and IF approaches are implemented in PyTorch, and tested on a machine with AMD
EPYC 7282 16-core processors, 4 GeForce GTX-3090 Ti GPUs with 24GB memory size.
B. A LGORITHMS
We summarize the IF algorithm to correct the loss and provide the pseudo codes in Algorithm 1.
For instance-dependent Label noise generation, we provide pseudo codes with controllable noise rate are provided in Algorithm 2.
This algorithm follows the state-of-the-art method (Xia et al., 2020; Zhu et al., 2021a), where the overall noise rate is τ . Input
Output 1 Estimate the NTM ˆ T ( x ) according to (Xia et al., 2020) ; 2 while Validation accuracy has increases in the last t epochs do
Calculate the PTM ˆ W ( x ) based on Equation (9) ; Obtain the uncertainty for noise and PTM ; Calculate the Kalman gain and Kalman transition matrix based on Equations (10) and (11) ; Correct the loss as posterior forward loss
Kalman transition
W
; 7 end
B. L EARNING N OISE T RANSITION M ATRIX VIA D ECOMPOSITION
For the instance-dependent label noise, the NTM may be different for different samples.
Thus, the complexity of NTM estimation is O ( Nc 2 ) , which leads to ill-posed estimation problem.
(Xia et al., 2020) introduces a part-dependent decomposition on the NTM, i.e., the noise of an instance depends only on its parts.
Speciﬁcally, given L part-dependent transition matrix, e.g, P l ∈ [0 , 1] c × c , where j = 1 , · · · , L , the instance-dependent matrix can be approximated by a combination of partdependent transition matrix and the combination coefﬁcients, deﬁned as h ( x ) ∈ [0 , 1] L , depend on the feature vectors.
Then the instance-dependent transition matrix W ( x ) can be approximated as follows [Equation]
[Equation]
where h j ( x ) represents the j -th elements of h ( x ) and satisﬁes the normalization condition
|| h ( x ) ||
= 1 for any features vector to maintain that the summation of column is 1 for NTM.
In this work, we follow (Xia et al., 2020) to learn the NTM in three steps: (i) learning combination coefﬁcients h ( x ) via parts-based features representation (Lee & Seung, 1999; Tao et al., 2017); (ii)
(Xia et al., 2020) justiﬁes the part-dependent decomposition assumption via parts-based representations.
For example, (Hosseini-Asl et al., 2016) shows that the input features rely on parts-based representations in object recognition.
Thus, it is natural to approximate the label noise in the part level.
In display advertising, the delay feedback of the label (Ktena et al., 2019; Yasui et al., 2020) introduces label noise.
Earlier the data is generated, the less probability the feedback exists.
Thus, the label noise highly depends on part of features.
Since the instance-dependent NTM may be poorly learned, the slack variable trick (Xia et al., 2019; 2020) is also adopted to modify the instance-dependent transition matrix, which signiﬁcantly improve the accuracy.
However, the slack variable trick ignores the posterior information for each instance, which limits the accuracy.
In addition, the variance of accuracy in (Xia et al., 2020) is large since there is not any guide information to modify the instance-dependent NTM except minimization of the corrected loss.
In this paper, we claim that the measurement information is useful to modify the NTM.
We will introduce how to estimate the PTM and information fusion in the next two subsections.
B. I MPLEMENTATION D ETAILS
Warm-up Training Strategy Aiming to estimate the transition matrix, we adopt warm-up training strategy that training the neural network via cross entropy loss with noisy data.
we use SGD as the optimizer with 0 .
9 momentum, 128 batch size, and 10 epochs.
With the beneﬁt of memorization effect (Arpit et al., 2017), warm-up training with appropriate epochs can guarantee that the predicted probability of neural network is reliable and can approximate underlying clean probability.
Training with
Corrected
Loss
Our proposed PTM estimation relies on the neural network predicted probability.
Although the warm-up training can guarantee the prediction accuracy for clean instances with the beneﬁts of memorization effect, the prediction of the PTM estimation may still be not reliable.
To ease the dilemma, we adopt sample selection to choose the reliable prediction and thus guarantee the reliable PTM estimation for speciﬁc instance.
Similar to (Cheng et al., 2021), the small-loss criteria is adopted to select the instances with small training loss for training.
Speciﬁcally, we select the instance via the weighted loss if the instance loss is less than the average loss over all classes.
For the instance satisfying the small loss criterion, we have conﬁdence that the instance is clean, which means the PTM is identity.
To put it simply, we sum the estimated PTM and identity and then adopt column normalization.
It is worth noting that the noisy labels are adopted in forward propagation, which is highly different with previous transition matrix-based works (Xia et al., 2019;
Liu & Tao, 2015; Patrini et al., 2017; Xia et al., 2020).
Compared with hard sample selection based methods (Cheng et al., 2021), our method achieves the soft sample selection and adaptively correct the loss for “clean" instance.
C A DDITIONAL E XPERIMENTAL R ESULTS C. S TABILITY ON S YNTHETIC N OISY D ATASETS
We study the training stability via inspecting the test accuracy over epochs.
Figures 6 and 7 show the test accuracy with respect to epochs on SVHN and F-MNIST datasets with noise rate IDN- 10% to IDN- 40% .
It is seen that our proposed method IF can achieve better accuracy and stable training compare with baseline PTD, which implies that the PTM fusion can decrease the transition matrix estimation error and improve the training stability.
T-Revision trick (-V method) proposed in (Xia et al., 2019) aims to compensate the NTM and thus improve the accuracy performance.
It is seen that
PTD-F-V can achieve better performance than PTD-F, which veriﬁes that the learned NTM is not very accurate.
For our proposed method, IF-V can achieve comparable performance than IF-F and demonstrates that transition matrix revision trick can not improve the performance since PTM has already been employed to compensate the transition matrix error. [Figure] [Figure]
Compared with different noise rates, it is seen that the accuracy of PTD method may decrease with respect to epochs, especially for large noise rate.
Under large noise rate, the number of clean samples is limited and the neural network may already overﬁt noisy samples during warm-up training and thus hurt the transition matrix estimation.
In this scenario, PTD method does not have the capability to adjust such poor NTM estimation and thus leads to the accuracy drop during training.
Our proposed method IF and T-Revision trick both adjust the estimated transition matrix and mitigate the accuracy drop.
C. E XPERIMENTAL R ESULTS ON H IGH N OISE R ATIO
We investigate the performance of IF for a high noise ratio compared with PTM and PTD. Table.shows the test accuracy for high noise ratio IDN- 70% , IDN- 80% and IDN- 90% in CIFAR 10 dataset. We have two observations: (1) Comparing with PTD at the noise ratios of IDN- 80% and IDN- 90% , our PTM could still outperform PTD.
Although the network output is not perfect, the performance deterioration of PTD is still higher than PTM.
(2) Our IF consistently delivers the most superior performance no matter under the low or high noise.
That is because IF adaptively and linearly combines NTM and PTM in instance level and thus achieves more accurate estimation.
[Table]
C. E XPERIMENTAL R ESULTS ON M ORE B ASELINES
Although many transition matrix-based methods are compared in Table.
1, we are also interested in several more baselines, including label smoothing (LS) (Szegedy et al., 2016), early-learning regularization (ELR) (Liu et al., 2020), (Berthon et al., 2021), progressive label correction (PLC) (202, 2021), conﬁdence regularized sample sieve (CORES)
(Cheng et al., 2021).
It could be observed that our method can still achieve state-of-the-art performance compared with all other baselines.
Table.
5 demonstrates the test accuracy over IDN- 10% to IDN- 50% label noise ratio in CIFAR 10.
It could be observed that our method can still achieve state-of-the-art performance, based on which we can conﬁdently validate the effectiveness of our method
[Table] [Table] D R ELATED WORK Label Noise Model There are three types of label noise model, including the random classiﬁcation noise (RCN) model (Manwani & Sastry, 2013;
Natarajan et al., 2013), the class-conditional label noise (CCN) model (Patrini et al., 2017; Xia et al., 2019; Zhang & Sabuncu, 2018; Liu & Guo, 2020) and the e instance-dependent label noise (IDN) model (Cheng et al., 2020; Xia et al., 2020; Cheng et al., 2021; Berthon et al., 2021; Zhu et al., 2021a).
Speciﬁcally, RCN means that each label is ﬂipped independently with a constant probability, CCN assumes that the ﬂip probability (noise rates) are the same for the instance with the same clean labels.
IDN is the most general case, where the ﬂip probability depends on its instance.
Loss Correction via Transition Matrix
The transition matrix bridges the gap between the model predicted probability for noisy and clean data, which can achieve risk-consistent classiﬁer with label noise.
The transition matrix can be estimated via cross-validation method (Natarajan et al., 2013), anchor points assumption (Xia et al., 2019; Yu et al., 2018; Yao et al., 2020).
Subsequently, the estimated transition matrix can be adopted to correct the loss function via forward correction (Patrini et al., 2017), backward correction (Patrini et al., 2017) and reweight (Xia et al., 2019; Liu & Tao, 2015).
Loss Correction via Sample Selection Due to the experiments observation– the neural network learns clean instances ﬁrst (Arpit et al., 2017) and memorizes the instances gradually, there are lots of methods attempting to identify mislabeled training examples and then ﬁlter them out (Brodley &
Friedl, 1999; Jiang et al., 2018; Han et al., 2018b; Yu et al., 2019; Angelova et al., 2005; Huang et al., 2019; Li et al., 2019).
The core idea is to select the “small loss" samples as clean one.