A N I NFORMATION F USION A PPROACH TO L EARNING WITH I NSTANCE -D EPENDENT L ABEL N OISE
A BSTRACT
Instance-dependent label noise (IDN) widely exists in real-world datasets and usually misleads the training of deep neural networks.
Noise transition matrix (NTM) (i.e., the probability that clean labels flip into noisy labels) is used to characterize the label noise and can be adopted to bridge the gap between clean and noisy underlying data distributions.
However, most instances are long-tail, i.e., the number of occurrences of each instance is usually limited, which leads to the gap between the underlying distribution and the empirical distribution.
Therefore, the genuine problem caused by IDN is empirical , instead of underlying, data distribution mismatch during training.
To directly tackle the empirical distribution mismatch problem, we propose posterior transition matrix (PTM) to posteriorly model label noise given limited observed noisy labels, which achieves statistically consistent classifiers .
Note that even if an instance is corrupted by the same NTM, the intrinsic randomness incurs different noisy labels, and thus requires different correction methods.
Motivated by this observation, we propose an I nformation F usion (IF) approach to fine-tune the NTM based on the estimated PTM.
Specifically, we adopt the noisy labels and model predicted probabilities to estimate the PTM and then correct the NTM in forward propagation .
Empirical evaluations on synthetic and real-world datasets demonstrate that our method is superior to the state-of-the-art approaches, and achieves more stable training for instance-dependent label noise.
I NTRODUCTION
Data labels annotated from human efforts, such as crowdsourcing (Yan et al., 2014; Chen et al., 2017) and online queries (Divvala et al., 2014), may be heavily noisy in practice (Wei et al., 2022).
To make it worse, the label noise stemmed from human annotations is often instance-dependent.
For example, the images close to the decision boundary are usually prone to be mislabeled (Zhang et al., 2021b; Zhu et al., 2021b).
On the other hand, the remarkable success of deep neural networks (DNNs) on supervised learning tasks heavily relies on the expressive power and a large number of data with accurate labels.
Unfortunately, deep neural networks memorizes noisy labels leading to poor generalization (Zhang et al., 2017).
It is challenging to learn with practical instance-dependent label noise (IDN) due to the hidden and complicated label noise properties (Liu, 2021; Zhu et al., 2022b; Cheng et al., 2021b; Zhu et al., 2022a).
The methods dealing with noisy labels fall into two lines, including heuristically identifying noisy samples and statistical label noise modeling.
The training of deep neural networks often learns clean labels first (Arpit et al., 2017) and then gradually memorizes noisy labels, which is recognized as the memorization effect.
Based on the general memorization effect, the heuristic methods are all designed by following the anomaly detection strategy: identify noisy samples based on different behaviors (e.g., loss values) between clean and noisy samples during training Cheng et al.
(2021a). The typical methods contain sample selection (Yu et al., 2019; Han et al., 2018b), reweight samples (Cheng et al., 2021a; Jiang et al., 2018; Ren et al., 2018), label correction (Ma et al., 2018; Tanaka et al., 2018), and regularization (Han et al., 2018a).
Although these algorithms empirically work well, the reliability cannot be guaranteed explicitly without modeling label noise.
Another line of works relies on noise transition matrix (NTM) to model label noise statistically (Xia et al., 2019; 2020; Patrini et al., 2017) by quantifying the probabilities that clean labels flip into noisy labels.
Although the NTM-based methods possess theoretical guarantee, NTM estimation for each instance under IDN is pretty challenging.
To ease the estimation, some unrealistic assumptions have been made on NTM, including instance-independent transition matrix (Liu & Guo, 2020; Wei & Liu, 2021;
Li et al., 2021), symmetric transition matrix (Menon et al., 2018), upper bounded noise rate (Cheng et al., 2020), and part-dependent label noise (Xia et al., 2020).
However, under the complex IDN, the empirical noise distribution could be highly different from the underlying noise distribution.
For example, in Figure 1, the underlying and empirical noisy distributions for long-tail instances are different since the empirical noisy label can be either the same as or different from the clean label.
Additionally, observed noisy labels provide inductive bias toward label corruption.
In other words, the genuine problem arising from IDN is the empirical , instead of underlying, clean and noisy distribution mismatch problem.
To mitigate the empirical distribution mismatch problem, we propose the posterior transition matrix (PTM) to model label noise given the observed noisy labels.
We adopt PTM to provably bridge the gap between clean and noisy underlying data distributions, and empirical distribution of all anchor points (i.e., data points that belong to a specific class almost surely (Xia et al., 2020)) simultaneously.
We also provide an easy-to-compute PTM estimation method under the low label noise condition.
To further extend the applicability, motivated by Kalman filtering (Kalman, 1960), we propose the information fusion (IF) method to linearly combine the estimated NTM and PTM, which achieves lower transition matrix estimation error.
We empirically show that the proposed IF method can achieve higher accuracy and more stable training.
The main contributions of this work are summarized below.
• We propose a new concept of PTM to achieve statistically consistent classifiers for underlying distribution mismatch and anchor point empirical distribution mismatch simultaneously.
• We propose a simple yet effective PTM estimation method based on observed noisy labels under the condition of low label noise ratio.
To extend the applicability, we propose an IF method, which combines the estimated NTM and PTM to achieve lower estimation error with both theoretical and experimental justifications.
• We experimentally demonstrate that the proposed IF method achieves higher accuracy and more stable training under synthetic and real-world label noise.
[Figure]
[Figure] [Figure] TM
C
a s e0 -- 1 -- ----
[-Posterior TM 
-]
[Figure]
C a s e
[Figure]
P RELIMINARIES
Throughout this paper, we adopt uppercase letters to denote random variables, and lowercase letters to denote particular realization of the random variables.
We target the c -class classification problem with instance-dependent label noise (IDN).
Let P ( X, Y ) be the underlying clean distribution of the random variables ( X, Y ) , where X and Y represent the instance and clean label, respectively.
In real-world scenarios, clean labels are usually corrupted, and thus only noisy labels, denoted as ˜ Y , are observed.
Given a set of N training instances denoted by ˜ D :
= { ( x n , ˜ y n ) } N n =1 , where x n is the instance vector of the n -th sample and ˜ y n ∈ [ c ] := { 1 , · · · , c } is the corresponding observed noisy label, our goal is to predict the clean label y n for any given instance x n .
Data distribution. We assume that the unobserved clean samples ( x n , y n ) and available noisy samples ( x n , ˜ y n ) are drawn from the unknown underlying clean distribution P ( X, Y ) and the underlying noisy distribution P ( X, ˜ Y ) , respectively.
For the noisy samples ˜ D , we may approximate the underlying noisy distribution by the empirical noisy distribution , i.e., ˆ P ˜ D
( X, ˜ Y )
= 1 N (cid:80) Nn
=1 δ ( X = x n , ˜ Y =
˜
y n ) , where δ ( X = x n , ˜ Y =
˜
y n ) is a Dirac mass centered at ( x n , ˜ y n ) .
Since clean data samples are unobserved, we define the posterior empirical clean distribution ˆ P ( X, Y | ˜ D ) as the inferred posterior empirical clean distribution from the noisy samples ˜ D .
Based on Bayes’ rule, the posterior empirical clean distribution is given by ˆ P ( X, Y | ˜ D )
= 1 N (cid:80) Nn
=1 (cid:80) cy n =1 δ ( X = x n , Y = ˜
y n ) P ( Y =
y n | ˜ Y =
˜
y n , X = x n ) .
Noise transition matrix T ( x ) . To describe the corruption process of the clean label, the NTM
T ( x ) ∈ R c × c is defined as T i,j ( x ) =
P ( ˜ Y =
j | Y =
i, X = x ) , which represents the transition probability of flipping the instance x ’s label from the clean i -th class to the noisy j -th class.
The probability of the underlying noisy label ˜
Y given the instance x satisfies:
[Equation]
i = T ij ( x ) P ( Y = i | X = x ) . The NTM T ( x ) hence bridges the gap between the underlying clean label probability P ( Y | X = x )
= (cid:104) P
( Y = 1 | X = x ) , · · · , P ( Y = c | X = x ) (cid:105) ⊤ and underlying noisy label probability
P ( ˜Y | X = x ) = (cid:104) P ( ˜ Y = 1 | X = x ) , · · · , P ( ˜ Y = c
| X = x ) (cid:105) ⊤ given the instance x , i.e., P ( ˜Y | X = x ) = T ( x ) ⊤ P ( Y | X = x ) .
Loss correction method. Let function f ( · ) represent a neural network and f ( x ) denote the cdimensional output probability for instance x , where the i th index of the output f i ( x ) represents the predicted probability for class i .
The common approach is to minimize the cross-entropy (CE) loss l ( f ( x ) , y ) :
= − log( f y ( x )) to force the output f y ( x ) to approximate 1 .
However, the label noise may mislead a deep learning model.
The existing NTM-based methods first estimate NTM T ( x ) and then adopt it to correct the loss function.
For example, in the forward correction procedure (Patrini et al., 2017), the estimated NTM is adopted to corrupt the predicted probability f ( x ) , i.e., the corrupted predicted probability is ˜ f ( x ) =
T ( x ) ⊤ f ( x ) , and then the corrupted predicted probability is enforced to approximate the noisy label ˜ y .
Suppose T ( x ) is non-singular and the loss function is proper and composite. The forward loss correction can achieve a consistent classifier , i.e., the optimal classifier for the corrected loss with respect to the underlying noisy distribution is the same as that for the CE loss with respect to the underlying clean distribution:
[Equation]
L EARNING WITH I NFORMATION F USION
Even though the existing loss correction methods could achieve consistent classifiers theoretically, their performances are still undesirable in practice.
Since a deep learning model is often trained on the empirical distribution with limited samples, the NTM cannot correctly bridge the empirical clean distribution and empirical noisy distribution, i.e., ˆ P ( ˜ Y =
j | X = x ) ̸ = (cid:80) ci
=1 T ij ( x )ˆ P ˜ D
( Y = i
| X = x ) .
In other words, the NTM-based method may not work well in practice due to the empirical clean and noisy distribution mismatch problem in IDN.
With the goal of utilizing the observed noisy labels, we propose a concept, named posterior transition matrix (PTM), to describe the transition probabilities given the observed noisy labels (formally defined in Section 3.1).
The motivation for PTM stems from a simple thought experiment as shown in Figure 1, which shows that PTM can model the empirical label noise better than NTM.
Figure 2 illustrates the
[Figure]
adopts overall framework of IF, including NTM and PTM estimations (Section 3.2), information fusion via linear combination (Section 3.3), and posterior loss function (Section 3.1).
3. T HE L OSS C ORRECTION M ETHOD
The main goal is to train a c -class neural network classifier f ( x , ω ) to predict the clean label probability P ( Y | X ) .
Since only the noisy labels are observed, there is a gap between the clean and noisy label, described via NTM (Goldberger & Ben-Reuven, 2017).
Motivated by the observed noisy labels (i.e., posterior information), we define the PTM W ( x ) to describe the posterior clean label probability given noisy labels, where W i,j ( x ) =
P ( Y = i | ˜ Y =
j, X = x ) .
We provide the relationship between the PTM W ( x ) and NTM T ( x ) via Bayes’ rule:
[Equation]
Notice that the summation of any column is 1 for PTM W ( x ) , while the summation of any row isfor NTM T ( x ) .
Subsequently, we provide a posterior reweight loss correction method via NTM.
Definition 1 (Posterior reweight loss) Assume the model prediction is f ( x ) for noisy sample ( x , ˜ y ) and W ( x ) is the PTM associated with the noisy sample.
The posterior reweight loss is defined as
[Equation]
We next analyze the property of the posterior reweight loss and provide the theoretical justification.
Specifically, we analyze the expected risk R P ( X, ˜ Y )
( f ) =
E ( x , ˜ y ) ∼ P ( X, ˜ Y )
[ l p − rew (cid:16) ˜
y, f ( x ) (cid:17) ] and empirical risk ˆ R ˆ D
[Equation]
= [Equation]
under the noisy samples.
Theorem 3.1 (Statistically Consistent Classifier)
The posterior reweight loss can achieve an consistent classifier for the underlying distribution and empirical distribution.
(i) For the underlying distribution, the expected risk satisfies
[Equation]
(ii)
For the anchor point samples ( x ap , ˜ y ap ) with the underlying clean probability P ( Y = y ap |
X = x ap ) , the empirical risk satisfies l p − rew (cid:0)
˜
y ap , f ( x ap ) (cid:1)
=
l ( y ap , f ( x ap )) .
For the empirical distribution, the empirical risk satisfies
[Equation]
We also show that the PTM can be adopted in the forward manner.
The definition of the posterior reweight loss is given below.
Definition 2 (Posterior forward loss) Assume the model prediction is f ( x ) for noisy sample ( x , ˜ y ) and W ( x ) is the PTM associated with the noisy sample.
The posterior forward loss is defined as
[Equation]
We next analyze the property of the posterior forward loss and provide the theoretical justification.
Lemma 3.2 The posterior forward loss is not larger than the posterior reweight loss
[Equation]
Theorem 3.3 (Statistically Consistent Classifier) Given that the loss function l ( y, f ) is convex with respect to f (the convex condition can be commonly satisfied, e.g., the cross-entropy loss) and that the minimum expected risk R P ( X,Y ) ( f ) can achieve 0 , the posterior forward loss can achieve an consistent classifier for the underlying distribution,
[Equation]
By comparing the posterior forward loss with the forward loss (Patrini et al., 2017), it can be seen that both of them correct the output of a neural network with statistically consistent risk guarantee, while the difference falls into the different transition matrices used in the loss correction.
In addition, the statistically consistent risks for the forward loss and posterior forward loss both require accurate transition matrix estimation.
However, the NTM and PTM estimation is highly challenging since label noise is instance-dependent in reality.
Such an observation motivates us to combine both NTM T ( x ) and PTM W ( x ) to correct the loss.
Motivated by Kalman filtering (Kalman, 1960), a famous estimation method combining prior knowledge and measurement information, we propose an Information Fusion (IF) approach to tackle instance-dependent label noise via loss correction.
Specifically, we first claim that there is prior knowledge information and measurement information in the label noise, where prior knowledge information and measurement information correspond to the estimated NTM and PTM, respectively.
[Figure]
3. PTM E STIMATION
The posterior loss correction requires knowing
PTM W ( x ) given any instance x .
However, the
PTM is unknown and needs to be estimated.
In this subsection, we provide a simple yet effective PTM estimation method under the condition that the neural network output f ( X ) probability can well approximate the underlying probability P ( Y | X ) , i.e., f ( x ) ≈ W
( X = x )ˆ P ( ˜Y | X = x ) .
Intuitively, this condition seems strong since a neural network is prone to overfit noisy data.
To achieve this condition, the warm-up training strategy and iterative PTM estimation are adopted.
Warm-up training can make “good” neural network outputs approximating the empirical clean distribution since the network fits clean samples in the beginning of training (Liu et al., 2020).
Subsequently, we iteratively estimate the PTM for loss correction during training.
For the case that the number of occurrences of instance x is large, the empirical noisy distribution converges to the underlying noisy distribution, i.e., ˆ P ( ˜Y | X = x ) −→ P ( ˜Y | X = x ) .
For a long-tail instance with only a single occurrence ( x , ˜ y ) , the empirical noisy label distribution is always one-hot.
Thus, this condition requires that the output of neural network f ( x ) should approximate to the underlying posterior clean probability, which is quite strong.
Under this condition, we propose a PTM estimation method based on the noisy labels and the model predicted probability.
Compared with the NTM estimation method (Xia et al., 2019; 2020), the proposed estimation method does not require the anchor point assumption that some data points belong to a specific class almost surely.
Specifically, the PTM can be estimated as ˆ W i,j ( x ) =
f i ( x ) if ˜ y =
j , otherwise is 0 . For the case that the number of occurrences is larger than one, the PTM estimation is ill-posed since the number of unknown variables c 2 is larger than the number of constraint c (i.e., number of classes).
Thus, we choose the PTM with the minimum Frobenius norm, which is consistent with the case of having only a single occurrence.
Specifically, we have the following theorem.
Theorem 3.4 (Posterior transition matrix estimation)
The optimal solution to finding the PTM with the minimal Frobenius norm, i.e.,
[Equation]
is given below:
[Equation]
Theorem 3.4 provides the PTM estimation for general empirical noisy label distributions.
For the case of instance x with only a single occurrence, the empirical noisy distribution satisfies || ˆ P ( ˜ Y | x ) || 2 = 1 , and achieves consistent estimated PTM in Equation (8).
3. I NFORMATION F USION
Section 3.2 introduces the PTM estimation method based on the observed noisy labels. However, the condition that a neural network approximates clean labels could still be strong even after the warm-up strategy and iterative estimation are adopted, and PTM estimation error could be large for large IDN rates.
To further reduce the estimation error, motivated by Kalman filtering, we propose the information fusion (IF) approach to obtain more accurate transition matrix estimation via weighted average of PTM estimation and NTM estimation with both intuitive and theoretical justifications 1 .
Intuitively, for each instance, the estimated NTM and PTM may have different estimation accuracy, and, therefore, it is possible to obtain a more accurate transition matrix estimation by adaptively and linearly combine these two matrices.
Specifically, we first quantify the estimation uncertainty and assign higher weight for the estimation with lower uncertainty.
In this way, a more accurate estimated transition matrix can be generated.
Before illustrating the information fusion algorithm, we need to quantify the transition matrix estimation error without ground truth.
Suppose posterior forward loss is adopted.
For the estimated NTM ˆ T ( x ) , the corrupted model predicted probability is ˜ f
( x ) =
ˆ T ⊤ ( x ) f ( x ) .
If the corrupted model predicted probability is accurate, the noisy label ˜
Y satisfies c -dimension Bernoulli distribution with parameter ˜ f
( x ) , i.e., ˜ Y ∼ Bernoulli (cid:0) ˆ T ⊤ ( x ) f ( x ) (cid:1) , where Bernoulli ( · ) represents a multidimension Bernoulli distribution.
Furthermore, we define the uncertainty σ ˆ T for transition matrix ˆ T ( x ) as the trace of the covariance matrix for ˜ Y , i.e., σ ˆ T ( x ) =
Tr ( K ˜ Y ( x )) , where Tr ( · ) means the trace of a matrix, K ˜ Y ( x ) represents the covariance matrix of random variable ˜ Y .
For the multi-dimension Bernoulli distribution, we have the covariance matrix as follows,
[Equation]
Taking trace operation on the covariance matrix, we have the uncertainty for NTM as σ ˆ T ( x ) =
||
ˆ T ⊤ ( x ) f ( x ) ||
−
|| ˆ T ⊤ ( x ) f ( x ) || 2 .
Similarly, the uncertainty for the PTM is given by σ ˆ W ( x ) =
|| ˆ W ⊤ ( x ) f ( x ) ||[-
-]
−
||
ˆ W ⊤ ( x ) f ( x ) || 2 .
Once the uncertainty has been established, we further integrate the two estimated transition matrices into a Kalman transition matrix, defined as W km ( x ) , via a weighted average operation.
Mathematically, the Kalman transition matrix is given by
[Equation]
where the Kalman gain λ ( x ) is carefully selected to minimize the estimation error.
Let the true
PTM be W ∗ ( x ) , where W ∗ ij ( x )
=
if y = i and ˜ y
=
j , otherwise 0 .
Subsequently, we define the reconstruction error for NTM and PTM as e T ( x ) =
W ∗ ( x ) − ˆ T ( x ) 2 and e W ( x ) =
W ∗ ( x ) − ˆ W ( x ) , respectively.
The parameter λ ( x ) is determined via minimizing the mean square reconstruction error.
We theoretically justify the superiority of IF for a general scenario.
Theorem 3.5 (Theoretical justification of IF) Let the correlation coefficient between reconstruction error e T ( x ) and e W ( x ) be cov ( x ) ∈ [ − 1 , 1] . The trace of the covariance matrix for error e ˆ T ( x ) and e ˆ W can be quantified by σ ˆ T ( x ) and σ ˆ W ( x ) , i.e., Tr ( e ˆ T ( x ) e ⊤ ˆ T ( x ))
=
σ ˆ T ( x ) and Tr ( e ˆ W ( x ) e ⊤ ˆ W ( x ))
=
σ ˆ W ( x ) .
For the Kalman transition matrix W km , let the reconstruction error be L mse [Equation]
x [Equation] − W [Equation]
x [Equation] ) ⊤ (cid:17) . Then the optimal Kalman gain to minimize the reconstruction error is given by
[Equation]
and the minimum reconstruction error satisfies L mse ( λ ∗ ( x )) ≤ min( σ ˆ T ( x ) , σ ˆ W ( x
[Equation]
Theorem 3.5 implies that the Kalman transition matrix W km ( x ) provably achieves lower estimation error for general e T ( x ) and e W ( x ) , either correlated or independent.
E XPERIMENTS
In this section, we empirically evaluate the effectiveness and stability of our IF approach on synthetic and real-world noisy datasets.
We aim to answer two questions as follows.
Q1 :
Compared with the state-of-the-art transition matrix based methods, can IF achieve higher accuracy?
Q2 :
Can IF lead to more accurate and robust transition matrix estimation and more stable training?
4. E XPERIMENTAL S ETTINGS
Datasets. We verify the superiority of IF on three manually corrupted datasets, i.e., F-MNIST,
SVHN, CIFAR-10, and one real-world noisy dataset Clothing1M.
The first three datasets contain clean data, and we manually corrupt the labels of the training datasets by following (Xia et al., 2020).
IDN- τ means that the controlled noise rate is τ .
All experiments on those datasets with synthetic instance-dependent label noise are repeated five times.
The real-world dataset Clothing1M has 1 M images with real-world noisy labels and 10 k images with clean labels for testing.
In the experiments, we leave out 10% of the noisy training samples as a noisy validation set for model selection.
Baselines.
We compare the proposed IF method with (i) CE, which trains a standard deep network with the cross-entropy loss on noisy datasets; (ii) DMI, a novel information-theoretic robust loss function for instance-independent label noise (Xu et al., 2019); (iii) Forward (Patrini et al., 2017),
Reweight (Liu & Tao, 2015), and T-Revision (Xia et al., 2019); (iv) part-dependent transition matrix (PTD)
(Xia et al., 2020); (v) instance-level forward correction (ILFC) (Berthon et al., 2021).
All these approaches utilize the NTM to correct the loss function.
Implementations. We choose standard neural networks and optimizers.
Specifically, we use a ResNet-18 network for F-MNIST, a ResNet-34 network for SVHN and CIFAR-10.
For the optimization, we first use SGD with 0 .
9 momentum, 10 − 4 weight decay, 128 batch size, 50 epochs and an initial learning rate of 10 − 2 to initialize the network.
Then, we adopt Adam optimizer and 5 × 10 − 7 learning rate to learn the NTM following PTD (Xia et al., 2020).
Once the NTM is obtained, we retrain the neural network.
In the forward propagation, the Kalman transition matrix is obtained and then adopted to correct the loss function.
In addition, the revision trick (Xia et al., 2019) is also adopted for our proposed IF method, which introduces the slack variables ∆ W for the estimated [Table] Kalman transition matrix.
Specifically, the slack variable ∆ W is initialized as all zero elements in the experiments and can be optimized during the training.
To guarantee valid transition matrices, We first project their negative entries of W km ( x ) + ∆ W to zero and then normalize the summation of each row to be 1 .
Following (Xia et al., 2020), we do not use any data augmentation technique in the experiments.
For Clothing1M, we use a ResNet-50 pre-trained model on ImageNet.
We do not use any clean data to learn the transition matrices and classifiers.
After the NTM ˆ T ( x ) is obtained according to (Xia et al., 2020), we use SGD with 0 .
9 momentum, 10 − 3 weight decay, 32 batch size, and run with 10 − 3 learning rate and 10 epochs.
For the evaluation, we adopt the test accuracy of the epoch with best accuracy on validation datasets.
4. E XPERIMENTAL R ESULTS
Accuracy on synthetic and real-world noisy datasets. To answer Q1 , Figure 3 shows the estimation errors of PTD, PTM (i.e., only adopt PTM in loss correction) and IF methods with noise rate IDN- 10% to IDN- 40% , where the estimation error is defined as the average difference of the transition probability err = 1 N (cid:80) Nn =1 (1 − ˆ W y n ˆ y n ) .
It can be observed that IF can achieve the lowest estimation error over all noise ratios 3 .
Table 1 reports the classification accuracy on the SVHN, CIFAR-10 and F-MNIST with noise rate IDN- 10% to IDN- 50% , where the highest accuracy is bold faced.
We can observe that IF achieves better performance than all baselines across all the three datasets and five noise rates.
For example, in CIFAR-10, the improvements over the strongest baseline [Table] is 1 .
22% , 9 . 89% for noise rate IDN- 10% to IDN- 50% , respectively.
The higher accuracy implies that the posterior information is more important for higher noise rates.
In addition, the methods IF-F and IF-F-V are comparable across the three datasets and noise rates, which demonstrates that the revision on the transition matrix is not significantly beneficial compared with the method PTD.
This observation validates the effectiveness of posterior information on transition matrix modification.
For the real-world dataset Clothing1M, IF outperforms the baselines as shown in Table 2.
Our IF method achieves 0 .
5% improvements over the strongest baseline early-learning regularization (ELR) (Liu et al., 2020).
Transition matrix estimation evaluation.
We study the transition matrix estimation error in the training data for IF and PTM to answer Q2 .
Figure 4 shows the training accuracy and estimation error of each epoch on the CIFAR10 dataset with noise rate IDN- 10% to IDN- 40% , respectively.
The shadow area represents the standard deviation.
For all IDN rates, IF can achieve higher training accuracy and lower estimation error compared with PTM.
Stability on synthetic noisy datasets. We study the training stability via inspecting the test accuracy over epochs to further answer Q2 .
Figure 5 shows the test accuracy with respect to epochs on the CIFAR10 dataset with noise rate IDN- 10% to IDN- 40% , respectively.
The shadow area represents the standard deviation of the accuracy on the same epoch across five experiments.
We can clearly observe that, on both low-level and high-level noise, IF shows higher accuracy in a more stable way since the posterior information of each instance provides reliable guidance on loss correction.
C ONCLUSIONS
This paper presents a novel notion of PTM and a simple yet effective learning paradigm named information fusion, which trains deep neural networks under instance-dependent label noise.
The main goal of PTM is to characterize the label noise via limited observed noisy labels, which can bridge the empirical noisy and posterior empirical clean distributions.
To effectively combine the benefits of NTM and PTM, we propose an information fusion approach to integrate both transition matrices to correct the loss function.
Experiments on synthetic and real-world noisy datasets show that IF can achieve higher accuracy and more stable training compared with state-of-the-art methods, especially for high noise rates.
In the future, we can extend to incorporate more prior knowledge of the transition matrix, e.g., sparsity or low rank, into the end-to-end learning algorithm.
A T HEORETICAL ANALYSIS A. P ROOF OF T HEOREM 3.
Based on the definition of posterior reweight loss
[Equation]
we have l p − rew (cid:16) ˜
y, f ( x ) (cid:17) =
(cid:80) cy
=1 P ( Y = y | ˜ Y =
˜
y, X = x ) l ( y, f ( x )) .
For noisy underlying distribution, the expected risk can be calculated as follows,
[Equation]
which implies that posterior reweight loss over noisy underlying distribution equals to cross entropy loss over clean underlying distribution and thus posterior reweight loss induces to statistically consistent classifier.
For the anchor point samples ( x ap , ˜ y ap ) with underlying clean probability P ( Y = y ap | X = x ap ) , based on equation (2), the PTM value satisfies W y ap , ˜ y ap ( x )
= 1 and W k, ˜ y
ap
= 0 for any k ̸
= y ap , which implies that empirical distribution mismatch on anchor points samples can be completely mitigated.
As for empirical distribution, the empirical risk satisfies
[Equation]
For noisy empirical distribution, the expected risk satisfies
[Equation]
(13) which implies that posterior reweight loss over noisy empiricial distribution equals to cross entropy loss over posterior clean empirical distribution.
A. P ROOF OF L EMMA 3.
Note that the loss function l ( y, f ) is convex with respect to f and (cid:80) ci =1 W i, ˜ y ( x ) =
(cid:80) ci =1 P
( Y = i | ˜ Y =
˜
y, X = x ) = 1 for any ˜ y and x , according to Jensen’s inequality, we have
[Equation]
which shows the relation between posterior reweight loss and posterior forward loss.
A. P ROOF OF T HEOREM 3.
Based on Lemma 3.2 and Theorem 3.
[Equation]
we have the inequality on the expected risk as follows,
[Equation]
Define the optimal classifier f ∗ to minimize expected risk for underlying clean distribution and cross entropy loss as
[Equation]
then R P ( X,Y ) ( f ∗ ) = 0 and for l p − rew (cid:0) ˜
y, f ∗ ( x ) (cid:1) =
for any x .
Note that the expected risk for posterior forward loss is non-negative and not larger than that for posterior reweight loss, we also have l p − fw (cid:0)
˜
y, f ∗ ( x ) (cid:1) =
for any x .
In addition, the difference of expected risk for two different classifiers satisfies
[Equation]
(16) which implies that the classifier f ∗ can also minimize expected risk for underlying noisy distribution and posterior reweight loss.
This completes the proof.
A. P ROOF OF T HEOREM 3.
Aiming to standardize the constraint, we vectorize W ( x ) as c 2 × 1 dimension w vec ( x ) :=
[ W 11 ( x ) , · · · , W c 1 ( x ) , · · · , W 1 c ( x ) , · · · , W cc ( x )] ⊤ and define the coefficient matrix A ( x ) as follows, A (
[Equation]
 c
[Equation]
(17) where P ⊤ ( ˜Y | x ) =
[ P ( ˜ Y = 1 | x ) , · · · , P ( ˜ Y =
c | x )] 1 × c .
Therefore, the constraint W ⊤ ( x )ˆ P ( ˜ Y | x ) = f
( x ) can be transformed as A ( x ) w vec ( x ) =
f
( x ) , which is a least-norm problem with undetermined equations constraint.
Note that matrix
A
( x ) is full column rank, we can prove that the least-norm solutions is
[Equation]
Considering any possible solution w vec ( x ) satisfying the constraint A ( x ) w vec ( x ) =
f
( x ) , then
A ( x )[ w vec ( x ) − w ∗ vec ( x )] = 0 and [ w
[Equation]
x
[Equation]
which implies that the vectors w vec ( x ) − w ∗ vec ( x ) and w ∗ vec ( x ) are perpendicular.
Thus, the Lnorm of w vec ( x ) satisfies
[Equation]
which shows that the equation (18) is the optimal solution.
Via direct calculation, it is easy to obtain that A ( x ) A ⊤ ( x ) = (cid:80) cj =
k ˆ P 2 ( ˜ Y
= k | x ) I c × c .
Therefore, the element of PTM satisfies
[Equation]
i.e., W ∗ ( x ) = f
( x )ˆ P (˜ Y | x ) ⊤ || ˆ P
( ˜ Y | x {+) || 
+}
and completes the proof.
A. P ROOF OF T HEOREM 3.
Considering the general reconstruction error relationship, the trace of the covariance matrix for reconstruction error satisfies
Tr ( e ˆ T ( x ) e ⊤ ˆ W ( x ))
=
cov ( x ) (cid:112) σ T ( x ) σ W ( x ) .
For Kalman transition matrix, the reconstruction deviation satisfies W ∗ ( x ) − W km ( x ) =
(1 − λ )
e T ( x ) + λe W ( x ) .
For simplicity, we remove the instance x to abbreviate the notations.
Therefore, the reconstruction error is given by
[Equation]
[Equation]
For such quadratic function
[Equation]
it is easy to obtain the optimal parameter via the local minimal
[Equation] [Equation]
and thus the optimal weight is given by λ ∗ =
[Equation]
Plugging in the optimal weight into the reconstruction error, we obtain the minimal reconstruction error is given by
[Equation]
Aiming to investigate the relation between minimal reconstruction error for IF, PTM and PTD, we adopt variable substitution t =
σ T + σ W − 2 cov √ σ T σ W ≥ 0 to further simplify minimal reconstruction error L mse ( λ ∗ ) , then we have
[Equation]
where the inequality holds according to arithmetic mean-geometric mean inequality.
Therefore, the linear combination can provably achieve lower estimation error for general e T ( x ) and e W ( x ) , either correlated or independent.
B R EPRODUCIBILITY B. D ATASET S TATISTICS
For fairly comparison with previous work, we performance the image classification task on the three manually corrupted datasets, i.e., F-MNIST, SVHN, CIFAR-10, and one real-world noisy dataset
Clothing1M.
They have been widely adopted to study label noise problem.
The detailed statistics are listed in Table 3.
[Table]
B. R UNNING E NVIRONMENT
All baselines and IF approaches are implemented in PyTorch, and tested on a machine with AMD
EPYC 7282 16-core processors, 4 GeForce GTX-3090 Ti GPUs with 24GB memory size.
B. A LGORITHMS
We summarize the IF algorithm to correct the loss and provide the pseudo codes in Algorithm 1.
For instance-dependent Label noise generation, we provide pseudo codes with controllable noise rate are provided in Algorithm 2.
This algorithm follows the state-of-the-art method (Xia et al., 2020; Zhu et al., 2021a), where the overall noise rate is τ .
Input
Output
1 Estimate the NTM ˆ T ( x ) according to (Xia et al., 2020) ; 2 while Validation accuracy has increases in the last t epochs do
Calculate the PTM ˆ W ( x ) based on Equation (8) ;
Obtain the uncertainty for noise and PTM ;
Calculate Kalman gain Kalman transition
Equations loss posterior forward loss
Kalman
; 7 end
B. L EARNING N OISE T RANSITION M ATRIX VIA D ECOMPOSITION
For the instance-dependent label noise, the NTM may be different for different samples.
Thus, the complexity of NTM estimation is O ( Nc 2 ) , which leads to ill-posed estimation problem.
(Xia et al., 2020) introduces a part-dependent decomposition on the NTM, i.e., the noise of an instance depends only on its parts.
Specifically, given L part-dependent transition matrix, e.g, P l ∈ [0 , 1] c × c , where j = 1 , · · · , L , the instance-dependent matrix can be approximated by a combination of partdependent transition matrix and the combination coefficients, defined as h ( x ) ∈ [0 , 1] L , depend on the feature vectors.
Then the instance-dependent transition matrix W ( x ) can be approximated as follows
[Equation]
where h j ( x ) represents the j -th elements of h ( x ) and satisfies the normalization condition
|| h ( x ) ||
= 1 for any features vector to maintain that the summation of column is 1 for NTM.
In this work, we follow (Xia et al., 2020) to learn the NTM in three steps: (i) learning combination coefficients h ( x ) via parts-based features representation (Lee & Seung, 1999; Tao et al., 2017); (ii) learning the instance-dependent transition matrix by exploiting anchor points (Liu & Tao, 2015; Xia et al., 2019); (iii) learning the part-dependent transition matrices via minimization of reconstruction error for instance-dependent transition matrix.
(Xia et al., 2020) justifies the part-dependent decomposition assumption via parts-based representations.
For example, (Hosseini-Asl et al., 2016) shows that the input features rely on parts-based representations in object recognition.
Thus, it is natural to approximate the label noise in the part level.
In display advertising, the delay feedback of the label (Ktena et al., 2019; Yasui et al., 2020) introduces label noise.
Earlier the data is generated, the less probability the feedback exists.
Thus, the label noise highly depends on part of features.
Since the instance-dependent NTM may be poorly learned, the slack variable trick (Xia et al., 2019; 2020) is also adopted to modify the instance-dependent transition matrix, which significantly improve the accuracy.
However, the slack variable trick ignores the posterior information for each instance, which limits the accuracy.
In addition, the variance of accuracy in (Xia et al., 2020) is large since there is not any guide information to modify the instance-dependent NTM except minimization of the corrected loss.
In this paper, we claim that the measurement information is useful to modify the NTM.
We will introduce how to estimate the PTM and information fusion in the next two subsections.
B. I MPLEMENTATION D ETAILS
Warm-up Training Strategy Aiming to estimate the transition matrix, we adopt warm-up training strategy that training the neural network via cross entropy loss with noisy data.
we use SGD as the optimizer with 0 .
9 momentum, 128 batch size, and 10 epochs.
With the benefit of memorization effect (Arpit et al., 2017), warm-up training with appropriate epochs can guarantee that the predicted probability of neural network is reliable and can approximate underlying clean probability.
Training with Corrected Loss Our proposed PTM estimation relies on the neural network predicted probability.
Although the warm-up training can guarantee the prediction accuracy for clean instances with the benefits of memorization effect, the prediction of the PTM estimation may still be not reliable.
To ease the dilemma, we adopt sample selection to choose the reliable prediction and thus guarantee the reliable PTM estimation for specific instance.
Similar to (Cheng et al., 2021a), the small-loss criteria is adopted to select the instances with small training loss for training.
Specifically, we select the instance via the weighted loss if the instance loss is less than the average loss over all classes.
For the instance satisfying the small loss criterion, we have confidence that the instance is clean, which means the PTM is identity.
To put it simply, we sum the estimated PTM and identity and then adopt column normalization.
It is worth noting that the noisy labels are adopted in forward propagation, which is highly different with previous transition matrix-based works (Xia et al., 2019;
Liu & Tao, 2015; Patrini et al., 2017; Xia et al., 2020).
Compared with hard sample selection based methods (Cheng et al., 2021a), our method achieves the soft sample selection and adaptively correct the loss for “clean" instance.
C A DDITIONAL E XPERIMENTAL R ESULTS C. S TABILITY ON S YNTHETIC N OISY D ATASETS
We study the training stability via inspecting the test accuracy over epochs.
Figures 6 and 7 show the test accuracy with respect to epochs on SVHN and F-MNIST datasets with noise rate IDN- 10% to IDN- 40% .
It is seen that our proposed method IF can achieve better accuracy and stable training compare with baseline PTD, which implies that the PTM fusion can decrease the transition matrix estimation error and improve the training stability.
T-Revision trick (-V method) proposed in (Xia et al., 2019) aims to compensate the NTM and thus improve the accuracy performance.
It is seen that
PTD-F-V can achieve better performance than PTD-F, which verifies that the learned NTM is not very accurate.
For our proposed method, IF-V can achieve comparable performance than IF-F and demonstrates that transition matrix revision trick can not improve the performance since PTM has already been employed to compensate the transition matrix error.
Compared with different noise rates, it is seen that the accuracy of PTD method may decrease with respect to epochs, especially for large noise rate.
Under large noise rate, the number of clean samples [Figure] [Figure] is limited and the neural network may already overfit noisy samples during warm-up training and thus hurt the transition matrix estimation.
In this scenario, PTD method does not have the capability to adjust such poor NTM estimation and thus leads to the accuracy drop during training.
Our proposed method IF and T-Revision trick both adjust the estimated transition matrix and mitigate the accuracy drop.
C. E XPERIMENTAL R ESULTS ON H IGH N OISE R ATIO
We investigate the performance of IF for a high noise ratio compared with PTM and PTD.
Table.shows the test accuracy for high noise ratio IDN- 70% , IDN- 80% and IDN- 90% in CIFAR 10 dataset.
We have two observations: (1) Comparing with PTD at the noise ratios of IDN- 80% and IDN- 90% , our PTM could still outperform PTD.
Although the network output is not perfect, the performance deterioration of PTD is still higher than PTM.
(2) Our IF consistently delivers the most superior performance no matter under the low or high noise.
That is because IF adaptively and linearly combines NTM and PTM in instance level and thus achieves more accurate estimation.
[Table]
E XPERIMENTAL R ESULTS ON M ORE B ASELINES
Although many transition matrix-based methods are compared in Table.
[Table] we are also interested in several more baselines, including label smoothing (LS) (Szegedy et al., 2016), early-learning regularization (ELR) (Liu et al., 2020), (Berthon et al., 2021), progressive label correction (PLC) (Zhang et al., 2021a), confidence regularized sample sieve (CORES)
(Cheng et al., 2021a).
It could be observed that our method can still achieve state-of-the-art performance compared with all other baselines.
Table.
5 demonstrates the test accuracy over IDN- 10% to IDN- 50% label noise ratio in CIFAR 10.
It could be observed that our method can still achieve state-of-the-art performance, based on which we can confidently validate the effectiveness of our method.
[Table]
D R ELATED WORK
Label Noise Model There are three types of label noise model, including the random classification noise (RCN) model (Manwani & Sastry, 2013; Natarajan et al., 2013), the class-conditional label noise (CCN) model (Patrini et al., 2017; Xia et al., 2019; Zhang & Sabuncu, 2018; Liu & Guo, 2020) and the e instance-dependent label noise (IDN) model (Cheng et al., 2020; Xia et al., 2020; Cheng et al., 2021a; Berthon et al., 2021; ? ).
Specifically, RCN means that each label is flipped independently with a constant probability, CCN assumes that the flip probability (noise rates) are the same for the instance with the same clean labels.
IDN is the most general case, where the flip probability depends on its instance.
Loss Correction via Transition Matrix
The transition matrix bridges the gap between the model predicted probability for noisy and clean data, which can achieve risk-consistent classifier with label noise.
The transition matrix can be estimated via cross-validation method (Natarajan et al., 2013), anchor points assumption (Xia et al., 2019; Yu et al., 2018; Yao et al., 2020).
Subsequently, the estimated transition matrix can be adopted to correct the loss function via forward correction (Patrini et al., 2017), backward correction (Patrini et al., 2017) and reweight (Xia et al., 2019; Liu & Tao, 2015).
Loss Correction via Sample Selection Due to the experiments observation– the neural network learns clean instances first (Arpit et al., 2017) and memorizes the instances gradually, there are lots of methods attempting to identify mislabeled training examples and then filter them out (Brodley &
Friedl, 1999; Jiang et al., 2018; Han et al., 2018b; Yu et al., 2019; Angelova et al., 2005; Huang et al., 2019; Li et al., 2019).
The core idea is to select the “small loss" samples as clean one.