R ESOLVING T RAINING B IASES VIA I NFLUENCEBASED D ATA R ELABELING
A BSTRACT
The performance of supervised learning methods easily suffers from the training bias issue caused by train-test distribution mismatch or label noise.
Inﬂuence function is a technique that estimates the impacts of a training sample on the model’s predictions.
Recent studies on data resampling have employed inﬂuence functions to identify harmful training samples that will degrade model’s test performance.
They have shown that discarding or downweighting the identiﬁed harmful training samples is an effective way to resolve training biases.
In this work, we move one step forward and propose an inﬂuence-based relabeling framework named
RDIA for reusing harmful training samples toward better model performance.
To achieve this, we use inﬂuence functions to estimate how relabeling a training sample would affect model’s test performance and further develop a novel relabeling function R .
We theoretically prove that applying R to relabel harmful training samples allows the model to achieve lower test loss than simply discarding them for any classiﬁcation tasks using cross-entropy loss.
Extensive experiments on ten real-world datasets demonstrate RDIA outperforms the state-of-the-art data resampling methods and improves model’s robustness against label noise.
I NTRODUCTION
Training data plays an inevitably important role in delivering the model’s ﬁnal performance.
It has been well recognized that the training bias issue will compromise model performance to a large extent (Arpit et al., 2017).
Speciﬁcally, there are two major scenarios where training biases show up.
The ﬁrst and most common scenario is that training samples involve corrupted labels that could be originated at possibly every step of the data lifecycle (Anderson & McGrew, 2017;
Dolatshah et al., 2018; Pei et al., 2020; Yu & Qin, 2020).
The second scenario is that the training and test sets are sampled from the respective distributions P train ( x, y ) and P test ( x, y ) , but P train is different from
P test (Guo et al., 2020; He & Garcia, 2009; Zou et al., 2019).
Both corrupted labels and distribution mismatch will hurt the generalization ability of a trained model (Fang et al., 2020; Zhang et al., 2017; Chen et al., 2021).
We generally refer to training samples with corrupted labels or those inducing distribution mismatch as harmful samples.
Data resampling is a widely used strategy to deal with harmful training samples.
Existing resampling approaches (Chawla et al., 2002; Mahmoody et al., 2016; Malach & Shalev-Shwartz, 2017;
Ren et al., 2018)
propose to assign different weights to training samples, which aim to mitigate the negative impacts of harmful samples on model’s generalization ability.
Among them, most resampling approaches (Arazo et al., 2019; Han et al., 2018; Li et al., 2020; Wang et al., 2020a) rely on training loss to identify harmful samples from the whole training set.
They follow the insight that the samples with higher training losses are very likely to have corrupted labels, and hence it is often beneﬁcial to downweight them during the process of model training.
However, such loss-based resampling methods have two limitations.
First, they are only able to deal with the training biases caused by training samples with corrupted labels (aka noisy samples).
Second, the small-loss trick typically holds true for deep models but not for any predictive models (Zhang et al., 2017).
To address the limitations, one recent work (Wang et al., 2020b) proposes a new resampling scheme based on inﬂuence functions (Cook & Weisberg, 1980).
The idea is to estimate the inﬂuence of each training sample on model’s predictions over the test set.
Any training samples that would cause an increase in the test loss are considered as harmful and will be downweighted afterwards.
It is worth mentioning that the inﬂuence functions have been proved to deal with two forms of training biases effectively, and it is agnostic to a speciﬁc model or data type (Koh & Liang, 2017; Koh et al., 2019).
Inspired by the success of inﬂuence-based data resampling, in this paper, we would like to ask the following question: what would happen if we relabel harmful training data based on inﬂuence analysis results ?
Our motivations on performing data relabeling via inﬂuence analysis are twofold.
(i ) Relabeling noisy samples is able to prevent the model from memorizing the corrupted labels.
(ii)
Relabeling clean but biased samples is helpful to improve model’s robustness to harmful samples.
Despite the potential beneﬁts of data relabeling, it is still challenging to develop an inﬂuence-based relabeling approach that has a theoretical guarantee on the model’s performance improvement after training with relabeled data.
To answer the question, we ﬁrst follow (Koh et al., 2019) to measure the inﬂuence of each training sample on model’s predictions and identify the harmful training samples which would cause an increase in the test loss.
Next, we investigate whether relabeling the identiﬁed harmful samples rather than discarding them can improve the test performance.
To achieve this, we start from binary classiﬁcation tasks where relabeling a training sample is to convert its binary label from y to 1 − y .
We theoretically prove that relabeling harmful training data via inﬂuence analysis can achieve lower test loss than simply discarding them for binary classiﬁcation.
Furthermore, we design a novel relabeling function R for multi-class classiﬁcation tasks and prove that the advantage of relabeling the identiﬁed harmful samples using R in reducing model’s test loss still holds.
Following the inﬂuence-based resampling approaches (Wang et al., 2018; Ting & Brochu, 2018; Ren et al., 2020;
Wang et al., 2020b), we only use the test loss for theoretical analysis and empirically calculate inﬂuence function with a small but unbiased validation set by assuming the validation set is sampled from the same distribution as the test set.
In this way, using the validation loss to calculate the inﬂuence function is an unbiased estimation of the true inﬂuence function.
Otherwise, the problem may lie in the category of transfer learning which is beyond the scope of this work.
To summarize, this work makes the following contributions.
First , we propose to combine inﬂuence functions with data relabeling for reducing training biases and we develop an end-to-end inﬂuencebased relabeling framework named RDIA that reuses harmful training samples toward better model performance.
Second , we design a novel relabeling function R and theoretically prove that applying R over harmful training samples identiﬁed by inﬂuence functions is able to achieve lower test loss for any classiﬁcation tasks using cross-entropy loss function.
Third , we conduct extensive experiments on real datasets in different domains.
The results demonstrate that (i) RDIA is effective in reusing harmful training samples towards higher model performance, surpassing the existing inﬂuence-based resampling approaches, and (ii) RDIA improves model’s robustness to label noise, outperforming the current resampling methods by large margins.
B ACKGROUND
Let D =
( x i , y i ) ∈
X × Y | 1 ≤ i ≤ N } be the training set which are sampled from
P train ( x, y ) .
Let z i =
( x i , y i )
where x i ∈ R d and y i ∈ R K .
Let ϕ ( x, θ ) be a model’s prediction for x , where θ ∈ R p is the parameter set of the model.
We denote the loss of
sample z i by l ( z i , θ ) =
L ( y i , ϕ ( x i , θ )) and use l i ( θ ) to represent l ( z i , θ ) .
We consider the standard empirical risk minimization (ERM) as the optimization objective.
Formally, the empirical risk over D is deﬁned as: L ( D ; θ ) =
1 N (cid:80) Ni =
1 l i ( θ ) .
Since our relabeling function is dependent to the loss function, we focus on the most effective and versatile loss, i.e., Cross Entropy loss for any classiﬁcation tasks.
Inﬂuence functions.
Inﬂuence functions, stemming from Robust Statistics (Huber, 2004), have provided an efﬁcient way to estimate how a small perturbation of a training sample would change the model’s predictions (Koh & Liang, 2017;
Koh et al., 2019; Yu et al., 2020).
Let ˆ θ = arg min θ 1 N (cid:80) Nn =1 l n ( θ ) be the optimal model parameters on convergence.
When upweighting a training sample z i on its loss term by an inﬁnitesimal step (cid:15) i , we obtain the new optimal parameters ˆ θ (cid:15) i on convergence as : ˆ θ (cid:15) i =
arg min θ 1 N (cid:80)
Nn
= 1 l n ( θ i )+ (cid:15)
i l i ( θ ) .
Based on inﬂuence functions (Cook & Weisberg, 1980; Koh & Liang, 2017), we have the following closed-form expression to estimate the change in model parameters when upweighting z i by (cid:15) i :
[Equation]
where H ˆ θ (cid:44) 1 N (cid:80) Nn =1 (cid:79) 2 θ l n (ˆ θ ) is the Hessian matrix and (cid:79)
2 θ l n ( θ ) is the second derivative of the loss at training point z n with respect to θ .
Using the chain rule, we can estimate the change of model’s prediction at a test data z cj sampled from the given test distribution P test (Koh & Liang, 2017) :
[Equation]
At a ﬁne-grained level, we can measure the inﬂuence of perturbing training sample z i from ( x i , y i ) to ( x i , y i + δ ) .
Let z iδ =
( x i , y i + δ ) and the new loss l i ( z iδ , θ )
=
L ( y i + δ, ϕ ( x i , θ )) .
According to (Koh & Liang, 2017), the optimal parameters ˆ θ (cid:15) i δ i after performing perturbation on z i is ˆ θ (cid:15) i δ i
= arg
[Equation]
θ ) .
This allows us to estimate the change in model parameters after the ﬁne-grained data perturbation using inﬂuence functions as:
[Equation]
Further, the inﬂuence of perturbing z i by z iδ on model’s prediction at test sample z cj is the following:
[Equation]
It is important to notice that Eq.
(4) holds for arbitrary δ when (cid:15) i is approaching 0 .
This provides the feasibility of measuring how relabeling a training samples could inﬂuence the model’s predictions.
Inﬂuence-based resampling approaches.
Previous researches (Koh & Liang, 2017; Wang et al., 2020b) have shown that inﬂuence functions have strong ability to identify harmful samples from the whole training set, which is agnostic to the speciﬁc model or data structure.
Inspired by this, many inﬂuence-based resampling approaches (Ting & Brochu, 2018; Wang et al., 2018; 2020b) proposed to discard or downweight the identiﬁed harmful samples to reduce the test loss.
However, different from previous works which focus on estimating the inﬂuence of each training sample on the test performance using Eq.
(1)-(2), we perform the ﬁne-grained perturbation on a training sample’s label and evaluate its inﬂuence using Eq.
Further, our work tries to build an end-to-end inﬂuence-based relabeling framework to reuse the harmful samples with a theoretical guarantee on the ﬁnal model performance for any classiﬁcation tasks.
To be speciﬁc, we demonstrate that harmful training instances after being relabeled properly do make contributions to improve the ﬁnal model performance , which provides a novel viewpoint on handling biased training data .
M ETHODOLOGY
Assume we have Q = { ( x cj
, y cj ) ∈
X × Y | 1 ≤ j ≤ M } sampled from the test distribution P test and our objective is to minimize the test risk L ( Q ; θ )
=
1 M (cid:80) Mj
=1 l cj ( θ ) .
Due to the harmful training samples in D , the optimal ˆ θ which minimizes the empirical risk over training set D may not be the best risk minimizer over Q .
To solve this issue, we propose a novel data relabeling framework named RDIA which aims to identify and reuse harmful training instances towards better model performance.
We design a relabeling function R that allows the model to achieve lower test risk after being trained with the relabeled harmful instances for any classiﬁcation tasks.
In what follows, we ﬁrst give an overview of the RDIA framework.
Then we describe the details of the major steps in RDIA and provide theoretical analysis on how the relabeled harmful samples are useful to further reduce the test risk.
The algorithm of RDIA could be found in Appendix A
3. O VERVIEW OF RDIA
Figure 1 provides an overview of RDIA, which consists of four major steps: Model training , Harmful samples identiﬁcation , Relabeling harmful samples via inﬂuence analysis and Model retraining .
Training set D
Validation set 𝑫 + 𝑫 —
R(·)
Relabeling function 𝑫 —′
Step Ⅲ
Step Ⅳ
Training set ෡𝑫
Step Ⅱ Step Ⅰ
M od e l
R e t r a i n i ng
M od e l
T r a i n i ng
I n f l u e n ce f un c ti on 𝑫 +
Step I: Model training is to train a model based on the training set D until convergence and get the model parameters ˆ θ .
Step II: Harmful samples identiﬁcation is to compute the inﬂuence of perturbing the loss term of each training sample
z i ∈ D on test risk using Eq.
(2) and then use it to identify the harmful training samples from D .
We denote the set of identiﬁed harmful training samples as D − and the set of remaining training instances as D + .
The details of this step are provided in Section 3.2.
Step III: Relabeling harmful samples via inﬂuence analysis is to apply the relabeling function to modify the label of each identiﬁed harmful training sample in D − and obtain the set of relabeled harmful training samples denoted as D (cid:48)− .
We introduce our relabeling function R and theoretically prove that updating the model’s parameters with new training set ˆ D = D (cid:48)− ∪ D
+ can achieve lower test risk over Q than simply discarding or downweighting D − in Section 3.3.
Step IV :
Model retraining is to retrain the model using ˆ D till convergence to get the ﬁnal optimal parameters ˆ θ (cid:15) R .
3. H ARMFUL SAMPLES IDENTIFICATION
In the second step, we compute D − ⊆
D which contains harmful training samples from the original training set D .
Intuitively, a training sample is harmful to the model performance if removing it from the training set would reduce the test risk over Q .
Based on inﬂuence functions, we can measure one sample’s inﬂuence on test risk without prohibitive leave-one-out training.
According to Eq.
(1)-(2), if we add a small perturbation (cid:15) i on the loss term of z i to change its weight, the change of test loss at a test sample z cj ∈ Q can be estimated as follows:
[Equation]
where Φ θ ( · , · ) is computed by Eq.
We then estimate the inﬂuence of perturbing z i on the whole test risk as follows:
[Equation]
Henceforth, we denote by Φ θ ( z i ) =
(cid:80) Mj
=
Φ θ ( z i , z cj ) the inﬂuence of perturbing the loss term of z i on the test risk over Q .
It is worth mentioning that given (cid:15) i ∈
[ − 1 N , 0) , Eq.
(6) computes the inﬂuence of downweighting or discarding the training sample
z i .
We denote D − =
{ z i ∈ D | Φ θ ( z i ) > 0 } as harmful training samples.
Similar to (Wang et al., 2020b), we assume that each training sample inﬂuences the test risk independently.
We derive the Lemma 1.
Lemma 1.
Discarding or downweighting the training samples in D − = { z i ∈ D | Φ θ ( z i ) > 0 } from D could lead to a model with lower test risk over Q :
[Equation]
where ˆ θ (cid:15) denotes the optimal model parameters obtained by updating the model’s parameters with discarding or downweighting samples in D − .
Lemma 1 explains why inﬂuence-based resampling approaches have strong ability to resolve training biases and the proof of Lemma 1 is provided in Appendix B.
In practice, to further tolerate the estimation error in Φ θ ( z i ) which may result in the wrong identiﬁcation of harmful training samples, we select D − =
{ z i ∈ D | Φ θ ( z i ) > α } where the hyperparameter α controls the proportion of harmful samples to be relabeled eventually.
We conduct the experiment to show the effects of α and the validation set in Section 5.3.
3. R ELABELING HARMFUL SAMPLES VIA INFLUENCE ANALYSIS
In the third step, we propose a relabeling function R and ensure the test risk would be reduced after training with the relabeled harmful samples D (cid:48)− .
To achieve this, we start from a special case (i.e., binary classiﬁcation) and then extend to any classiﬁcation tasks.
Relabeling harmful instances on binary classiﬁcation.
We start with binary classiﬁcation where the relabeling function R is straightforward.
Since the label set Y is { 0 , 1 } , we have R ( z ) = 1 − y for any z = ( x, y ) ∈ D .
Recall that ϕ
( x i , θ ) denotes the model output for x i and the training loss of z i =
( x i , y i )
∈ D is: l i ( θ ) = − y
i log( ϕ ( x i , θ ))
− (1 − y i )
log(1 − ϕ ( x i , θ ))
To compute the inﬂuence of relabeling a training sample z i in D , we ﬁrst consider the case that y i = 1 and R ( z i )
= 0 .
The loss l i ( θ ) at z i is changed from − log( ϕ ( x i , θ )) to − log(1 − ϕ
( x i , θ )) .
Letting z iR
=
( x i , 1 − y i ) and w ( z i , θ ) =
(cid:79) θ l
i ( z iR , θ ) − (cid:79) θ l i ( θ ) , we have:
[Equation]
According to Eq.
(2),(4) and (8), the inﬂuence of relabeling z i on model’s prediction at test sample z cj is:
[Equation]
Similarly, when the label y i in z i is 0 and R ( z i )
= 1 , we can derive the inﬂuence of relabeling z
i at z cj as η θR ( z i , z cj ) =
− Φ θ ( z i ,z cj ) ϕ
( x i , ˆ θ ) .
Let ˆ θ (cid:15)
i R i denote the optimal parameters after relabeling z i .
Similar to Eq.
(6), we could extend the inﬂuence of relabeling z
i at z cj to the whole test risk over Q as:
[Equation]
According to Eq.
(9), the inﬂuence of relabeling training samples η θR ( z i , z cj ) is related to the inﬂuence of perturbing the loss term of training samples, i.e., Φ θ ( z i , z cj ) .
In this way, the change of test risk by relabeling z i
(Eq.
(10)) and that by perturbing z i (Eq.
(6)) are interrelated.
Then we derive the Theorem 1 and the proof can be found in Appendix B.
Theorem 1.
In binary classiﬁcation, let σ be the inﬁmum of ϕ ( x i , ˆ θ ) 1 − ϕ
( x i , ˆ θ ) and 1 − ϕ ( x i , ˆ θ ) ϕ
( x i , ˆ θ ) and D − = { z i ∈ D | Φ θ ( z i ) > 0 } .
Relabeling the samples in D − can achieve lower test risk than discarding or downweighting them from D , because the following inequality holds.
[Equation]
Theorem 1 shows that relabeling the samples in D − could achieve lower test risk than simply discarding or downweighting D − from the training set for binary classiﬁcation tasks.
We then provide some intuitions on the beneﬁts of relabeling harmful training samples.
In the context of binary classiﬁcation, if a training sample z in D − is noisy, our relabeling method corrects the label noise and improve training data quality; otherwise, z is very likely to be biased due to its negative impact on the test risk, and relabeling it might improve model’s robustness.
Relabeling harmful instances on any classiﬁcation tasks.
We now introduce a relabeling function
R that can be used for any classiﬁcation tasks.
For a classiﬁcation problem with K class labels ( K ≥ 2 ), we represent each label y as a K -dimensional one-hot vector.
The CE loss at z
i is: l i ( θ ) =
− (cid:80)
Kk
=1 y i,k log( ϕ k ( x i , θ )) .
Intuitively, the proposed relabeling function R should satisfy the following principles:
• Consistency : R should produce a K -dimensional label vector:
R ( x i , y i ) =
y (cid:48)
i ∈ [0 , 1] K .
• Effectiveness : applying R over harmful training samples D − should guarantee the resultant test risk is no larger than the one achieved by simply discarding them.
For the consistency principle, we require the new label y (cid:48)
i to be K -dimensional where y (cid:48)
ik describes the likelihood that x i takes the k -th class label, k ∈ [1 , K ] .
Here we do not require (cid:80) Kk =1 y (cid:48)
ik to be one, because we focus on leveraging the identiﬁed harmful training samples towards better model performance instead of ﬁnding their truth labels.
Consider a training sample
z i =
( x i , y i ) belonging to the m -th class ( m ∈ [1 , K ] ), i.e., y i,m = 1 .
Let R ( x i , y i ) =
y (cid:48)
i , we propose the following relabeling function R that fulﬁlls the above two principles:
[Equation]
where ϕ ( x i , ˆ θ ) =
( ϕ 1 , · · · , ϕ K ) is the probability distribution over K classes produced by the model with parameters ˆ θ , i.e., ϕ i ∈ [0 , 1] and (cid:80) Ki =
1 ϕ i = 1 .
It is easy to check that our proposed relabeling function R in Eq.
(12) satisﬁes the ﬁrst principle.
Interestingly, we can verify that for
K = 2 , we have R ( z i ) =
1 − y i .
We further prove the effectiveness of R using the Lemma 2.
Lemma 2.
When applying the relabeling function R in Eq.
(12) over a training sample z i ∈ D with a class label m , the CE loss l i ( θ ) at z i is changed from − log( ϕ m ( x i , θ )) to − log(1 − ϕ
m
( x i , θ )) .
It is interesting to verify the change in loss l i ( θ ) acts as an extension of the binary classiﬁcation.
Similar to Theorem 1, we can drive the following theorem using Eq.
Theorem 2.
In multi-class classiﬁcation, let ϕ y
( x i , ˆ θ ) denote the probability that z i is classiﬁed as its truth class label by the model with the optimal parameters ˆ θ on D , and σ be the inﬁmum of ϕ y ( x i , ˆ θ ) 1 − ϕ y
( x i , ˆ θ ) .
Relabeling the samples in D − = { z i ∈ D | Φ θ ( z i ) > 0 } with R leads to a test risk lower than the one achieved by discarding or downweighting D − .
Formally, we have:
[Equation]
Theorem 2 shows that using our proposed relabeling R can further reduce the test risk than simply discarding or downweighting D − from the training set for any classiﬁcation tasks.
The detailed proofs of Lemma 2 and Theorem 2 are provided in Appendix B.
D ISCUSSIONS
In this section, we provide numerical analysis on the superior performance of RDIA against other inﬂuence-based resampling approaches (Wang et al., 2018; 2020b).
Then we discuss the extension of RDIA in exploiting training loss information.
Numerical analysis.
Consider a training point z i ∈ D − belonging to class m , where D − is speciﬁed in Section 3.2.
According to Eq.
(13), if we use R to assign z i with a new label y (cid:48) i
=
R ( z i ) instead of discarding or downweighting z i , the difference in the test risk over Q can be computed as:
[Equation]
z i ∈ D − means the Φ θ ( z i ) in Eq.
(14) is positive, and hence we have g ( z i ) < 0 .
If the model’s prediction for z i with the optimal parameters ˆ θ is correct, ϕ m ( x i , ˆ θ ) is the largest component in the vector ϕ
( x i , ˆ θ ) .
We consider z i as a more harmful sample because it has negative inﬂuence on the test loss yet the model has learnt some features from z i that connects x i to class m .
In practice, z i is very likely to be a noisy or biased training sample.
Interestingly, from Eq.
(14), we can see that a small increase in ϕ m ( x i , ˆ θ ) will lead to a rapid increase in g ( z i ) .
This indicates relabeling such more harmful training samples leads to signiﬁcant performance gain.
Extension of RDIA.
In practice, due to the complexity of calculating the inﬂuence functions, identifying harmful samples via inﬂuence analysis could incur high computational cost, especially when training complex models like deep neural networks.
To address the problems, we further extend
RDIA by using training loss to identify harmful samples for deep models, and we dub this extension as RDIA-LS.
We empirically show that RDIA-LS is effective and efﬁcient to handle training data with corrupted labels for deep learning, which spotlights the great scalability of our approach.
The details of RDIA-LS are provided in Appendix F.
E XPERIMENTS
In this section, we conduct experiments to evaluate the effectiveness and robustness of our RDIA.
We also perform the ablation study to show how hyperparameter α and the size of validation set affect the performance of RDIA.
The visualization of identiﬁed harmful samples and comparison with other loss-based approaches are provided in Appendix D and Appendix G.
5. E XPERIMENTAL S ETTINGS
Datasets.
To verify the effectiveness of RDIA, we perform extensive experiments on ten public data sets from different domains, including NLP, CV, CTR, etc.
Since all the datasets are clean, we build a noise transition matrix P
= { P ij } K × K to verify the robustness of our proposed approaches for combating noisy labels, where K denotes the class number and P ij denotes the probability of a clean label i being ﬂipped to a noisy label j .
In our experiment, we use the noise ratio τ to determine the rate of how many labels are manually corrupted and each clean label has the same probability of being ﬂipped to other classes, i.e., P ij = τK − 1 .
More details about the statistics of the datasets and
Tr-Va-Te divisions are provided in Appendix C.
Comparison methods.
We compared our proposed relabeling method RDIA with the following baselines, all of which are agnostic of speciﬁc model or data structure.
(1) ERM : it means training a model with all the training data with the cross-entropy loss.
(2) Random : it is a basic relabeling method that randomly selects and changes the label of training samples.
(3) OptLR (Ting &
Brochu, 2018): it is a weighted sampling method which assigns each training sample with a weight proportional to its impact on the change in model’s parameters ψ θ .
Speciﬁcally, the weight of z i is max { α, min { 1 , λψ θ ( z i ) }} .
We set α and λ to be 1 / max { ψ θ ( z i ) } and 1 / max { Φ θ ( z i ) } , respectively.
(4) Dropout (Wang et al., 2018):
it is an unweighted subsampling method which simply discards D − from the training set, i.e., removing all training data with negative inﬂuence on the test loss.
(5) UIDS (Wang et al., 2020b): it it is an unweighted subsampling method which uses Linear sampling method or Sigmoid sampling method to resample the training data based on inﬂuence functions Φ θ ( z i ) .
It is the best-performing method among all the existing inﬂuence-based methods.
We implemented all the comparison methods by using their published source codes in Pytorch and ran all the experiments on a server with 2 Intel Xeon 1.7GHz CPUs, 128 GB of RAM and a single
NVIDIA
Ti GPU.
All the baselines are tuned with clean validation data for best model performance.
To measure the performance of all the approaches, we followed (Wang et al., 2020b) and used the test loss as the metric since we aim to optimize the test loss via inﬂuence analysis.
Implementation details.
For each of the ten datasets, we adopted logistic regression (convex optimization) as the binary classiﬁer (for MNIST and CIFAR10, we randomly choose two classes to perform binary classiﬁcation).
As for multi-class classiﬁcation, we implemented two deep models (non-convex optimization), LeNet (2 convolutional layers and 1 fully connected layers) and a CNN with 6 convolutional layers followed by 2 fully connected layers used in (Wang et al., 2019) on
MNIST and CIFAR10.
The hyperparameter α is also tuned in [0, 0.001, 0.002, ...,0.01] with the clean validation set for best performance.
More detailed settings are provided in Appendix C.
5. E XPERIMENTAL RESULTS
Effectiveness of RDIA.
To verify the effectiveness of RDIA , we conduct experiments on 10 clean datasets with three different models.
The experiments are repeated 5 times and the averaged test loss with standard deviation results are reported in Table 1 and Table 2.We have the following important observations.
First, our proposed RDIA yields the lowest test loss over 9 out of 10 datasets using logistic regression.
It outperforms ERM on all the datasets, which indicates the effectiveness of relabeling
[Table] [Table]
training samples via inﬂuence functions to resolve training biases.
Furthermore, RDIA outperforms the state-of-the-art resampling method UIDS on all the datasets except Avazu, which indicates the effectiveness of reusing harmful training samples via relabeling towards higher model performance.
Second, when training deep models, RDIA achieves the best test loss on MNIST+LeNet,
MNIST+CNN, and CIFAR10+CNN, where it outperforms UIDS by a large margin.
We observe
LeNet performs much worse than CNN on CIFAR10 using the original training set (i.e., the results of ERM) due to its simple architecture.
Note that the poor classiﬁcation results for clean and unbiased training data would interfere the identiﬁcation of true harmful training samples.
Hence,
RDIA performs similarly to Random which introduces random noises into the training set and the performance suffers.
But we want to emphasize that when training a more suitable model (e.g.,
CNN) on CIFAR10, RDIA is more effective to improve model’s performance.
Third, Random performs worse than ERM on all the cases except on Adult.
This indicates that randomly relabeling harmful training samples may easily inject noisy data that hurt model’s performance signiﬁcantly.
In contrast, our proposed relabeling function is effective to assign appropriate labels to harmful training samples that beneﬁt the test loss.
 
       1RLVHUDWLR       7 H V W  / R VV
E 5 M 5DQGRP8,'6'URSRXW5',$ (a) Breast-cancer  
       1RLVHUDWLR       
 7 H V
W  / R VV
E 5 M 5DQGRP8,'6'URSRXW5',$ (b) News  
       1RLVHUDWLR       7 H V W  / R VV
E 5 M 5DQGRP8,'6'URSRXW5',$ (c) Real-sim  
       1RLVHUDWLR   
      7 H V
[Figure]
Robustness to label noises.
In order to investigate the robustness of RDIA to noise labels, we set the noise ratio τ from 0 to 0 .
8 to manually corrupt the labels in the four datasets from different domains, while the results on the other datasets have similar trends.
Figure 2 reports the average test loss of all the inﬂuence-based approaches on four noisy datasets with different noise ratios.
First, thanks to the high accuracy of estimating the inﬂuence functions on logistic regression, all inﬂuence-based approaches consistently outperform ERM, which indicates the effectiveness of using inﬂuence functions to identify noisy samples.
Figure 2(a), 2(b) and 2(c) show that RDIA performs signiﬁcantly better than the other inﬂuence-based approaches.
As the noise ratio becomes larger, the test loss of all the other approaches increases while the test loss reported by RDIA is generally unchanged.
This veriﬁes the robustness of RDIAto high noise ratios.
We surprisingly ﬁnd that RDIA at 0.8 noise ratio achieves lower test loss than ERM at zero noise ratio.
The reason might be that RDIA could leverage all the training samples and ﬁx noisy labels properly to boost the per-
[Table]
[Table]
formance.
Second, Figure 2(d) shows that when combating noisy labels for deep models, RDIA still suffers from the noisy labels like other baselines because the estimation of inﬂuence functions with deep models is not accurate enough to ﬁlter out all noisy labels.
However, RDIA could still relabel the most negative samples to reduce the test loss
[Table]
5. A BLATION STUDY
Finally, we investigate the effect of different values of hyperparameters α and size of validation set on the performance of RDIA using MNIST with logistic regression.
Hyparameter α .
As discussed in Section 3.3, by varying α , we can derive the percentage of relabeled training data against the complete training set in RDIA.
Table 3 provides the results of how many samples are relabeled and how test loss is changed with different values of α under different noise ratios.
First, when noise ratio equals to 0, there are few biased samples in the training set.
In this case, simply relabeling all the identiﬁed harmful samples will hurt the performance while using relatively larger α could report lower test loss.
Second, when noise ratio is 0.8, RDIA achieves better performance with smaller α .
This is reasonable since most of training samples involve label noises and increasing α facilitates the relabeling of noisy samples.
Size of the validation set.
As discussed in Section 3.3, we use the validation set instead of the test set to estimate the inﬂuence of each training sample.
Table 4 shows the results of how the number of validation samples affects the model performance.
We conduct the experiments under 40% noise rates and ﬁnd the optimal hyperparameter α ∈ [0 . 0002 , 0 .
01] to get the best results of RDIA.
We have the following observations.
1) Using only 100 validation samples with RDIA achieves 35% lower test loss than ERM. 2)
As the number of validation samples increases, RDIA signiﬁcantly outperforms ERM, achieving up to 90% relative lower in test loss.
The reason is that, as the number of validation sets increases, the validation set can gradually reﬂect the true distribution of test data.
In this way, the estimated inﬂuence functions are accurate enough to ﬁlter out most harmful training samples for the test set.
3) RDIA consistently outperforms UIDS with different sizes of validation set, which empirically shows the effectiveness of our relabeling function R .
C ONCLUSION
In this paper, we propose to perform data relabeling based on inﬂuence functions to resolve the training bias issue.
We develop a novel relabeling framework named RDIA, which reuses the information of harmful training samples identiﬁed by inﬂuence analysis towards higher model performance.
We theoretically prove that RDIA can further reduce the test loss than simply discarding harmful training samples on any classiﬁcation tasks using the cross-entropy loss function.
Extensive experiments on real datasets verify the effectiveness of RDIA in enhancing model’s robustness and ﬁnal performance, compared with various resampling and relabeling techniques.
Reproducibility: We clarify the assumptions in Section 2 and provide the complete proofs of Lemmas, Theorems in
Appendix B.
The statistics of datasets, the data processing, and the details of the experimental settings are described in Appendix C.
Our code could be found in the https://github.com/Viperccc/RDIA . A CKNOWLEDGMENT
This work is supported by Shanghai Municipal Science and Technology Major Project (2021SHZDZX0102), the Tencent Wechat Rhino-Bird Focused Research Program and SJTU
Global Strategic Partnership Fund (2021 SJTU-HKUST).
Yanyan Shen is the corresponding author of this paper.
Appendix
In this appendix, we ﬁrst provide the algorithm of RDIA (Appendix A) and the complete proofs of the Lemmas and Theorems (Appendix B) in the main text.
Then we give the details of the experimental settings (Appendix C), the extensive analysis of our approach (Appendix D), and the visualization of identiﬁed harmful samples (Appendix E).
We then describe RDIA-LS, an extension of RDIA, to spotlight the scalability of our approach RDIA (Appendix F) and provide empirical results to show that RDIA-LS is effective and efﬁcient to handle training data with corrupted labels for deep learning (Appendix G).
Finally, we provide additional discussions about the existing data relabeling approaches (Appendix H)
A RDIA ALGORITHM
Input: Training model θ , biased training set D
= { ( x i , y i ) }
Ni =1 , learning rate β , sample selection ratio α such that 0 ≤ α ≤ 1 , small and unbaised set Q = { ( x cj , y cj ) }
Mi = 1 Train the model θ with D until convergence to get ˆ θ ; 2 Initialize
+
∅ N do Calculate the the z Eq.
(6): if then
Relabel the identiﬁed harmful training samples with
(cid:48)
i ← R ( z i );
D
← D
∪ {
} else
) <
then
D + ← D + ∪ { i } end 11 end 12 Obtain the new training set
D ←
13 Retrain the ﬁnal
B P ROOFS FOR L EMMAS AND T HEOREMS B. P ROOF OF L EMMA
Assume the perturbation (cid:15) i on z i is inﬁnitesimal and the inﬂuence of each training sample on the test risk is independent.
Lemma 1.
Discarding or downweighting the training samples in D − = { z i ∈ D | Φ θ ( z i ) > 0 } from D could lead to a model with lower test risk over Q :
[Equation]
where ˆ θ (cid:15) denotes the optimal model parameters obtained by updating the model’s parameters with discarding or downweighting samples in D − .
Proof.
Recall that ˆ θ (cid:15) i =
arg min θ 1 N (cid:80)
Nn =
1 l n ( θ i )+ (cid:15)
i l i ( θ ) .
In this way, downweighting the training sample z i in D − means setting (cid:15) i ∈
[ − 1 N , 0)
(Noticed that (cid:15) i = − 1 N means discarding training sample z i ).
For convenience of analysis, we set all (cid:15) i equal to − 1 N and have Φ θ ( z i ) (cid:44) (cid:80) Mj
=
Φ θ ( z i , z cj ) .
According to Eq.
(6), we can estimate how the test risk is changed by discarding or downweighting z i ∈ D − as follows:
[Equation]
B. P ROOF OF T HEOREM
Theorem 1.
In binary classiﬁcation, let σ be the inﬁmum of ϕ ( x i , ˆ θ ) 1 − ϕ
( x i , ˆ θ ) and 1 − ϕ ( x i , ˆ θ ) ϕ
( x i , ˆ θ ) and D − = { z i ∈ D | Φ θ ( z i ) > 0 } .
Relabeling the samples in D − can achieve lower test risk than discarding or downweighting them from D , because the following inequality holds.
[Equation]
Proof.
Based on Eq.
(9), we have:
[Equation]
It is worth mentioning that ˆ θ (cid:15) i
[Equation]
i =
[Equation]
min θ
[Equation]
[Equation]
) − (cid:15)
i l i ( θ ) .
In this way, relabeling the training sample z i in D − means setting (cid:15) i = 1 N .
Similar to the proof of Lemma 1, according to Eq.
(6) and Eq.
(10), we have:
[Equation]
B. P ROOF OF L EMMA
Lemma 2.
When applying the relabeling function R in Eq.
(12) over a training sample z i ∈ D with a class label m , the CE loss l i ( θ ) at z i is changed from − log( ϕ m ( x i , θ )) to − log(1 − ϕ
m
( x i , θ )) .
Proof.
Recall that the model’s prediction at x i is ϕ
( x i , θ ) =
( ϕ 1 , ϕ 2 , ..., ϕ K )
and our relabeling function is:
[Equation]
ϕ k K − 1 √ 1 − ϕ m , otherwise
Here we assume the training example
z i belongs to class m which means that y im =
and the other components in the one-hot vector
y i are 0 .
The prime CE loss is − log( ϕ m ( x i , θ )) .
If we use our relabeling function to change the label of x i , the loss at z i will be:
[Equation]
In this way, if we use relabeling function R to change the label of example z i , the loss function will be changed from − log( ϕ m ( x i , θ )) to − log(1 − ϕ m
( x i , θ )) .
B. P ROOF OF T HEOREM
Theorem 2.
In multi-class classiﬁcation, let ϕ y
( x i , ˆ θ ) denote the probability that z i is classiﬁed as its truth class label by the model with the optimal parameters ˆ θ on D , and σ be the inﬁmum of ϕ y ( x i , ˆ θ ) 1 − ϕ y
( x i , ˆ θ ) .
Relabeling the samples in D − = { z i ∈ D | Φ θ ( z i ) > 0 } with R leads to a test risk lower than the one achieved by discarding or downweighting D − .
Formally, we have:
[Equation]
Proof.
According to Lemma 2, Eq.
(8)and Eq.
(10), we can estimate the change of test loss at a test sample z cj ∈ Q caused by relabeling as follows:
[Equation]
Further
[Equation]
we can derive the following:
[Equation]
C E XPERIMENTAL S ETTINGS C. T HE S TATISTICS OF THE D ATASETS
Table 5 shows the statistics of the datasets.
We perform extensive experiments on public datasets from different domains to verify the effectiveness and robustness of our approach RDIA.
All the datasets could be found in https://www.csie.ntu.edu.tw/cjlin/libsvmtools/datasets/.
[Table]
C. T R -V A -T E D IVISIONS
We follow the Tr-Va-Te (Training/Validation/Test set divisions) setting in Wang et al.
(2020b) to measure the generalization ability of our approach RDIA.
Speciﬁcally, the inﬂuence of each training instance is estimated with the validation set using the validation loss and the model’s performance is tested by an additional out-of-sample test set which ensures we do not utilize any information of the test data.
When training logistic regression, we randomly pick up 30% samples from the training set as the validation set.
For different inﬂuence-based approaches, the training/validation/test sets are kept the same for fair comparison.
Both MNIST and CIFAR10 are 10-classes image classiﬁcation datasets while logistic regression can only handle binary classiﬁcation.
On MNIST, we select the numberand 7 as positive and negative classes, respectively;
On CIFAR10, we perform binary classiﬁcation on cat and dog.
For each image, we convert all the pixels into a ﬂattened feature vector where each pixel is scaled by 1/255.
When training deep models, due to the high time complexity of estimating inﬂuence functions, we randomly exclude 100 samples (1%) from the test sets of MNIST and CIFAR10 as the respective validation sets, and the remaining data is used for testing.
C. I MPLEMENTATION D ETAILS
We used the Newton-CG algorithm Martens (2010) to calculate the inﬂuence functions for the logistic regression model and applied Stochastic estimation Agarwal et al.
(2017) for two deep models with 1000 clean data in the validation set.
For logistic regression model, we select the regularization term C = 0.1 for fair comparison.
We adopt the Adam optimizer with the learning rate of 0.
to train the LeNet on MNIST.
After calculating the inﬂuence functions and relabeling the identiﬁed harmful training samples using R , we reduce the learning rate to 10 − 5 and update the models until convergence.
For CIFAR10, we use the SGD optimizer with the learning rate of 0.01 and the momentum of 0.9 to train the CNN.
Then we change the learning rate to 0.001 and update the models based on the relabeled training set.
Here we use different optimizers to train the models.
This indicates RDIA is independent of the update strategy used for model training.
The batch size is set to 64 in all the experiments and the hyperparameter α is tuned with the validation set for best performance.
D E XTENSIVE ANALYSIS OF RDIA D. C OMPLEXITY ANALYSIS
According to Koh & Liang (2017), the time complexity of calculating inﬂuence function for one training sample(i.e.
, Eq. 2) is O ( NP ) , where N and P stand for the sizes of training set and model’s parameter set, respectively.
Note that the time complexity of relabeling one sample is
O ( N ) .
Considering the complexity of calculating inﬂuence functions, the time cost of relabeling harmful samples is negligible which means our RDIA is as fast as any inﬂuence-based approaches.
[Figure]
D. R ELATIONSHIP WITH PROPENSITY SCORE
Propensity score (Rosenbaum & Rubin, 1983; Bickel et al., 2009) is a well-studied technique to solve the distribution mismatch (also called covariate shift) problem where training and testing sets are sampled from two different distributions P train ( x, y ) and P test ( x, y ) , respectively.
Its basic idea is to assign the propensity score to each training sample to make the test risk unbiased.
Unlike the inﬂuence function calculated by measuring the change of test loss, propensity score is calculated directly by estimating the probability of each training sample belonging to the test distribution. If we could estimate the training and test distribution accurately, we could also use the propensity score to replace the inﬂuence function for identifying whether the training sample is harmful. We leave it for the future work. E V ISUALIZATION OF IDENTIFIED HARMFUL SAMPLES
We provide examples of harmful samples identiﬁed by inﬂuence functions to illustrate the effectiveness of inﬂuence analysis.
We apply the logistic regression on MNIST (class 1 and 7) and CIFAR (class cat and dog).
The inﬂuence functions are estimated by Newton-CG algorithm (Martens, 2010).
We provide the three most harmful images which have the highest inﬂuence scores and share the same label with the test sample.
Figure 3(a) shows three identiﬁed harmful training images for each test image when there are no ﬂipped labels in the training set.
We can see that the identiﬁed harmful training samples are visually different from the original pictures, which disturbs the model’s prediction on the test image.
That is, the presence of clean but harmful training images would damage the model’s performance.
Figure 3(b) shows the identiﬁed harmful training images when 50% labels of training data have been ﬂipped.
It is easy to see that the harmful images have corrupted labels, which conﬁrms the effectiveness of applying inﬂuence analysis to locate noisy samples.
F RDIA-LS: A L OSS -B ASED R ELABELING A PPROACH F. L IMITATIONS OF RDIA
In the main paper, we have discussed a novel data relabeling framework RDIA via inﬂuence analysis.
Based on the advantages of inﬂuence functions, RDIA is able to handle different types of training biases and is agnostic to a speciﬁc model or data type.
However, the time complexity of estimating the inﬂuence of one training sample is O ( NP ) , where N and P stand for the sizes of training set and model’s parameter set, respectively.
This is relatively high for deep models which have thousands of parameters.
Moreover, according to (Koh & Liang, 2017), the approximate estimation of inﬂuence functions on deep models may not be accurate and hence the second step of RDIA suffers from false positives and false negatives.
When harmful samples account for the majority of the training set, e.g., high noise rates, it is difﬁcult to ﬁlter most of harmful samples using the estimated inﬂuence.
F. RDIA-LS
To address the aforementioned limitations, we aim to extend RDIA to solve the speciﬁc problem.
Here we focus on combating noisy labels with deep models since label noise is usually a primary root cause of training bias.
We notice that training loss has been used to ﬁlter out training samples with corrupted labels in many previous works (Arpit et al., 2017; Han et al., 2018; Wei et al., 2020;
Yu et al., 2019).
It is worth mentioning that the noisy training samples identiﬁed by training loss are not equivalent to the harmful ones identiﬁed by inﬂuence functions because the latter are evaluated to have negative inﬂuence on the test performance.
Nevertheless, since the selected high-loss training samples are very likely to involve corrupted labels, applying our relabeling function over them has the potential of correcting corrupted labels and beneﬁting the test performance.
Besides, using training loss to identify harmful samples is more efﬁcient as it avoids the estimation of inﬂuence functions.
Hence, we propose to use training loss to identify noisy samples and develop a lossbased data relabeling approach named RDIA-LS, which can be viewed as an extension of RDIA for combating corrupted labels with deep models.
RDIA-LS consists of three steps: noisy samples identiﬁcation , noisy samples relabeling and model updating .
It shares the same last two steps with RDIA.
The only difference between RDIA-LS and
RDIA is that RDIA-LS uses training loss to identify noisy samples in each training epoch so that it does not need to train the model until convergence ﬁrst.
Speciﬁcally, given a mini-batch of training instances ¯ D ⊆ D , RDIA-LS feeds forward all the samples in ¯ D and then sorts them in an ascending order of their training losses.
Following the prior works (He & Garcia, 2009), we regard the largeloss instances as noisy and the small-loss instances as clean.
We use the rate of ρ to select the possibly clean training instances in ¯ D , i.e., ¯ D + = arg
min ¯ D : | ¯ D |≥ ρ | ¯ D | L ( ¯ D, θ ) .
The remaining highloss training instances are treated as noisy samples, i.e., ¯ D − =
¯ D \ ¯ D
+ .
We follow (Han et al., 2020) to determine the value of the selection ratio ρ .
After we have ¯ D − , we use our relabeling function R to relabel the samples in ¯ D − and then update the model with ¯ D + ∪ ¯ D (cid:48)− .
In our implementation, we simply modify the loss of the identiﬁed noisy samples based on Lemma 2 without performing actual relabeling.
We use the hyperparameter γ ∈ [0 , 1] to control the model tendency of learning from the clean instances and the relabeled noisy instances.
The detailed procedure of RDIA-LS is provided in Algorithm 2. [Table]
[Table] [Table] We conduct the additional experiments in Appendix G to empirically show that RDIA-LS is effective and efﬁcient to handle training data with corrupted labels for deep learning, which spotlights the great scalability of our approach RDIA.
G P ERFORMANCE E VALUATION OF RDIA-LS
We now conduct the experiments to evaluate the effectiveness and efﬁciency of RDIA-LS using
DNNs on MNIST, CIFAR10, CIFAR100 and Clothing1M.
The ﬁrst three datasets are clean and corrupted artiﬁcially.
Clothing1M is a widely used real-world dataset with noisy labels (Patrini et al., 2017).
G. I MPLEMENTATION D ETAILS
We apply the same network structures used in the main paper and use LeNet (2 convolutional layers and 1 fully connected layer) for MNIST, a CNN with 6 convolutional layers followed by 2 fully connected layers used in (Wang et al., 2019) for CIFAR10 and CIFAR100, and a 18-layer ResNet for Clothing1M.
We follow the settings in (Han et al., 2018) for all the comparison methods.
Specifically, for MNIST, CIFAR10 and CIFAR100, we use the Adam optimizer with the momentum of 0.9, initial learning rate of 0.001, and the batch size of 128.
We run 200 epochs (T=200) in total and linearly decay the learning rate till zero from 80 to 200 epochs.
As for Clothing1M, we use the
Adam optimizer with the momentum 0.9 and set the batch size to be 64.
We run 15 epochs in total and set learning rate to 8 × 10 − 4 , 5 × 10 − 4 and 5 × 10 − 5 for average ﬁve epochs.
We set the ratio of small-loss instances as ρ = 1 − min { tT k ∗ τ, τ } which is changed dynamically with the current training epoch t and T k = 5 for Clothing1M and T k = 10 for the other datasets.
In this way, we can determine D − and D + in each training epoch.
If the noise ratio τ is not known in advance, we could use the method (Yu et al., 2018) to estimate τ .
The hyperparameter γ is tuned in { 0 . 05 , 0 . 10 , 0 . 15 , · · · , 0 .
95 } with the validation set for best performance.
If there is no validation set, we could use training loss to select a clean subset from the training set as the validation set.
Following loss-based approach (Han et al., 2020; 2018; Jiang et al., 2018), we use the test accuracy as the metric, i.e., (#correct predictions) /
(#test instances).
[Table]
[Table]
G. C OMPARISON M ETHODS
We compare our proposed RDIA-LS with the following baselines.
S-model (Goldberger & BenReuven, 2017) and F-correction (Patrini et al., 2017) are the existing data relabeling approach which aims to estimate the noisy transition matrix to correct the noisy labels.
The last three approaches are the state-of-the-art loss-based resampling approaches.
(1) ERM : it trains one network with all the training data using cross-entropy loss.
(2) S-model (Goldberger & Ben-Reuven, 2017): it uses an additional softmax layer to model the noise transition matrix to correct the model (3) Fcorrection (Patrini et al., 2017): it corrects the prediction by the noise transition matrix estimated by the other network.
(4) Self-teaching (Jiang et al., 2018): it trains one network with only the selected small-loss instances D + .
(5) Co-teaching (Han et al., 2018): it trains two networks simultaneously and improves self-teaching by updating the parameters of each network with the small-loss instances D + selected by the peer network.
(6) SIGUA (Han et al., 2020): it trains one network with the selected small-loss instances D + and high-loss instances D − via gradient descent and gradient ascent, respectively.
G. E XPERIMENTAL R ESULTS
Comparison with the Baselines
RDIA-LS is proposed to combat noisy labels for deep learning.
In order to evaluate how RDIALS improves the robustness of deep models, we perform experiments on MNIST+LeNet, CIFAR10+CNN and CIFAR100+CNN with different noise ratios and the real-world noisy dataset
Clothing1M+Resnet18.
The average results of test accuracy are reported in Table 6, Table 7, Tableand Table 9 .
We have the following observations.
(1) RDIA-LS achieves the highest test accuracy in all the cases.
When noise ratio is 0.2, the improvement of RDIA-LS is relatively small.
This is reasonable as the performance gain of RDIA-LS obtained from utilizing noisy data is restricted due to the low noise ratio.
When the noise ratio exceeds 0.4, RDIA-LS signiﬁcantly outperforms the existing loss-based approaches, achieving up to 5% relative improvement in test accuracy.
It indicates that RDIA-LS can still effectively reuse harmful training instances to improve model’s robustness under high noise ratios.
(2) RDIA-LS consistently performs better than S-model, F-correction, and
SIGUA, which implies that using R to relabel noisy training samples identiﬁed by training loss is more effective than modeling the noise transition matrix or performing gradient ascent with identiﬁed noisy training instances.
(3) RDIA-LS outperforms all the baselines on the real-world noisy dataset Clothing1M, which demonstrates the effectiveness of applying RDIA-LS in practice.
Comparison with RDIA.
Table 10 reports the running time of harmful/noisy samples identiﬁcation in RDIA and RDIA-LS.
We exclude the results of LS on logistic regression since training loss can only be used to ﬁlter out noisy samples for training deep models.
From the table, we can see that using inﬂuence function to identify harmful samples for logistic regression is efﬁcient.
However, when training deep models with millions of parameters, using training loss to ﬁlter out noisy samples is much more efﬁcient.
[Table]
RDIA-LS is an extension of RDIA to combat noisy samples with deep models.
The aforementioned experimental results show that RDIA-LS is effective and efﬁcient to handle training data with corrupted labels for deep learning.
However, it is worth noticing that RDIA-LS relies on the small-loss trick that the samples which have the larger training loss may contain the corrupted labels.
In this way, RDIA-LS is only suitable for the deep models against the corrupted labels and could fail in the situation where the small-loss trick does not hold while RDIA has no such constraint.
H A DDITIONAL D ISCUSSION ON D ATA R ELABELING
Existing relabeling approaches (Goldberger & Ben-Reuven, 2017; Jiang et al., 2018; Lee et al., 2018) are proposed to combat noisy labels with DNNs.
They focus on estimating the noise transition matrix to convert the corrupted labels to clean labels.
However, current relabeling methods suffer from two limitations.
First, they aim to ﬁnd the true labels of the training samples, which means they can only deal with label noise.
Second, they employ some additional structures to correct the labels, which are dependent of the speciﬁc model structures.
For example, Goldberger et al.
(Goldberger & Ben-Reuven, 2017) added an additional softmax layer to present the noise transition matrix and
CleanNet (Lee et al., 2018) used the auto-encoder to update the corrupted labels.
In contrast, we aim to develop a relabeling function based on inﬂuence analysis to change the labels of harmful samples towards better model performance.
We do not require output labels to be one-hot vector since our objective is not to ﬁnd the truth labels of training samples.
Besides, we extend our approach to
RDIA-LS to effectively combat noisy samples for training DNNs, which outperforms the existing data relabeling approaches (Goldberger & Ben-Reuven, 2017; Jiang et al., 2018).
DUTI (Zhang et al., 2018) is an effective data relabeling approach which could debug and correct the wrong labels in the training set.
It uses a bi-level optimization scheme to recommend the most inﬂuential training samples for cleaning and suggest the possibly cleaned labels.
The proposed relabeling function in DUTI is different from our approach.
Speciﬁcally, the relabeling function in DUTI is trained by a bi-level optimization using the gradient of the validation loss while our proposed relabeling function has nothing to do with gradient, validation loss.