H ETEROSKEDASTIC AND I MBALANCED D EEP L EARNING WITH A DAPTIVE R EGULARIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Real-world large-scale datasets are heteroskedastic and imbalanced – labels have varying levels of uncertainty and label distributions are long-tailed.
Heteroskedasticity and imbalance challenge deep learning algorithms due to the difﬁculty of distinguishing among mislabeled, ambiguous, and rare examples.
Addressing heteroskedasticity and imbalance simultaneously is under-explored.
We propose a data-dependent regularization technique for heteroskedastic datasets that regularizes different regions of the input space differently.
Inspired by the theoretical derivation of the optimal regularization strength in a one-dimensional nonparametric classiﬁcation setting, our approach adaptively regularizes the data points in higher-uncertainty, lower-density regions more heavily.
We test our method on several benchmark tasks, including a real-world heteroskedastic and imbalanced dataset, WebVision.
Our experiments corroborate our theory and demonstrate a signiﬁcant improvement over other methods in noise-robust deep learning.
I NTRODUCTION
In real-world machine learning applications, even well-curated training datasets have various types of heterogeneity.
Two main types of heterogeneity are: (1) data imbalance: the input or label distribution often has a long-tailed density, and (2) heteroskedasticity: the labels given inputs have varying levels of uncertainties across subsets of data stemming from various sources such as the intrinsic ambiguity of the data or annotation errors.
Many deep learning algorithms have been proposed for imbalanced datasets (e.g., see (Wang et al., 2017;
Cao et al., 2019; Cui et al., 2019; Liu et al., 2019) and the reference therein).
However, heteroskedasticity, a classical notion studied extensively in the statistical community (Pintore et al., 2006;
Wang et al., 2013; Tibshirani et al., 2014), has so far been under-explored in deep learning.
This paper focuses on addressing heteroskedasticity and its interaction with data imbalance in deep learning.
Heteroskedasticity is often studied in regression analysis and refers to the property that the distribution of the error varies across inputs.
In this work, we mostly focus on classiﬁcation, though the developed technique also applies to regression.
Here, heteroskedasticity reﬂects how the uncertainty in the conditional distribution y | x , or the entropy of y | x , varies as a function of x .
Real-world datasets are often heteroskedastic.
For example, Li et al.
(2017) shows that the WebVision dataset has a varying number of ambiguous or true noisy examples across classes.
Conversely, we consider a dataset to be homoscedastic if every example is mislabeled with a ﬁxed probably (cid:15) , as assumed by many prior theoretical and empirical works on label corruption (Ghosh et al., 2017; Han et al., 2018; Jiang et al., 2018).
We note that varying uncertainty in y | x can come from at least two sources: the intrinsic semantic ambiguity of the input, and the (data-dependent) mislabeling introduced by the annotation process.
Our approach can handle both types of noisy examples in a uniﬁed way, but for the sake of comparisons with past methods, we call them “ambiguous examples” and “mislabeled examples” respectively, and refer to both of them as “noisy examples”.
Overparameterized deep learning models tend to overﬁt more to the noisy examples (Arpit et al., 2017; Frénay & Verleysen, 2013; Zhang et al., 2016).
To address this issue, a common approach is to detect noisy examples by selecting those with large training losses, and then remove them
[Figure] [Figure]
from the (re-)training process.
However, an input’s training loss can also be big because it is rare or ambiguous (Hacohen & Weinshall, 2019; Wang et al., 2019), as shown in Figure 1.
Noise-cleaning methods could fail to distinguish mislabeled from rare/ambiguous examples (see Section 3.1 for empirical proofs).
Though dropping the former is desirable, dropping the latter loses important information.
Another popular approach is reweighting methods that reduce the contribution of noisy examples in optimization.
However, failing to distinguish between mislabeled and rare/ambiguous examples makes the decision of the weights tricky – mislabeled examples require small weights, whereas rare / ambiguous examples beneﬁt from larger weights (Cao et al., 2019; Shu et al., 2019).
We propose a regularization method that deals with noisy and rare examples in a uniﬁed way.
We observe that mislabeled, ambiguous, and rare examples all beneﬁt from stronger regularization (Hu et al., 2020; Cao et al., 2019).
We apply a Lipschitz regularizer (Wei & Ma, 2019a;b) with varying regularization strength depending on the particular data point.
Through theoretical analysis in the one-dimensional setting, we derive the optimal regularization strength for each training example.
The optimal strength is larger for rarer and noisier examples.
Our proposed algorithm, heteroskedastic adaptive regularization (HAR), ﬁrst estimates the noise level and density of each example, and then optimizes a Lipschitz-regularized objective with input-dependent regularization with strength provided by the theoretical formula.
In summary, our main contributions are: (i) we propose to learn heteroskedastic imbalanced datasets under a uniﬁed framework, and theoretically study the optimal regularization strength on onedimensional data.
(ii) we propose an algorithm, heteroskedastic adaptive regularization (HAR), which applies stronger regularization to data points with high uncertainty and low density.
(iii) we experimentally show that HAR achieves signiﬁcant improvements over other noise-robust deep learning methods on simulated vision and language datasets with controllable degrees of data noise and data imbalance, as well as a real-world heteroskedastic and imbalanced dataset, WebVision.
A DAPTIVE R EGULARIZATION FOR H ETEROSKEDASTIC D ATASETS 2. B ACKGROUNDS
We ﬁrst introduce general nonparametric tools that we use in our analysis, and review the dependency of optimal regularization strength on the sample size and noise level.
Over-parameterized neural networks as nonparametric methods.
We use nonparametric method as a surrogate for neural networks because they have been shown to be closely related.
Recent work (Savarese et al., 2019) shows that the minimum norm two-layer ReLU network that ﬁts the training data is in fact a linear spline interpolation.
Parhi & Nowak (2019) extend this result to a broader family of neural networks with a broader family of activations.
[Figure]
Given a training dataset { ( x i , y i ) } ni =1 , nonparametric method with penalty works as follows.
Let
F : R → R be a twice-differentiable model family.
We aim to ﬁt the data with smoothness penalty
[Equation]
Lipschitz regularization for neural networks.
Lipschitz regularization has been shown to be effective for deep neural networks as well.
Wei & Ma (2019a) proves a generalization bound of neural networks dependent on the Lipschitzness of each layer with respect to all intermediate layers on the training data, and show that, empirically, regularizing the Lipschitzness improve the generalization.
Sokoli ´ c et al.
(2017) shows similar results in data-limited settings.
In Section 2.3, we extend the
Lipschitz regularization technique to heteroskedastic setting.
Regularization strength as a function of noise level and sample size.
Finally, we brieﬂy review existing theoretical insights on the optimal choice of regularization strength.
Generally, the optimal regularization strength for a given model family increases with the label noise level and decreases in the sample size.
As a simple example, consider linear ridge regression min θ 1 n (cid:80) ni
=1 ( x (cid:62) i θ − y i )
+ λ (cid:107) θ (cid:107) 2 , where x i , θ ∈ R d and y
i ∈
R .
We assume y i
=
x (cid:62) i θ ∗ + ξ for some ground truth parameter θ ∗ , and ξ ∼ N (0 , σ 2 ) .
Then the optimal regularization strength λ opt = dσ 2 /n (cid:107) θ ∗ (cid:107) 22 .
Results of similar nature can also be found in nonparametric statistics (Wang et al., 2013; Tibshirani et al., 2014).
2. H ETEROSKEDASTIC N ONPARAMETRIC C LASSIFICATION ON O NE -D IMENSIONAL D ATA
We consider a one-dimensional binary classiﬁcation problem where X = [0 , 1] ⊂ R and Y = {− 1 , 1 } .
We assume Y given X follows a logistic model with ground-truth function f (cid:63)
[Equation]
i.e.
[Equation]
The training objective is cross-entropy loss plus Lipschitz regularization, i.e.
[Equation]
where (cid:96) ( a, y )
=
− log(1 + exp( − ya )) , and ρ ( x ) is a smoothing parameter as a function of the noise level and density of x .
Let I ( x ) be the ﬁsher information matrix conditioned on the input, i.e.
I ( x ) (cid:44) E [ ∇ 2 a (cid:96) ( a, Y ) | a = f (cid:63) ( X ) | X = x ] .
When ( X, Y ) follows the logistic model in equation 2,
I ( x ) =
[Equation]
Therefore, I ( x ) captures the aleatoric uncertainty of x .
For example, when Y is deterministic conditioned on X = x , we have I ( x ) = 0 , indicating perfect certainty.
Deﬁne the test metric as the mean-squared-error on the test set { ( x i , y i ) } ni =1 , i.e.,
[Equation]
Our main goal is to derive the optimal choice of ρ ( x ) that minimizes the MSE.
We start with an analytical characterization of the test error.
Let W
= { f (cid:48) is absolute continuous and f (cid:48)(cid:48)
∈
L 2 [0 , 1] } .
We denote the density of X as q ( x ) .
The following theorem analytically computes the MSE under the regularization strength ρ ( · ) , building upon (Wang et al., 2013) for regression problems.
The proof of the Theorem is deferred to Appendix A.
Theorem 1.
Assume that f (cid:63) , q, I ∈ W 22 .
Let r ( t ) =
− 1 / ( q ( t ) I ( t )) and L 0 = (cid:82) ∞
−∞ 14 exp( − 2 | t | ) dt .
If we choose λ = C 0 n − 2 / 5 for some constant C 0 > 0 , the asymptotic mean squared error is
[Equation]
in probability, where C n is a scalar that only depends on n .
Using the analytical formula of the test error above, we want to derive an approximately optimal choice of ρ ( x ) .
A precise computation is infeasible, so we restrict ourselves to consider only ρ ( x ) that is constant within groups of examples.
We introduce an additional structure – we assume the data can be divided into k groups [ a 0 , a 1 ) , [ a 1 , a 2 ) , · · · , [ a k − 1 , a k ) .
Each group [ a j , a j +1 ) consists of an interval of data with approximately the same aleatoric uncertainty.
We approximate ρ ( t ) is constant on each of the group [ a i , a i +1 ) with value
ρ i .
Plugging this piece-wise constant ρ into the asymptotic
MSE in Theorem 1, we obtain
[Equation]
Minimizing the above formula over ρ 1 , . .
. , ρ k separately, we derive the optimal weights, ρ j = (cid:34)
[Equation]
In practice, we do not know f (cid:63) and q ( x ) , so we make the following simpliﬁcations.
We assume that q ( t ) and I ( t ) are constant on each interval [ a j , a j +1 ] .
In other words, we assume that q ( t ) =
q j and I ( t ) =
I j for all t ∈ [ a j , a j +1 ] .
We further assume that d 2 dt 2 f (cid:63) ( t ) is close to a constant on the entire space, because estimating the curvature in high dimension is difﬁcult.
This simpliﬁcation yields ρ j ∝
[Equation]
=
q
[Equation] [Equation]
We ﬁnd the simpliﬁcation works well in practice.
Adaptive regularization with importance sampling.
It is practically infeasible to implement the integration in equation 3 for high-dimensional data.
We use importance sampling to approximate the integral:
[Equation]
Suppose x i ∈
[ a j , a j +1 ) , we have that τ i should satisfy that
τ i q j = ρ j so that the expectation of the regularization term in equation 5 is equal to that in equation 3.
Hence,
[Equation]
Adaptive regularization for multi-class classiﬁcation and regression.
In fact, the proof of
Theorem 1 is proved for general loss (cid:96) ( a, y ) .
Therefore, we can directly generalize it to multiclass classiﬁcation and regression problems.
For a regression problem, (cid:96) ( a, y ) is the square loss: (cid:96) ( y, a )
= 0 .
5( y − a ) 2 , the Fisher information I ( x ) = 1 .
Therefore, for a regression problem, we can choose regularization weight
τ i = q ( x i ) − 2 / 5 .
2. P RACTICAL I MPLEMENTATION ON N EURAL N ETWORKS WITH H IGH - DIMENSIONAL D ATA
We heuristically extend the Lipschitz regularization technique discussed in Section 2.2 from nonparametric models to over-parameterized deep neural networks.
Let ( x, y ) be an example and f θ be an r -layer neural network.
We denote by h ( j ) the j -th hidden layer of the network, by
J ( j ) ( x ) (cid:44) ∂
∂h ( j ) L ( f ( x ) , y ) , i.e., the Jacobian of the loss w.r.t h ( j ) .
We replace the regularization term f (cid:48)
( x ) 2 in equation 5 by R ( x )
= (cid:16)(cid:80) rj =1 (cid:107)
J
( j ) ( x ) (cid:107) 2 F (cid:17) 1 / 2 , which was proposed by (Wei & Ma, 2019a).
As a proof of concept, we visualize the behavior of our algorithm in Figure 3, where we observe that the rare and noisy examples have signiﬁcantly improved error due to stronger regularization.
In contrast, a uniform regularization either overﬁts or underﬁts different subsets.
Note that the differences from the 1-D case include the following three aspects.
The derivative is taken w.r.t to all the hidden layers for deep models, which has been shown to have superior generalization guarantees for neural networks by (Wei & Ma, 2019a;b).
An additional square root is taken in computing R ( x ) .
This modiﬁed version may have milder curvature and be easier to tune.
We take the derivative of the loss instead of the derivative of the model, which outputs k numbers for multi-class classiﬁcation.
This is because the derivative of the model requires k times more time to compute.
The regularized training objective is consequently minimize
[Equation]
(6) where τ i is chosen to be τ i
=
I ( x i ) 3 / 5 /q ( x i ) 2 / 5 following the formula equation 5 in Section 2.2 and λ is a hyperparameter to control the overall scale of the regularization strength.
We note that we do not expect this choice of τ i to be optimal for the high-dimensional case with all the modiﬁcations above – the optimal choice does depend on the nuances.
However, we also observe that the empirical performance is not sensitive to the form of τ as long as it’s increasing in I ( x ) and decreasing in q ( x ) .
That is, the more uncertain or rare an example is, the stronger regularization should be applied.
In order to estimate the relative regularization strength τ i , the key difﬁculty lies in the estimation of uncertainty I ( x ) .
As in the 1-D setting, we divide the examples into k groups G 1 , . .
, G k
(e.g., each group can correspond to a class), and estimate the uncertainty on each group.
In the binary setting, I ( x ) =
Var ( Y | X = x ) =
Pr[ Y = 1 | X ] · Pr[ Y =
| X ] can be approximated by ˜
I
( x ) = 1 − max k ∈{ 0 , 1 } Pr[ Y = k | X
= x ] up to a factor of at most 2.
We use the same formula for multi-class setting as the approximation of the uncertainty.
(As a sanity check, when Y is concentrated on a single outcome, the uncertainty is 0.)
Note that ˜
I
( x ) is essentially the minimum possible error of any deterministic prediction on the data point x .
Assume that we have a sufﬁciently accurate pre-trained model, we can use its validation error to estimate ˜
I
( x ) :
Then for all x ∈ G j , we estimate q ( x ) and I ( x ) by ∀ x ∈ G j , q ( x ) ∝ | G j | , I ( x ) ∝ average validation error of a pre-trained model f ˜ θ on G j (7)
The whole training pipeline is summarized in Algorithm 1.
Algorithm
Require: Dataset D = { ( x i , y i ) } ni =1 .
A parameterized model f θ
E XPERIMENTS
We experimentally show that our proposed algorithm HAR(Algorithm 1) improves the test performance of the noisier and rarer groups of examples (by stronger regularization) without negatively
[Table]
affecting the training and test performance of the other groups.
We evaluate our algorithms on three vision datasets and one NLP dataset: CIFAR-10 and CIFAR-100 (Krizhevsky et al., 2009),
IMDB-review (Maas et al., 2011) (see Appendix C.3), and WebVision (Li et al., 2017), a real-world heteroskedastic and imbalanced dataset.
Please refer to Appendix B for low-level implementation details.
Code will be made publicly available at the time of the publication.
Baselines.
We compare our proposed HAR with the following baselines.
The simplest one is (1)
Empirical risk minimization (ERM): the vanilla cross-entropy loss with all examples having the same weights of losses.
We select two representatives from the noise-cleaning line of approach.
Co-teaching (Han et al., 2018): two deep networks are trained simultaneously.
Each network aims to identify clean data points that have small losses and use them to guide the training of the other network.
(3) INCV (Chen et al., 2019): it extends Co-teacing to an interative version to estimate the noise ratio and select data.
We consider three representatives from the reweighting-based methods , including two that learn the weighting using meta-learning .
(4) MentorNet (Jiang et al., 2018): it pretrains a teacher network that outputs weights for examples that are used to train the student network with reweighting.
(5) L2RW
(Ren et al., 2018): it directly optimizes weights of each example in the training set by minimizing its corresponding loss on a small meta validation set.
(6) MW-Net (Shu et al., 2019): it extends L2RW by explicitly deﬁning a weighting function which depends only on the loss of the example.
We also compare against two representatives from the robust loss function .
(7) GCE (Zhang & Sabuncu, 2018): it generalizes mean average error and cross-entropy loss to obtain a new loss function.
(8) DMI (Xu et al., 2019): it designs a new loss function based on generalized mutual information.
In addition, as an essential ablation study, we consider vanilla uniform regularization.
(9) Unif-reg: we apply the Jacobian regularizer on all examples with equal strength, and tune the strength to get the best possible validation accuracy.
3. S IMULATING HETEROSKEDASTIC AND IMBALANCED DATASETS ON CIFAR
Setting.
Unlike previous works that test on uniform random or asymmetric noise, which is often not the case in reality, in this paper we test our method on more realistic noisy settings, as suggested by Patrini et al.
; Zhang & Sabuncu (2018).
In order to simulate heteroskedasticity, we only corrupt semantically-similar classes.
For CIFAR-10, we exchange 40% of the labels between classes ‘cat’ and ‘dog’, and between ‘truck’ and ‘automobile’.
CIFAR-100 has 100 classes grouped into 20 super classes.
For each class of the 5 classes under the super class ‘vehicles 1’ and ‘vehicles 2’, we corrupt the labels with 40% probability uniformly randomly to the rest of four classes under the same super class.
As a result, the 10 classes under super class ‘vehicle 1’ and ‘vehicle 2’ have high label noise level and the corruption are only within the same super class.
Heteroskedasticity of the labels and imbalance of the inputs commonly coexist in the real world settings.
HAR can take both of them into account.
To understand the challenge imposed by the entanglements of heteroskedasticity and imbalance, and compare HAR with the aforementioned baselines, we inject data imbalance concurrently with the heteroskedastic noise.
We remove samples from the corrupted classes to simulate the most difﬁcult scenario — the rare and noisy groups overﬁt signiﬁcantly.
(A more benign
[Table]
interaction between the noises and imbalance is that the rare classes have lower noise level, we defer it to Appendix C.5.)
We use the imbalance ratio to denote the frequency ratio between the frequent (and clean) classes to the rare (and corrupted) classes.
We consider imbalance ratio to be 10 and 100.
Result.
Table 1 summarizes the results.
Since examples from rare classes tend to have larger training and validation loss regardless of whether the labels are correct or not, noise-cleaning based methods might drop excessive examples with correct labels.
We examined the noise ratio of dropped samples for INCV under the setting of imbalance ratio equals 10.
Among all dropped examples, there is only 19.2% of true noise examples.
In addition, the rare class examples selected still have 29.8% of label noise.
This explains that the signiﬁcant decrease of accuracies of Co-teaching and INCV on corrupted and rare classes.
Reweighting-based methods tend to suffer from the loss of accuracy in other more frequent classes, which is aligned with the ﬁndings in Cao et al.
While the aforementioned baselines struggle to deal with heteroskedasticity and imbalance together, HAR is able to put them under the same regularization framework and achieve signiﬁcant improvements.
Notably, HAR also shows improvement over uniform regularization with optimally tuned strength.
This clearly demonstrates the importance of introducing adaptive regularization among all examples for a better trade-off.
A more detailed ablation study on the trade-off between training accuracy and validation accuracy can be found in Section 3.2.
Ablation study.
We disentangle the problem setting to show the effectiveness of our uniﬁed framework.
In Appendix C.1 , we demonstrate state-of-the-art results on a balanced and heteroskedastic dataset where only the uncertainty part is effective.
In Appendix C.2, we show state-of-the-art results on an imbalanced and hemosekdeastic dataset where only the density part is effective.
3. E VALUATION ON W EB V ISION WITH REAL - WORLD HETEROGENEITY
WebVision (Li et al., 2017) contains 2.4 million images crawled from Google and Flickr using 1, labels shared with the ImageNet dataset.
Its training set is both heteroskedastic and imbalanced (detailed statistics can be found in (Li et al., 2017)), and it is considered as a popular benchmark for noise robust learning.
As the full dataset is very large, we follow (Jiang et al., 2018) to use a mini version, which contains the ﬁrst 50 classes of the Google subset of the data.
Following the standard protocol (Jiang et al., 2018), we test the trained model on the WebVision validation set and the ImageNet validation set.
We use ResNet-50 for ablation study and InceptionResNet-v2 for a fair comparison with the baselines.
We report results comparing against other state-of-the-art approaches in Table 3.
Strikingly, HAR achieves signiﬁcant improvement.
Ablation study.
We demonstrate the trade-off between training accuracy and validation accuracy on mini WebVision with various uniform regularization strength and HAR in Table 2.
It’s evident that when we gradually increase the overall uniform regularization strength, the training accuracy continues to decrease, and the validation accuracy reaches its peak at 5e-2.
While a strong regularization could improve generalization, it reduces deep networks’ capacity to ﬁt the training data.
However, with our proposed HAR, we only enforce strong regularization on a subset so that we improve the generalization on noisier groups while maintaining the overall training accuracy not affected.
R ELATED W ORK
Our work is closely related to the following methods and directions.
HAR demonstrates signiﬁcant improvements over the previous state-of-the-arts.
[Table]
Noise-cleaning.
The key idea of noise-cleaning is to identify and remove (or re-label) examples with wrong annotations.
The general procedure for identifying mislabeled instances has a long history (Brodley & Friedl, 1999; Wilson & Martinez, 1997; Zhao & Nishida, 1995).
Some recent works tailored this idea for deep neural networks.
Veit et al.
(2017) trains a label cleaning network on a small set of data with clean labels, and uses this model to identify noises in large datasets.
To circumvent the requirement of a clean subset, Malach & Shalev-Shwartz (2017) train two networks simultaneously and perform update steps only in case of disagreement.
Similarly, in co-teaching (Han et al., 2018), each network selects a certain number of small-loss samples and feeds them to its peer network.
Chen et al.
(2019) further extends the co-training strategy and comes up with an iterative version that performs even better empirically.
Recently Song et al.
(2020) discovers that it is not necessary to maintain two networks.
Removing examples whose training loss exceeds a certain threshold before learning rate decay can also get robust performance.
Reweighting.
Reweighting training data has shown its effectiveness on noisy data (Liu & Tao, 2015).
Its challenge lies in the difﬁculty of weights estimation.
Ren et al.
(2018) proposes a meta-learning algorithm to assign weights to training examples based on its gradient direction with the one on a clean validation set.
Recently, Shu et al.
(2019) proposes to learn an explicit loss-weight function to mitigate the optimizing issue of (Ren et al., 2018).
Another line of work resorts to curriculum learning by either designing an easy-to-hard strategy of training (Guo et al., 2018) or introducing an extra network (Jiang et al., 2018) to assign weights.
Noise-cleaning and reweighting methods usually rely on the empirical loss to determine if a sample is noisy.
However, when the dataset is heteroskedastic, each example’s training/validation loss no longer correlates well with its noise level.
In such cases, we argue that changing the strength of regularization is a more conservative adaption and suffers less from uncertain estimation, compared to changing the weights of losses (Please refer to Section C.6 for empirical justiﬁcations).
Robust loss function.
Another line of works has attempted to design robust loss functions (Ghosh et al., 2017; Xu et al., 2019; Zhang & Sabuncu, 2018; Patrini et al., 2017; Cheng et al., 2017; Menon et al., 2016).
They usually rely on prior assumption about latent transition matrix that might not hold in practice.
On the contrary, we focus on more realistic settings.
Regularization.
Regularization based techniques have also been explored to combat label noise.
Li et al.
(2019) proves that SGD with early stopping is robust to label noise.
Hu et al.
(2020) provides theoretical analysis of two additional regularization methods.
While these methods consider a uniform regularization on all training examples, our work emphasizes on adjusting the weights of regularizers in search of a better generalization than uniform assignment.
C ONCLUSION
We propose a uniﬁed framework (HAR) for training on heteroskedastic and imbalanced datasets.
Our method achieves signiﬁcant improvements over the previous state-of-the-arts on a variety of benchmark vision and language tasks.
We provide theoretical results as well as empirical justiﬁcations by showing that ambiguous, mislabeled, and rare examples all beneﬁt from stronger regularization.
We further provide the formula for optimal weighting of regularization.
Heteroskedasticity of datasets is a fascinating direction worth exploring, and it is an important step towards a better understanding of real-world scenarios in the wild.
A P ROOFS OF T HEOREM
We prove a general theorem here.
In particular, we have the general theorem below.
Theorem 2.
Assume that f (cid:63) , q, I ∈ W 22 .
Suppose (1) (cid:96) ( a, y ) is convex and three times continously differentiable with respect to a , (2) there exist constants 0 < c < C < ∞
such that c ≤ I ( X ) ≤ C almost surely, and (cid:15) (cid:44) ∇ a (cid:96) ( f (cid:63) ( X ) , Y ) satisﬁes E [ (cid:15) | X ] = 0 , E [ (cid:15) 2 | X ]
= I ( X ) and E [ (cid:15)
| X ] < ∞ almost surely.
Let r ( t ) =
− 1 / ( q ( t ) I ( t )) and L 0 = (cid:82) ∞
−∞ 14 exp( − 2 | t | ) dt .
If we choose λ = C 0 n − 2 /for
some constant C 0 > 0 , the asymptotic mean squared error of ˆ f by equation 4 is
[Equation]
in probability, where C n is a scalar that only depends on n .
It is easy to check that the logistic loss satisﬁes the condition of the theorem.
The proof strategy of Theorem 2 is adopted from the proof of Theorem 2 of (Wang et al., 2013) by generalizing it from the least square loss to logistic loss.
The high level idea is to reformulate ˆ f as solutions to ordinary differential equations.
Let ( γ v , h v ) be the (normalized) solution of the following equation
[Equation]
We deﬁne the the leading term of ˆ f − f (cid:63) as S n,λ ( f (cid:63) ) as
S n,λ ( f (cid:63)
[Equation]
i K X
i − W λ f (cid:63) , where
[Equation]
By Proposition 2.1 and Theorem 3.4 of Shang et al.
(2013), we have
[Equation]
Following the same proof of Theorem 2 of (Wang et al., 2013), we can simplify the deﬁnition of K t and W λ as
[Equation]
where J ( t, s )
= βρ ( s ) Q (cid:48)
β ( s ) L 0 ( β | Q β ( t ) − Q β ( s ) | ) and Q β ( t, s ) =
(cid:82) t 0 ( r ( s ) ρ ( s ))
− 1 / 2 (1 +
O ( β − 1 )) ds and β = 1 / √ λ .
Plugging equation 11 into equation 10, we then have
[Equation]
0 λ 2 r 2 ( t ) (cid:20) d dt ( ρ ( t )( f ∗ ) (cid:48) ( t ))
(cid:21)+ L 0 r ( t ) 1 / 2 ρ ( t ) − 1 / 2 dt
B I MPLEMENTATION DETAILS
We develop our core algorithm in PyTorch (Paszke et al., 2017).
Implementation details for CIFAR.
We follow the simple data augmentation used in (He et al., 2016) with only random crop and horizontal ﬂip.
We use ResNet-32 as our base network and repeat all experiments for 3 runs.
We use standard SGD with momentum of 0.9, weight decay of 1 × 10 −for training.
The model is trained with a batch size of 128 for 120 epochs.
We anneal the learning rate by a factor of 10 at 80 and 100 epochs.
We group the data by class labels, and by default we split D equally and randomly into D train and D val .
As for the Jacobian regularizer, we sum over the frobenius norm of the gradients of all normalization layers’ (BN by default) activations with respect to the classiﬁcation loss.
For experiments of HAR, we tune λ so that the largest enforced regularization strength ( λτ i ) is 0.1.
We train each model with 1 NVIDIA GeForce RTX 2080 Ti.
Implementation details for IMDB-review.
We train a two-layer bidirectional LSTM (Huang et al., 2015) with 256 units followed with 0.5 dropout before the linear classiﬁer.
The network is trained for 20 epochs with Adam optimizer (Kingma & Ba, 2014).
For HAR, we tune λ so that the largest enforced regularization strength ( λτ i ) is 0.1.
We train each model with 1 NVIDIA GeForce RTX 2080 Ti.
Implementation details for WebVision.
We use the standard data augmentation same as (He et al., 2016) including random crop and horizontal ﬂip.
For mini WebVision, We train the network forepochs using standard SGD with a batch size of 128.
The initial learning rate is 0.1 and is annealed by a factor of 10 at epoch 60 and 90.
For full WebVision, We train the network for 50 epochs using standard SGD with a batch size of 256.
The initial learning rate is 0.1 and is annealed by a factor of 10 at epoch 30 and 40.
For experiments of HAR, we tune λ so that the largest enforced regularization strength ( λτ i ) is 0.1.
We train each model with 8 NVIDIA Tesla V100 GPUs.
Runtime analysis.
Because the pre-trained model only trains on half of the training data and is only done once, the run-time of HAR is at most twice of the time for ERM.
Many baselines in our paper use sophisticated pipelines and are slower than HAR.
For example, INCV trains 2 models simultaneously for 4 times from random initialization to get a clean training set.
MW-Net has a very slow convergence rate, which is a common issue for meta-learning.
C A DDITIONAL R ESULTS C. S IMULATING HETEROSKEDASTIC NOISE ON CIFAR
[Table]
We study the uncertainty part of HAR by testing under the setting with only heteroskedastic noise.
The type of noise injection is the same as Section 3.
[Table]
We report the top-1 validation accuracy of various methods in Table 4.
Aligned with our analysis in
Section 4, we observe that both noise-cleaning and reweighting based methods don’t get a comparable accuracy on noisy classes with applying strong regularization ( λ = 0 . 1) under this heteroskedastic setting.
We observe the behavior that too strong regularization impede the model from ﬁtting informative samples, thus it could lead to a decrease on clean classes’ accuracy.
On the contrary, too weak regularization leads to overﬁtting the noisy examples thus the accuracy on noisy classes do not reach the optimal.
Interestingly, we ﬁnd that even the well-studied CIFAR-100 dataset has intrinsic heteroskedasticity and HAR can improve over uniform regularization to some extent.
Please refer to Appendix C.4 for the results on CIFAR-100 and Appendix C.3 for results on IMDB-review.
C. S IMULATING DATA IMBALANCE ON CIFAR
We study the density part of HAR by testing under the setting with only data imbalance.
We follow the same setting as Cao et al.
(2019) to create imbalanced CIFAR.
Long-tailed imbalance follows an exponential decay in sample sizes across different classes.
For step imbalance setting, all rare classes have the same sample size, as do all frequent classes.
HAR achieves better results than LDAM-DRW and is comparable to recent state-of-the-art methods under the imbalanced setting.
[Table]
C. S IMULATING HETEROSKEDASTIC NOISE ON IMDB- REVIEW .
IMDB-review dataset has a total of 50,000 (25,000 positive and 25,000 negative reviews) movie reviews for binary sentiment classiﬁcation (Maas et al., 2011).
To simulate heteroskedastic noise for this binary classiﬁcation problem, we project 5% of the labels of negative reviews to positive, and 40% in the reverse direction.
Table 6 summarizes the results.
The proposed HAR outperforms the
ERM baseline with various strength of uniform regularization.
[Table]
C. E VALUATION ON CIFAR-100 WITH REAL - WORLD HETEROSKEDASTICITY
It is acknowledged that CIFAR-100 training set contains noisy examples.
For instance, some “tiger” examples are labeled as “leopard” (“tiger” is a deﬁned class as well).
There are also noisy examples that contain multiple objects, or are more ambiguous in terms of indentity (Song et al., 2020).
We ﬁnd that HAR can improve over uniform regularization on the well-studied CIFAR-100 due to its heteroskedasticity and the results are reported in Table 7. [Table] [Table]
[Table]
C. S IMULATING HETEROSKEDASTIC AND IMBALANCED DATASETS ON CIFAR
As mentioned in Section 3.
[Table]
we consider another variant of heteroskedastic and imbalanced dataset such that the rare classes have low noise level.
To simulate this setting, we make the clean classes have fewer labels than the corrupted classes on the heteroskedastic CIFAR-10 we created in Section C.1.
Table 8 summarizes the results.
For the setting of imbalance ratio equals 10, INCV automatically drops 34.1% of examples from the clean and rare classes, which results in a decrease of mean accuracy on the rare and clean classes.
HAR is able to achieve improvements on both noisy classes and rare classes by enforcing the optimal regularization
[Table]
[Table]
C. C OMPARING THE EFFECT OF WEIGHTS ON LOSSES AND REGULARIZERS
As discussed in Section
[Table]
we train several classiﬁers with alternative weights selection scheme which are not optimal.
We consider the following two alternatives.
(1) random : we draw the weights from a uniform distribution with the same mean as the weights of MW-Net and HAR.
(2) inverse : we take the inverse of the weights learned by MW-Net and HAR and then normalize them to ensure the average reweighting/regularization strength remains the same.
We conducted experiments on the heteroskedastic CIFAR-10 introduced in Section C.1 and the results are summarized in Table 9.
We could conclude that changing the weights of the regularizer is a more conservative adaption and less susceptible to uncertain estimation than reweighting [Table] [Table] C. V ISUALIZATION
In order to better understand how the proposed HAR works on real-world heteroskedastic datasets
[Table]
we plot the per-class key statistics used by HAR and validation errors in Figure 4.
We observe that
HAR outperforms the tuned uniform regularization baseline on the majority of the classes.
[Figure]