W ASSERSTEIN D ISTRIBUTIONAL N ORMALIZATION : N ONPARAMETRIC S TOCHASTIC M ODELING FOR H ANDLING N OISY L ABELS
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose a novel Wasserstein distributional normalization (WDN) algorithm to handle noisy labels for accurate classiﬁcation.
In this paper, we split our data into uncertain and certain samples based on small loss criteria.
We investigate the geometric relationship between these two different types of samples and enhance this relation to exploit useful information, even from uncertain samples.
To this end, we impose geometric constraints on the uncertain samples by normalizing them into the Wasserstein ball centered on certain samples.
Experimental results demonstrate that our WDN outperforms other state-of-the-art methods on the Clothing1M and
CIFAR-10/100 datasets, which have diverse noisy labels.
The proposed WDN is highly compatible with existing classiﬁcation methods, meaning it can be easily plugged into various methods to improve their accuracy signiﬁcantly
I NTRODUCTION
The successful results of Deep Neural Networks (DNNs) on supervised classiﬁcation task heavily relies on accurate and high-quality label information.
However, annotations for large-scale datasets is extremely expensive and time-consuming task.
While obtaining high-quality datasets is too costly, many enterprises uses alternative methods such as crowd-sourcing platforms Yu et al.
(2018) for obtaining large-scaled datasets which inevitably leads noisy labels in annotated samples.
Among numerous methods to deal with such noisy labeled data, some recent studies actively adopt the small loss criterion which is a powerful tool for constructing models that are not susceptible to noise corruption.
In this learning scheme, a neural network is trained using easy samples ﬁrst in the early stages of training.
Harder samples are then gradually selected to train mature models as training proceeds.
Jiang et al.
(2018) suggested collaborative learning models, in which a mentor network delivers data-driven curriculum loss to a student network.
Han et al.
Yu et al.
(2019) proposed dual networks to generate gradient information jointly using easy samples and employed this information to allow the networks to teach each other.
Wei et al.
(2020) adopted a disagreement strategy where the gradient information to update is decided by disagreement between dual networks.
Han et al.
(2020) implemented accumulated gradients to pull optimization back from over-parameterization to generalization.
We tackle to solve major issues raised from previous works based on small-loss criterion: In comprehensive experiments, these works gained empirical insight to network behavior under noisy labels, but theoretical and quantitative explanation have not been investigated. In this paper, we give strong theoretical/empirical explanations to understand the network under noisy labels. Speciﬁcally, we present an in-depth analysis of small loss criteria in a probabilistic sense.
In particular, we exploit the stochastic properties of noisy labeled data and develop probabilistic descriptions of data under small loss criteria as follows.
Let P be a probability measure for the pre-softmax logits of the training samples, l be an objective function for classiﬁcation, and 1 {·} be an indicator function.
Then, our central object to deal with is a truncated measure deﬁned as
[Equation]
where X and Y , which are sampled from µ | ζ and ξ | ζ , denote uncertain and certain samples deﬁned in the pre-softmax feature space 1 ( i .
e ., R d ) , respectively.
In equation 1, µ and ξ denote the probability measures of uncertain and certain samples, respectively, and ζ is a constant.
Most previous works have focused on the usage of Y and the sampling strategy of ζ , but poor generalization capabilities based on the abundance of uncertain samples X has not been thoroughly investigated, even though these samples potentially contain important information.
To understand the effect of noisy labels to generalization bounds, we provide the concentration inequality of uncertain measure µ which renders the probabilistic relation between µ and ξ and learnability of network under noisy labels. While major number of previous methods Han et al. ; Wei et al.
; Li et al. (2019a)
; Yu et al. (2019) require additional dual network to guide misinformed noisy samples, the scalability is not guaranteed due to the existence of dual architecture which have identically same number of parameters compared to base network. To alleviate this problem, in this paper, the main goal is set to build a statistical machinery which should be fully non-parametric, and simple to implement, and most importantly, computationally efﬁcient to reduce the computational complexity of previous methods while maintaining the concept of small-loss criterion.
Based on the empirical observation of ill-behaved certain/uncertain samples, we propose the gradient ﬂow in Wasserstein space which can be induced by simulating non-parametric stochastic differential equation (SDE) with respect to Ornstein-Ulenbeck type to control the ill-behaved dynamics. The reason for selecting such dynamics will be thoroughly discussed in following sections.
Overall, the key contributions of our work are as follows.
• We have investigated and veriﬁed theoretically/empirically that there exists strong correlation between model conﬁdence and statistical distance between X and Y .
The poor performance on test dataset is induced when the upper-bound of 2-Wasserstein distance W 2 ( µ, ξ ) ≤ ε ( i .
e ., distributional distance between certain and uncertain samples) is drastically increased.
Due to the empirical nature of upper-bound ε , it can be used as an estimator to determine if a network suffers from over-parameterization.
• Based on empirical observations, we developed simple, non-parametric , and computationally efﬁcient stochastic models to control the ill-behaved sample dynamics.
As a primal object, we propose the stochastic dynamics of gradient ﬂow ( i
e ., , Ornstein-Ulenbeck process) to simulate simple/non-parametric stochastic differential equation.
Thus our method do not require any additional learning parameters.
• We provide various important theoretical results.
First, the controllable upper-bound ε with inverse exponential ratio is suggested which indicates that our method can efﬁciently control the diverging effect of Wasserstein distance.
Second, the concentration inequality of transported uncertain measure is suggested which clearly renders the probabilistic relation between µ and ξ .
R ELATED W ORK
Curriculum Learning & Small-loss Criterion.
To handle noisy labels, Han et al.
Yu et al.
; Jiang et al.
; Wei et al.
; Lyu & Tsang (2020a); Han et al.
(2020) adopted curriculum learning or sample selection frameworks.
However, these methods only consider a small number of selected samples, where large portion of samples are excluded at the end of the training.
This inevitably leads to poor generalization capabilities.
However, this conﬂicts with sample selection methods because a large portion of training samples are gradually eliminated.
By contrast, our method can extract useful information from unselected samples X ∼ µ ( i .
e ., uncertain samples) and enhance these samples ( e .
g .
, X (cid:48) ∼ F µ ) for more accurate classiﬁcation.
Chen et al.
(2019) iteratively apply cross-validation to randomly partitioned noisy labeled data to identify most samples that have correct labels.
To generate such partitions, they adopt small-loss criterion for selecting samples.
Loss Correction & Label Correction.
Patrini et al.
(2017a)
; Hendrycks et al.
; Ren et al.
(2018) either explicitly or implicitly transformed noisy labels into clean labels by correcting classiﬁcation losses.
Unlike these methods, our method transforms the holistic information from uncertain samples into certain samples, which implicitly reduces the effects of potentially noisy labels.
While correction of label noisy by modifying the loss-dynamics do not perform well under extreme noise environments, Arazo et al.
(2019) adopt label augmentation method called MixUp Zhang et al.
Distillation.
Li et al.
(2019b) updated mean teacher parameters by calculating the exponential moving average of student parameters to mitigate the impact of gradients induced by noisy labels.
Lukasik et al.
(2020) deeply investigated the effects of label smearing for noisy labels and linked label smoothing to loss correction in a distillation framework.
Similar to these methods, our method leverages the useful properties of distillation models.
We set ν as a pivot measure, which guides our normalization functional F µ for uncertain measures.
This is similar to self-distillation because uncertain training samples are forced to be normalized to those of past states.
Other methods.
Lee et al.
(2019) induced a robust generative classiﬁer based on pre-trained deep models.
Similar to our method, Damodaran et al.
(2019) designed a constraint on the Wasserstein space and adopted an adversarial framework for classiﬁcation models of noisy labeled data by implementing semantic Wasserstein distance.
Pleiss et al.
(2020) identify noisy labeled samples by considering AUM statistics which exploits differences in training dynamics of clean and mislabeled samples.
In most recent work, Li et al.
(2019a) adopts semi-supervised learning (SSL) methods to deal with noisy labels where the student network utilizes both labeled/unlabeled samples to perform semi-supervised learning guided by the other teacher network.
D ISTRIBUTIONAL N ORMALIZATION
Because our main target object is a probability measure (distribution), we ﬁrst deﬁne an objective function in a distributional sense.
Let l be cross entropy and ˆ r be a corrupted label random vector for an unknown label transition matrix from a clean label r which is independent of X , with label transition matrix Q .
Then, a conventional objective function for classiﬁcation with noisy labels can be deﬁned as follows:
[Equation]
However, due to the signiﬁcant changes in label information, the conventional objective function deﬁned in equation 2 cannot be used for accurate classiﬁcation.
Instead of directly using uncertain samples X ∼ µ as in previous works, we normalize µ in the form of a metric ball and present a holistic constraint.
For a clear mathematical description, we ﬁrst introduce the following deﬁnition.
Deﬁnition 1.
(Wasserstein ambiguity set)
Let P 2 ( R d ) =
{ µ : E µ d 2 E ( x 0 , x ) < ∞ , ∀ x 0 ∈ R d } be a 2-Wasserstein space, where d denotes the number of classes, d E is Euclidean distance deﬁned on R d .
Then, we deﬁne a Wasserstein ambiguity set (i.e., metric ball) in this space as follows:
[Equation]
where W 2 denotes the 2 -Wasserstein distance and ν is the pivot measure.
Then, we propose a new objective function by imposing geometric constraints on µ as follows:
[Equation]
where F : P 2 ( R d ) → P 2 ( R d ) is a functional for probability measures, which assures the constraint on F µ ( i .
e ., F µ ∈ B W 2 ( ν, ε ) ) and our main objective.
The right-hand side of equation equationis equivalent vectorial form of distributional form in left-hand side.
While our main objects are deﬁned on pre-softmax, both probability measures µ θ and ξ θ is parameterized by neural network with parameters θ .
This newly proposed objective function uses the geometrically enhanced version of an uncertain measure F µ with a certain measure ξ .
In equation 4, we argue that the pivot measure ν is considered to be a good candidate if it can be represented as a mini-batch containing the best combination of certain samples.
Therefore, ν can be determined as ν =
arg min J [ ξ k (cid:63) ] , where ξ k denotes a certain measure at the k -th iteration and k (cid:63) ∈
I k − 1 = { 1 , · · · , k − 1 } .
In other words, our method ﬁnds the best probability measure that represents all certain samples so far, where the uncertain measures can be normalized into the 2-Wasserstein Ball centered on ν .
In equation 4, the
Wasserstein constraint on F µ enforces uncertain measures statistically resemble ν from a geometric perspective ( i .
e .,
W 2 ( ν, F µ ) ≤ ε ).
Now, an important question naturally stems from the aforementioned analysis: how can we select the optimal radius ε ?
Clearly, ﬁnding an F that induces a small ε ≈ 0 is suboptimal because F µ ≈ ν and using objective function J [ F µ ≈ ν ] can lead to the following critical problem.
As the optimization process proceeds, enhanced uncertain samples X (cid:48) ∼ F µ contribute less and less, because it is statistically identical to ν , meaning our objective in equation 4 would receive little beneﬁts from these uncertain samples.
By contrast, if we adopt a large radius for ε , enhanced uncertain samples [Figure] will be statistically and geometrically unrelated to ν , which causes the normalized measure F µ to yield large losses and violates our objective.
To overcome two problems above and select the best radius, we make a detour, i e ., a Gaussian measure, for cutting the path between ν and F µ ( i e ., ν → N ( m ν , Σ ν )
→ F µ ) rather than directly calculating the geodesic between ν and F µ ( i e ., ν → F µ ).
Speciﬁcally, we decompose the original constraint in equation 4 into two terms using the triangle inequality of the Wasserstein distance:
[Equation]
The ﬁrst intrinsic statistics term sets a detour point as a Gaussian measure, for which the mean and covariance are the same as those for ν ( i .
e .,
m ν =
E Y ∼ ν
[ Y ] and Σ ν =
Cov Y ∼ ν
[ Y ] ).
The
Wasserstein upper bound of this term is only dependent on the statistical structure of ν because ( m ν , Σ ν ) is dependent on ν .
Thus, this term induces a data-dependent, non-zero constant upper bound whenever ν (cid:54) =
N and can prevent the bound from collapsing to ε → 0 , regardless of F .
The second normalization term represents our central objective.
F facilitates geometric manipulation in the Wasserstein space and makes the holistic statistics of F µ mimic those of the best probability measure ν .
Using these terms, our method can prevent uncertain measure µ from diverging, where µ is normalized onto the Wasserstein ambiguity B W 2 ( ν, ε ) in Fig1.
This results in consistent performance enhancement for our method.
We will discuss this property in greater detail using Fig.2 in Section 4.
3. W ASSERSTEIN N ORMALIZATION
In the previous section, we present a novel objective function that imposes a geometric constraint on µ such that the transformed measure F µ lies in B W 2 ( ν, ε ) for ν .
Now, we specify F and relate it to the Gaussian measure (generally Gibbs measure).
For simplicity, we denote N ν = N
( m ν , Σ ν ) .
Proposition 1.
F : R + ×P 2 → P 2 is a functional on the probability measure such that F [ t, µ ] = µ t , where dµ t =
p t d N ν , d N ν = dq t dx , and µ t is a solution to the following continuity equations:
[Equation]
which is read as ∂ t p ( t, x ) = ∇ · ( p ( t, x ) ∇ log q ( t, x )) in a distributional sense.
Then, a uniquely deﬁned functional F t [ · ] = F [ t, · ] normalizes µ onto B W 2 ( N ν , e − t K 2 ( µ )) , where K 2 ( µ ) > 0 is a constant that depends on µ .
It is well known that the solution to equation 6 induces a geodesic in the 2-Wasserstein space (Villani (2008)), which is the shortest path from µ = µ t =0 to N ν .
The functional F t generates a path for µ t , in which the distance is exponentially decayed according to the auxiliary variable t and constant
K 2 , meaning W 2 ( N ν , F t µ ) ≤ K 2 e − t .
Thus, by setting a different t , we can efﬁciently control the second term in equation 5.
However, it is typically intractable to compute the partial differential equation (PDE) in equation 6.
Require: α ∈ [0 , 0 . 2] , (cid:37) ∈ [0 . 1 , 0 .
65] , T = 64 , ∆ t = 10 − 4 , τ = 0 .
001 , for k = 1 to K ( i . e ., the total training ) do Select
{ Y nk }
[Table]
[Table]
∼
[Table]
[Table] [Table] [Table] [Table]
[Table]
if J
[Table]
then [Table] [Table] and
[Table]
[ Y
[Table]
] end if 3)
[Table]
the
[Table]
Solve the Ricatti equation T
[Table] [Table]
end
[Table]
[Table]
[Table]
[Table]
end for
To solve this problem, we adopt particle-based stochastic dynamics, which enables tractable computation.
The PDE in equation 6 naturally induces the multi-dimensional Ornstein-Ulenbeck process , which can be approximated using particle-based dynamics.
In particular, we draw N (1 − (cid:37) ) uncertain samples from a single batch of N samples using equation 1 for hyper-parameter 0 ≤ (cid:37) ≤ 1 .
We then simulate a discrete stochastic differential equation (SDE) for each particle using the Euler-Maruyama scheme as follows:
[Equation]
where φ ( X t ; m ν )
=
τ 2 d 2 E ( X t , m ν ) , n ∈ { 1 · · · , N (1 − (cid:37) ) } , d E is a Euclidean distance, and N is a single mini-batch size.
We selected OU process as our stochastic dynamic due to the following reasons:
First, we want to build computationally efﬁcient, and non-parametric method to estimate/minimize the second term of equation 5.
The SDE in equation 7 corresponding OU process have simple form with ﬁxed drift and diffusion terms which is invariant over times which makes us to induce the nonparametric representations of simulation of SDE.
Second, when estimating empirical upper-bound of
Wasserstein distance, OU process allows us to use explicit form called Meheler’s formula which can be efﬁciently estimated (Please refer to Appendix for more details).
The overall procedure for our method is summarized in Algorithm 1.
3. W ASSERSTEIN M OVING G EODESIC A VERAGE
In our experiments, we observe that the best measure ν is not updated for a few epochs after the training begins.
This is problematic because ν diverges signiﬁcantly from the current certain measure ξ k , which is equivalent to the normalized measure F µ k diverging from ξ k , meaning X T and Y become increasingly statistically inconsistent.
To alleviate this statistical distortion, we modify detour measure from N ν to other Gaussian measure, which allows us to capture the statistics of both ξ k and ν .
Inspired by the moving average of Gaussian parameters in batch normalization Ioffe & Szegedy (2015), we propose the Wasserstein moving geodesic average .
Speciﬁcally, we replace
Gaussian parameters { m ν , Σ ν } with { m α , Σ α } such that m α
=
(1 − α ) m ν + α m ξ k and Σ
α =
((1 − α )
I d + α T ) Σ
ν ((1 − α )
I d + α T ) , where T is a solution to the Riccati equation T Σ
ν T = Σ ξ k .
Therefore our ﬁnal detour Gaussian measure is set to N αν :=
N ( m ( α ) , Σ( α )) , 0 ≤ α ≤ 1 2 .
T HEORETICAL A NALYSIS
In equation 5, we select the detour point as a Gaussian measure because this measure can provide a statistical structure, which is similar to that of the optimal ν .
In addition to this heuristic motivation, setting a detour point as a Gaussian measure (Gibbs measure) also provides theoretical advantages, e .
g ., the theoretical upper bound of the Wasserstein constraint terms.
In this section, we investigate the explicit upper bounds of two terms in equation 5, which are naturally induced by the SDE.
Proposition 2.
A scalar 0 < β < ∞ exists and depends on ν , resulting in the following inequality:
[Equation]
where λ max (Σ ν ) denotes the maximum eigenvalue of the covariance matrix Σ ν and for some constant 0 < K 1 < ∞ , we have K 1 ( ν ) =
(cid:112) dβλ max (Σ ν ) + (cid:107) E ν Y (cid:107) 2 which is only dependent on ν .
Intuitively, K 2 ( µ ) can be interpreted as an indicator that tells us how the uncertain measure µ is diffused, whereas the designed term e − t K 2 ( µ ) controls the upper bound of the Wasserstein distance using a variable t .
The other term K 2 ( ν ) does not vanish even with a very large t , which assures a non-collapsing upper-bound ε .
Proposition 3.
(Concentration inequality for the normalized uncertain measure).
Assume that there are some constants T ∈ [ 1 η , ∞ ) , η ≥
such that the following inequality holds:
[Equation]
for A ∈ Sym + d and D ( A , Σ ν ) ≤ aη for some a > 0 with any metric D deﬁned on Sym + d .
In this case, there is a δ such that the following probability inequality for an uncertain measure is induced:
[Equation]
where σ denotes a soft-max function.
In equation 10, we show that the label information induced by the normalized uncertain measure is close to that of most certain measure E ν [ σ ] , where the upper bound is exponentially relative to the initial diffuseness of µ ( i .
e ., K 2 ( µ )).
Because the upper bound of the probability inequality does not collapse to zero and F T µ is concentrated around the most certain labels ( i .
e .,
E ν [ σ ] ), the uncertain sample X T ∼ F T µ helps our method avoid over-parameterization.
4. E MPIRICAL U NDERSTANDINGS
We investigated the theoretical upper bound of the Wasserstein ambiguity ( i .
e ., radius of the Wasserstein ball) for F µ and its corresponding probability inequality.
To provide more in-depth insights into the proposed method, we approximate the upper bound and demonstrate that our Wasserstein normalization actually makes neural networks more robust to label noise.
As we veriﬁed previously, according to Proposition 2, the following inequality holds:
[Equation]
[Equation] Because the ﬁrst term K 1 ( ν ) is constant, dependent on ν , and generally small compared to the second term with t ≤ T , we only examine the behavior of the second term K 2 ( ν ) + K 2 ( F t µ ) , which can be efﬁciently approximated using a simple form.
Because our detour measure is Gaussian, we have the following inequality for any h ∈ C ∞
( R d ) 3 :
[Equation]
(12) where this equality holds if h is selected to induce a supremum over the set C ∞ 0 .
For approximation, we simply consider h ( X ) =
(cid:107) X (cid:107) 2 as a test function.
In this case, the following inequality naturally holds:
[Equation]
ε .
Thus, ˆ ε can be considered as an approximation of the theoretical upper bound ε suggested in Proposition 2.
We now investigate the effects of Wasserstein normalization based on the developed tool ˆ K 2 ( µ ) .
(1) The proposed WDN ensures that the Wasserstein ambiguity is bounded.
We examine the relation between ˆ ε and test accuracy in an experiment using the CIFAR-10 dataset with symmetric noise at a ratio of 0 . 5 .
Fig.2 presents the landscape for the log 10 -scaled cumulative average of ˆ ε and test accuracy over epochs.
The red dotted lines represent the landscape of the vanilla network with cross-entropy loss, where ˆ ε k
= ˆ K 2 ( ν k )
+ ˆ K 2 ( F t =0 µ k )
and k is the epoch index.
In this case, the time constant t is set to zero, because Wasserstein normalization is not employed for the vanilla network.
The black lines indicate the landscape of the proposed method, where ˆ ε k
= ˆ K 2 ( ν k )
+ ˆ K 2 ( F t = T µ k ) in this case.
It is noteworthy that the test accuracy of the vanilla network begins to decrease after 13epochs (red-dotted vertical lines in the top-right plot), whereas the Wasserstein ambiguity (i.e., upper bound of the Wasserstein distance) increases quadratically in the top-left plot.
These experimental results verify that the distance between uncertain and most certain measure ( i .
e ., ν ) becomes large in the 2-Wasserstein space without any constraints in vanilla networks.
They also indicate a deﬁnite relationship between Wasserstein ambiguity and test accuracy.
In the proposed WDN, Wasserstein ambiguity can be efﬁciently bounded ( i .
e ., lim sup k ˆ ε k ≈ 2 . 15 ) as the test accuracy continues to increase, even after 13 -epochs.
For detailed analysis, we compute the deviation of an empirical upper bound as follows:
ˆ∆ k
= ˆ ε k − ˆ ε k − 1 .
In the gray regions, the deviation for the vanilla network is grater than 2 .
5 × 10 − 2 , i .
e ., ∆ k > 2 .
5 × 10 − 2 .
Then, its test accuracy begins to drop, as shown in
Fig.2.
In contrast to the vanilla network, the maximum deviation of the proposed WDN is bounded above by a very small value ( sup k ˆ∆ k ≤ 8 × 10 − 3 ).
The proposed WDN helps networks to escape from over-parameterization.
To analyze the behavior of deep neural networks under over-parameterization with and without the proposed WDN, we design several variants of the WDN, which begin at delayed epochs.
The green, orange, and blue curves in the second row of Fig.2 represent the landscapes, when our WDN is applied after k d ∈ { 10 , 15 , 20 } epochs, respectively.
In this experiment, the upper bound ˆ ε k is deﬁned as
[Equation]
Consider k d = 20 , which is represented by the blue dotted vertical lines.
Before our WDN is applied ( i .
e ., k < k d ), the network suffers from over-parameterization, which induces a signiﬁcant performance drop, as indicated by the blue curve in the bottom-right plot.
However, the network rapidly recovers to normal accuracy following Wasserstein normalization ( i .
e ., k ≥ k d ).
Please note that similar behavior can be observed in the green and orange curves.
In particular, the orange curve produces less ﬂuctuations than the blue curve in terms of test accuracy.
This indicates that the proposed WDN can help a network escape from over-parameterization by imposing geometric constraints on the Wasserstein space with proposed method.
The proposed WDN can derive data-dependent bounds according to different noise levels.
Another interesting point in Fig.2 is that all curves, excluding the red curve, converge to speciﬁc numbers 2 .
15 = ε :=
lim inf k ˆ ε k ≤ lim sup k ˆ ε k :
= ¯ ε = 2 . 2 .
The upper bound ¯ ε is neither overly enlarged nor collapsed to zero, while the lower bound ε is ﬁxed for all curves.
We argue that this behavior stems from the geometric characteristics of the proposed method, where the ﬁrst term in equation 5, namely W 2 ( ν, N ν )
∝ ˆ K 2 ( ν ) , is a non-zero data-dependent term that is minimized by the proposed geometric constraint.
Therefore, we can derive the following relationship:
[Equation]
[Equation]
This empirical observation veriﬁes that a detour point, which is set as a Gaussian measure, can induce the data-dependent bound ( ε, ¯ ε ) , where our data-dependent bound can vary according to different [Table] [Table] noise levels and efﬁciently leverage data-dependent statistics.
Fig.2 indicates that classiﬁcation models with more stable data-dependent bounds also induce more stable convergence in test accuracy.
E XPERIMENTS 5. E XPERIMENTS ON THE CIFAR-10/100 DATASET
We used settings similar to those proposed by Laine & Aila (2016); Han et al.
(2018) for our experiments on the CIFAR10/100 dataset.
We used a 9-layered CNN as a baseline architecture with a batch size of 128 .
We used the Adam optimizer with ( β 1 , β 2 ) =
(0 . 9 , 0 . 99) , where the learning rate linearly decreased from 10 − 3 to 10 − 5 .
Synthetic Noise.
We injected label noise into clean datasets using a noise transition matrix Q i,j =
Pr (ˆ r = j | r = i ) , where a noisy label ˆ r is obtained from a true clean label r .
We deﬁned Q i,j by following the approach discussed by Han et al.
For symmetric noise, we used the polynomial, (cid:37) = − 1 .
11 r 2 + 1 .
78 r + 0 .
04 for 0 .
2 ≤ r ≤ 0 .
65 , where r is the noise ratio.
For the asymmetric noise, we set (cid:37) to 0 .
To select the enhanced detour measure, we set α to 0 .
2 for the Wasserstein moving geodesic average in all experiments.
We trained our classiﬁcation model over 500 epochs because the test accuracy of our method continued increasing, whereas those of the other methods did not.
We compared our method with other state-of-the-art methods, including [ MentorNet, Jiang et al.
(2018) ] , [ Co-teaching, Han et al.
(2018) ] , [ Co-teaching+, Yu et al.
(2019) ] , [ GCE, Zhang & Sabuncu (2018) ] , [ RoG, Lee et al.
(2019) ] , [ JoCoR, Wei et al.
(2020) ] , [ NPCL, Lyu & Tsang (2020b) ] , [ SIGUA,
Han et al.
(2020) ] , and [ DivideMix, Li et al. (2019a) ] .
As shown in Table 1, the proposed WDN signiﬁcantly outperformed other baseline methods.
Please note that our WDN utilizes a simple Gaussian measure as a target pivot measure.
Thus, there are potential risks when handling highly concentrated and non-smooth types of noise ( e .
g ., asymmetric noise).
Nevertheless, the proposed WDN still produced accurate results, even with asymmetric noise.
In this case, a variant of our WDN ( i .
e ., WDN cot ) exhibited the best performance.
Open-set Noise.
In this experiment, we considered the open-set noisy scenario suggested by Wang et al.
(2018), where a large number of training images were sampled from other CIFAR-100 dataset; however, these images were still labeled according to the classes in the CIFAR-10 dataset.
We used a 9-layered CNN, which also used in our previous experiment.
For hyper-parameters, we set (cid:37) and α to 0 .
5 and 0 . 2 , respectively.
As shown in Table 2, our method achieved state-of-the-art accuracy.
Collaboration with Other Methods.
Because our core methodology is based on small loss criteria, our method can collaborate with co-teaching methods.
In Han et al.
(2018), only certain samples ( Y ∼ ξ ) were used for updating colleague networks, where the number of uncertain samples gradually decreased until it reached a predetermined portion.
To enhance potentially bad statistics for co-teaching, we taught dual networks by considering a set of samples ( Y, X T ) , where X T ∼ F T µ are uncertain samples enhanced using equation 7.
Table 1 shows the test accuracy results for the proposed collaboration model with a co-teaching network (WDN cot ).
This collaboration model achieved the most accurate performance for the CIFAR100 dataset with asymmetric noise, which veriﬁes that our WDN can be integrated into existing methods to improve their performance signiﬁcantly, particularly when the density of pre-logits is highly-concentrated.
Fig.3 reveals that co-teaching quickly falls into over-parameterization and induces drastic drop in accuracy after the 15 th-epoch.
WDN cot also exhibits a slight accuracy drop.
However, it surpassed the baseline co-teaching method by a large margin ( +7% ) during training.
This demonstrates that our enhanced samples X T can alleviate the over-parameterization issues faced by conventional co-teaching models, which helps improve their accuracy signiﬁcantly.
5. E XPERIMENTS ON A R EAL - WORLD DATASET
To evaluate our method on real-world datasets, we employed the Clothing1M dataset presented by Xiao et al.
(2015), which consists of 1 M noisy, labeled, large-scale cloth images with 14 classes collected from shopping websites.
It contains 50 K, 10 K, and 14 K clean images for training, testing, and validation, respectively.
We only used a noisy set for training; for testing, we used a clean set.
We set α = 0 .
2 and (cid:37) = 0 .
For fair comparison, we followed the settings suggested in previous works.
We used a pre-trained ResNet50 for a baseline architecture with a batch size of 48 .
For the pre-processing steps, we applied a random center crop to 224 × 224 pixels, random ﬂipping and normalization.
We adopted the Adam optimizer with a learning rate starting at 10 − 5 that linearly decayed to 5 × 10 − 6 at 24 K iterations.
Regarding the baseline methods, we compared the proposed method to [ GCE, Zhang & Sabuncu (2018) ] , [ D2L, Ma et al.
(2018) ] , [ FW, Patrini et al.
(2017b) ] , [ WAR, Damodaran et al.
(2019) ] , [ SL, Wang et al.
(2019) ] , [ JOFL, Tanaka et al.
(2018) ] , [ DMI, Xu et al.
(2019) ] , [ PENCIL, Yi & Wu (2019) ] , and [ MLNT, Li et al. (2019b) ] .
Table 3 reveals that our method achieved comparable performance compared to other baseline methods. [Table]
[Table]
C OMPUTATIONAL C OST
As Co-teaching, JoCoR, DivideMix use additional network, the required network parameters are twice ( 8 . 86 M ) that of network Vanilla network ( 4 . 43 M ).
In the Table 4, we compare the average training time for ﬁrst 5 -epochs over various baseline methods under symmetric noise on CIFARdataset.
While the non-parametric methods such as GCE, WDN requires less than 12% additional time, other methods which requires additional network spent more time compared to non-parametric methods.
The averaging time can be varied in different experimental environments.
In the table 4, reported time is estimated based on open-source code authors.
[Table]
C ONCLUSION
We proposed a novel method called WDN for accurate classiﬁcation of noisy labels.
The proposed method normalizes uncertain measures to data-dependent Gaussian measures by imposing geometric constraints in the 2-Wasserstein space.
We simulated discrete SDE using the Euler-Maruyama scheme, which makes our method fast, computationally efﬁcient, and non-parametric.
In theoretical analysis, we derived the explicit upper-bound of the proposed Wasserstein normalization and experimentally demonstrated a strong relationship between this upper-bound and the over-parameterization.
We conducted experiments both on the CIFAR-10/100 and Clothing1M datasets.
The results demonstrated that the proposed WDN signiﬁcantly outperforms other state-of-the-art methods
A O PEN - SOURCE D ATASET
Transition matrix for CIFAR-10/100.
For the experiment summarized in Table 1, we implemented open-source code to generate the noise transition matrix discussed by Han et al.
(2018), as well as the 9-layered CNN architecture ( https://github.com/bhanML/Co-teaching ).
Open-set noise.
For the experiment summarized in Table 2, we used the same dataset for open-set noisy labels presented by Lee et al.
(2019) ( https://github.com/pokaxpoka/
RoGNoisyLabel ).
Clothing1M.
For the experiment summarized in Table 3, we used the open-source dataset presented by Xiao et al.
(2015) ( https://github.com/Cysu/noisy_label ).
B C OMPARISONS TO R ELATED W ORKS Methodology
[Table]
et al. [Table] [Table] B indicates that no previous methodologies can conceptually include our method.
Because the solution to the Fokker-plank equation can be explicitly calculated without any additional parameters, our method is fully non-parametric (in terms of additional parameters beyond those required by the original neural network).
By contrast, co-teaching is parametric because it requires a clone network with additional parameters that are copies of those in the original network.
Similarly, MLNT requires an additional teacher network for training, which also contains a number of parameters.
Many method based on small loss criteria select certain samples, whereas our method uses the combination of ρN certain and (1 − ρ ) N normalized uncertain samples.
Therefore, our method can fully leverage the batches of training datasets, where (1 − ρ ) N + ρN = N .
Additionally, our method does not assume any class-dependent prior knowledge.
Rather than considering class-wise prior knowledge, our method uses holistic information from both certain and uncertain samples ( i .
e .,
Y and X T ) in the logit space.
Other meta-class-based model, such as MLNT, assume class-wise meta prior knowledge from a teacher network.
In Arazo et al.
(2019), they assumed the beta-mixture model as a label distribution on label space. But due to the non-deterministic type of noisy label distribution, it sometimes fails to train with extremely non-uniform type of noise. For example, Arazo et al. (2019) reported failure case with Clothing1M dataset.
It seems that fundamental assumption on noise model of mixup will be improved in future work.
Similar to this method, our work have trouble when dealing with synthetic asymmetric noise with high ratio where relatively large performance drop is observed in Table 1 (despite our method produces second best performance in the table).
Most recent work Li et al.
(2019a)
, they also adopt Co-train by implementing additional dual network, but much sophisticated methodology called Co-divide/guessing based on SSL.
We predict that the
Wasserstein distance between labeled and unlabeled probability measures is well-controlled in their method.
We think that applying the OT/Markov theory (as in our paper) to their method will broaden the understanding of LNL problem.
In contrast to sample weight methods such as GCE and NPCL, which require prior knowledge regarding the cardinality of the training samples to be weighted, our method is free from such assumptions because our Wasserstein normalization is applied in a batch-wise manner.
C T ECHNICAL D IFFICULTY FOR APPLYING GENERAL O PTIMAL T RANSPORT /M ARKOV THEORY TO L ABEL S PACE .
Let X, Y be uncertain and certain samples in pre-softmax feature space.
And assume that we consider the distributional constraint on label-space (the space of σ ( X ) , σ ( Y ) , where σ denotes the soft-max function).
This space is not proper to deﬁne the objective function such as (5).
Because, all the samples in this label space is of the form σ ( X ) =
[ a 1 , a 2 , · · · , a n ] such that (cid:80) di =1 a i = 1 , thus label-space is d -dimensional afﬁne-simplex U
d which is subset of Euclidean space U d ⊂ R d .
In this case, the deﬁnition of Wasserstein space in equation (4) is unacceptable while d E is not true metric on U d .
The Wasserstein space W 2 ( U d ) is merely investigated in the mathematical literature which makes unable to use all the technical details and assumptions, theories developed in the W 2 ( R d ) which are theoretical ground of our work.
But, if we look this problem slightly different point of view, for example, consider pre-softmax R d , W 2 ( R d ) as our base space.
In this case, all the technical issues/problems when we try to use OT tools in W 2 ( U d ) can be overcome/ignored.
while softmax is non-parametric one-to-one function connecting pre-softmax feature space R d to U d , there exists a unique labels in U d as a mapped point of the manipulated uncertain samples.
Even though our objects are deﬁned on pre-softmax space, the theoretical analysis in Proposition 3 contains softmax function to evaluate the concentration inequality of proposed transformation F affecting in label-space U d .
D M ATHEMATICAL B ACKGROUND
In this section, we introduce important deﬁnitions, notations, and propositions used in our proofs and the main paper.
D. N OTATION
We denote f # µ as a push-forward of µ through f .
C ∞
( R d ) denotes the set of ∞ -class functions with compact support in R d .
For the L p -norm of the function f , we denote (cid:107) f (cid:107) p,ν = ( (cid:82) | f | p dν ) 1 p .
The Hessian matrix of the function f is denoted as Hess [ f ] =
[ ∂ i ∂ j f ] di,j .
Sym + d denotes the space for semi-deﬁnite positive symmetric matrices of size d × d .
(cid:107) f (cid:107) Lip denotes the Lipschitz norm of the function f .
For any matrix A ∈ M d , we let (cid:107) A (cid:107) op denote the operator norm of A .
D. D IFFUSION - INVARIANCE AND H YPER - CONTRACTIVITY
Deﬁnition 2.
The Markov semigroup ( P t ) t ≥ 0 in R d acting on a function f ∈ C ∞ 0 is deﬁned as follows:
[Equation]
15) where p t ( x, dx (cid:48) ) is a transition kernel that is the probability measure for all t ≥ 0 .
Deﬁnition 3.
(Diffusion Operator)
Given a Markov semi-group P t at time t , the diffusion operator (i.e., inﬁnitesimal generator) L of P t is deﬁned as
[Equation]
(16) where B and A are matrix and vector-valued measurable functions, respectively.
B ij denotes the ( i, j ) -th function of B and A i denotes the i -th component function of A .
Deﬁnition 4.
(Diffusion-invariant Measure)
Given the diffusion operator L , the probability measure µ is considered to be invariant measure to L when E X ∼ µ [ L f ( X )] =
for any f ∈ C ∞ 0 .
Lemma 1.
(Inﬁnitesimal generator for the multivariate Gaussian measure, Bolley & Gentil (2010).)
The Gaussian measure N ν := N ( m ν , Σ ν ) with a mean m ν and covariance Σ ν is an invariant measure according to the following diffusion-operator L :
[Equation]
where B ij ( x ) : =
[Σ ν ] ij is a constant function, and A i ( x ) :
= x i − m iν .
This generator serves as our main tool for the geometric analysis of the upper bound ε .
In Section 4.
in the main paper, we introduced an approximate upper-bound ˆ K 2 ( µ ) without any general description of the inequality involved.
We now introduce the underlying mathematics for equation 24.
Because our detour measure is Gaussian, there is a unique semi-group P t h called the multidimensional
Ornstein-Ulenbeck semi-group that is invariant to N ν .
Speciﬁcally, P t is deﬁned as follows:
[Equation]
The invariance property of P t relative to our detour measure is naturally induced by the following
Proposition:
Proposition 4.
We deﬁne C : R d → R d and C ( X ) =
A X + b such that A ∈ Sym + d , b ∈ R d , and select an arbitrary smooth h ∈ C ∞ 0 ( R d ) .
We then deﬁne the diffusion Markov semi-group P s h as follows:
[Equation]
Then, N ( A 2 , b ) is invariant with respect to P s , meaning the following equality holds for every h and s ≥ 0 :
[Equation]
Proof.
For simplicity, we denote N ( A 2 , b ) :
= N C .
[Equation]
The second equality holds because C is linear in R d .
Let e − s = cos θ and e − 2 s = sin θ for any 0 ≤ θ ≤ 2 π .
Then, we deﬁne φ as φ ( Z (cid:48) , Z ) =
e − s Z (cid:48)
+ √ 1 − e − 2 s Z = cos( θ ) Z (cid:48)
+ sin( θ ) Z , and π ( Z (cid:48) , Z ) =
Z .
Based on the rotation property of the standard Gaussian measure, one can induce the following equality.
[Equation]
However, we know that d N [ C − 1 ( X )]
=
d N C
( X ) =
(cid:0) (
[Equation]
π ) d | A 2 | (cid:1) [Equation] 12 e
[Equation]
( X − b ) T A − 2 ( X − b ) .
By combining equation 21 and equation 22, one can derive the following result:
[Equation]
Proposition 4 demonstrates the invariance property of the deﬁned semi-group.
If we set A = Σ 12 ν , b
= m ν , then we can recover equation 18.
We are now ready to deﬁne the approximation of K 2 ( µ ) in terms of semi-group invariance.
Speciﬁcally, for any real-valued smooth h , we deﬁne the following inequality:
[Equation]
[Equation]
This inequality holds if h is selected to induce a supremum over the set C ∞ 0 , where sup h ˆ K 2 ( µ, h )
=
sup h
E X ∼ µ [ L h ( X )]
= K 2 ( µ ) .
Although a more sophisticated design for the test function h will induce a tighter upper bound for ˆ K 2 , we determined that the L 2 -norm is generally sufﬁcient.
Deﬁnition 5.
(Diffuseness of the probability measure)
We deﬁne the integral operator K 2 :
W 2 ( R d ) → R + as follows:
[Equation]
According to Deﬁnition 4, we know that (cid:82) L f ( X )
d N ν
( X ) =
for any f .
Based on this observation, it is intuitive that K 2 estimates how the probability measure ν is distorted in terms of diffusion invariance.
While this measure takes a supremum over the function space C ∞ 0 , it searches for a function that enables the estimation of maximal distortion.
Because the value of K 2 is entirely dependent on the structure of µ , K 2 can be considered as a constant for the sake of simplicity if the uncertain measure µ is ﬁxed over one iteration of training.
Deﬁnition 6.
(Diffusion carré du champ)
Let f, g ∈ C ∞
( R d ) .
Then, we deﬁne a bilinear form Γ c in C ∞
( R d ) × C ∞ 0 ( R d ) as
[Equation]
We also denote Γ( f ) ≡ Γ( f, f ) .
The bilinear form Γ can be considered as a generalization of the integration by the parts formula, where (cid:82)
f L g + Γ( f ) dµ =
for the invariant measure µ of L .
Deﬁnition 7.
(Curvature-Dimension condition, Ambrosio et al.
We can say that the inﬁnitesimal generator L induces the CD ( ρ, ∞ ) curvature-dimension condition if it satisﬁes Γ 1 ( f ) ≤ ρ Γ 2 ( f ) for all f ∈ C ∞ 0 .
Because our diffusion operator generates a semi-group with respect to the Gibbs measure, the curvature-dimension condition can be calculated explicitly.
Through simple calculations, the ﬁrstorder ( c = 1) diffusion carré du champ can be induced as follows: [Equation]
[Equation]
Similarly, the second-order ( c = 2) diffusion carré du champ is calculated as follows:
[Equation]
for an arbitrary f ∈ C ∞
( R d ) .
While Tr (cid:16)(cid:2) Σ ∇ 2 f (cid:3) 2 (cid:17) is non-negative, we can infer that Γ 1 ≤ Γ 2 .
In this case, the diffusion operator L deﬁned in Lemma 1 induces the CD ( ρ = 1 , ∞ ) curvaturedimension condition.
For the other diffusion operators, please refer to Bolley & Gentil (2010).
Proposition 5.
(Decay of Fisher information along a Markov semigroup, Bakry et al.
If we assume the curvature-dimension condition CD ( ρ, ∞ ) , then I ( µ t |N ν ) ≤ e − 2 ρt I ( µ |N ν ) .
The exponential decay of the Fisher information in Proposition 5 is a core property of the exponential decay of the Wasserstein distance, which will be used in the proof of Proposition 2.
D. F OKKER -P LANK EQUATION , SDE
Deﬁnition 8.
(Over-damped Langevin Dynamics)
We have
[Equation]
(29) where φ ( X t ; m ν )
= τ 2 d 2 ( X t , m ν ) , W t denotes Brownian motion, and d denotes Euclidean distance.
The particle X t is distributed in X t ∼ p t .
The probability density lim
t →∞ p ( x, t ) with respect to X ∞ converges to the Gaussian density X ∞ = √ Σ
ν ( Z + m ν ) ∼ p ∞ ( x ) =
q ( x ) ∝ e − d ( x, m ν ) T Σ − 1 ν d ( x, m ν ) .
In classical SDE literature, it is stated that E (cid:104) sup 0 ≤ t ≤ T (cid:12)(cid:12)(cid:12) ˆ X t − X t (cid:12)(cid:12)(cid:12)(cid:105) ≤ G ( N(cid:37) ) − 12 , where G ( T ) is some constant that depends only on T and ˆ X denotes the true solution of the SDE in equation 29.
While the number of uncertain samples is greater than N(cid:37) > 40 , our method exhibits acceptable convergence.
D. G AUSSIAN W ASSERSTEIN S UBSPACES
It is known that the space of non-degenerate Gaussian measures ( i .
e ., covariance matrices are positivedeﬁnite) forms a subspace in the 2-Wasserstein space denoted as W 2 ,g ∼ =
Sym + d × R d .
Because the 2-Wasserstein space can be considered as a Riemannian manifold equipped with Riemannian metrics Villani (2008), W 2 ,g can be endowed with a Riemannian structure that also induces the
Wasserstein metric (McCann (1997)).
In the Riemannian sub-manifold of Gaussian measures, the geodesic between two points γ (0) =
N A and γ (1) =
N B is deﬁned as follows Malagò et al.
[Equation]
where m ( α ) =
(1 − α )
m
A + α m B and Σ( α ) =
[(1 − α )
I + α T ] Σ A
[(1 − α )
I + α T ] , where
T Σ A T = Σ B .
In Section 3.2, we set ( m A , Σ A ) → ( m ν , Σ ν ) and ( m B , Σ B ) → ( m ξ k , Σ ξ k ) .
Regardless of how ν is updated, the statistical information regarding the current certain measure ξ k is considered in the detour Gaussian measure, which yields a much smoother geometric constraint on µ .
E P ROOFS
Proposition 6.
Let Γ( µ, ν ) be a set of couplings between µ and ν , and assume that the noisy label ˆ r is independent of X .
For functional J [ µ ] =
E µ ∼ X l
( X ; ˆ r ) , we deﬁne D ( µ, ν ) as:
[Equation]
(31) where D : P 2 × P 2 → R .
Then, D is the metric deﬁned on P 2 , which is weaker than the Wasserstein metric, where D ( µ, ν ) ≤ α W 2 ( µ, ν ) for α = c − 1
ˆ r + c − 1 1 (1 − ˆ r ) and some constants c 0 , c 1 > 0 .
Proof.
[Equation]
By taking the inﬁmum of the aforementioned inequality with set of couplings γ ( µ, ν ) , we obtain the following inequality:
[Equation]
which completes the proof.
Proposition 6 follows from the Lipschitzness of the functional J , where D searches for the best coupling to derive the minimal loss difference between two probability measures.
This proposition indicates that inf |J [ ν ] − J [ F µ ] | is bounded by the Wasserstein distance , which justiﬁes our geometric constraint presented in equation 4.
It should be noted that the prior assumption regarding noisy labels is essential for Lipschitzness.
Proposition 7.
Let F : R + × P 2 be a functional on probability measures such that F [ t, µ ] = µ t , where dµ t =
p t d N ν , d N ν = dq t dx , and let µ t be a solution of the continuity equation in the 2-Wasserstein space deﬁned as follows:
[Equation]
(34) which is represented as ∂ t p ( t, x ) = ∇ · ( p ( t, x ) ∇ log q ( t, x )) in a distributional sense.
Then, the functional F t [ · ] = F [ t, · ] is deﬁned unique and normalizes µ onto B W 2 ( N ν , e − t K 2 ( µ )) , where
K 2 ( µ ) ≤ ∞ is an integral operator in Deﬁnition 5 with respect to µ .
Proof.
We assume that the probability measure µ t is absolutely continuous with respect to the detour
Gaussian measure N ( m ν , Σ ν )
= N ν , µ t (cid:28) N ν .
In this case, according to the Radon-Nikodym theorem, there is a corresponding unique probability density q ( t, x ) =
q t ( x ) ∈ C ∞ 0 such that dµ t =
q t d N ν .
Lemma 2.
( WI-inequality , Otto & Villani (2000))
If the stationary state of µ t with respect to P t satisﬁes lim t →∞ E µ [ P t f ] = 0 for any f ∈ C ∞ 0 , then the following inequality holds:
[Equation]
By integrating both sides of the inequality in Lemma 2 with respect to t ∈ (0 , ∞ ) , the following inequality can be obtained:
[Equation]
In the aforementioned inequality, we replace the Fisher information with the diffusion generator L as follows:
[Equation]
The second equality above is derived by leveraging the properties of the bilinear operator Γ (Bakry et al.
; Villani (2008)) with respect to the diffusion operator L , which is deﬁned as follows:
[Equation]
For simplicity, we denote | g | = g + for any g ∈ C ∞ 0 .
According to Proposition 5 , we can relate
F t µ = µ t to its initial term µ = µ t =0 as follows: (cid:90) ∞
[Equation]
The second inequality is naturally induced, because the proposed objective function is deﬁned to select the maximum elements over the set of functions g ∈ C ∞ 0 and L g ≤ L + g .
If the integral interval is set to (0 , s ) , then we can induce W 2 ( µ, F t µ ) ≤ 1 ρ (1 − e − s ) K 2 ( µ ) .
Our diffusion-operator induces ρ = 1 , which completes the proof.
Proposition 8.
There is a scalar 0 < β < ∞ dependent on ν such that the following inequality holds:
[Equation]
As a motivation for setting a detour measure to N ν , we mentioned the natural property of the non-collapsing Wasserstein distance of W 2 ( ν, N ν ) (cid:54) = 0 .
However, it is unclear from a geometric perspective exactly how the upper bound ( i .
e .,
W 2 ( ν, N ν ) ≤ ?) can be induced based on the intrinsic statistics term ( i .
e ., d 1 in Fig.1).
Speciﬁcally, in the situation where the covariance matrices of ν and N ν are identical, it is difﬁcult to determine a theoretical upper bound without additional tools.
The ﬁrst part of this proof focuses on resolving this important issue.
The second part of the proof is naturally induced by Proposition 1.
Please note that in the following proposition, parameter for
Wasserstein moving average is set to α = 0 for clarity.
Proof.
Before proceeding with the ﬁrst part of the proof, we deﬁne a constant β as follows:
[Equation]
If we assume a mild condition such that min s,j inf 1 ≤ j ≤ d O ( v s,j ) ≥ O ( √ s ) , then the integral term in β is ﬁnite and well-deﬁned.
This value will directly yield the upper bound of the Kullback–Leibler (KL) divergence of ν .
First, we introduce the following inequality.
Lemma 3.
( de Bruijn’s identity , Johnson & Suhov (2001);
Nourdin et al.
We let Y ∼ ν ,
Z ∼ N ( 0 , I ) denote a standard Gaussian random variable, and let deﬁne Y s = √ sY + √ 1 − s Σ 12 ν Z with the score function deﬁned as v s ( x ) = ∇ log p s ( x ) with respect to the random variable Y s .
Then, the following equality holds:
[Equation]
From equation 42, we can derive the relations between KL-divergence and the constant β deﬁned earlier.
[Equation]
The second inequality holds based on the following element property of symmetric positive-deﬁnite matrices:
[Equation]
It should be noted that because the distribution of ν is compactly supported ( i .
e ., supp ( q ) is compact), the maximum eigenvalue of the covariance Σ ν is ﬁnite.
The other relations are induced by the aforementioned deﬁnition.
Next, we relate the KL-divergence and 2-Wasserstein distance naturally.
Deﬁnition 9.
( Talagrand inequality for Gaussian measures , Otto & Villani (2000))
For any nondegenerate Gaussian measure N with a mean 0 , the following inequality is satisﬁed:
[Equation]
By combining Deﬁnition 9 and equation 43, we can derive the following expression:
[Equation] [Equation]
According to the triangle inequality for the 2-Wasserstein distance, we obtain:
[Equation]
[Equation]
In Appendix C.3, we investigated that the geodesic distance between two Gaussian measures having the same covariance is equivalent to the Euclidean distance between two means.
Therefore, we can obtain the following equality:
[Equation]
where ι a ( X ) =
X
+ a for any vector a ∈ supp ( q ) .
Now, by adding the two inequalities deﬁned earlier, we can obtain
[Equation]
where it is easily shown that the upper-bound is only dependent on the statistical structure of ν .
Speciﬁcally, the term (cid:107) E ν Y (cid:107) 2 represents the center of mass for a density of ν and (cid:112) dβλ max (Σ ν ) is related to the covariance structure of ν .
By applying Proposition 8 to both F t µ and ν , we can easily recover equation 5 as follows:
[Equation]
The second inequality is easily obtained as ( a ∧ b ) + c ≤ a ∨ ( b + c ) for any a, b, c ≥ 0 , which completes the proof.
Proposition 9.
(Concentration inequality for uncertain measures).
Assume that there are some constants s (cid:63) ∈ [ 1 η , ∞ ) , η ≥ 0 such that the following inequality is satisﬁed:
[Equation]
for A ∈ Sym + d , D
( A , Σ ν ) ≤ aη for some a > 0 , and for any metric D deﬁned on Sym + d .
In this case, there is a δ such that the following probability inequality for an uncertain measure is induced:
[Equation]
where κ denotes the Lipschitz constant of σ .
Proof.
Before proceeding with the main proof, we ﬁrst prove the existence of s (cid:63) .
The limit of the interval with respect to η converges to a singleton {∞} as I = lim
η → 0 [ 1 η , ∞ ) .
In this case, equation 51 is the same as the Poincaré inequality for a Gaussian measure N ν , which can be written as
[Equation]
While the Poincaré inequality in equation 53 is uniquely deﬁned, we can ﬁnd at least one value s (cid:63) satisfying equation 51.
Let X ( t, w ) =
X t ( w ) denote the stochastic process with respect to q t ( x ) deﬁned in the proof of Proposition 2.
Additionally, let c = E ν [ σ ] − E F s(cid:63) µ [ σ ] .
Then, we can obtain the following inequality:
[Equation]
The ﬁrst inequality is induced by the assumption regarding the κ -Lipschitzness of the function σ and the second inequality is induced by the Kantorovich-Rubinstein theorem.
The third inequality is natural because W a ( · , · ) ≤ W b ( · , · ) for any 1 ≤ a ≤ b < ∞ .
because equation 51 is equivalent to the
Poincaré inequality for the measure F s (cid:63) µ , it satisﬁes the Bakry-emery curvature-dimension condition
CD (1 + η, ∞ ) .
Thus, as shown in the proof of Proposition 2 ( i .
e ., equation 39), the last inequality is induced.
Additionally, based on the concentration inequality of F s (cid:63) µ [Proposition 4.4.2 Bakry et al.
(2013)], we can derive the following probability inequality:
[Equation]
where the Poincaré constant for F s (cid:63) µ is naturally 1 + η and (cid:107) σ (cid:107) Lip = κ .
Next, we will derive the desired form from equation 55.
First, we introduce the following inequality.
[Equation]
The last inequality is directly induced by equation 54 because − c ≥ − κ 1+ η K 2 .
While η, κ , and Kare constants with respect to w , the following set inclusion can be obtained naturally:
[Equation]
For the modiﬁed version of the original probability inequality, we take probability measure F s (cid:63) µ [ · ] for the sets S 1 , S 2 , which is deﬁned as
[Equation]
The concentration inequality around E ν [ σ ] is obtained by combining the inequalities induced by σ and − σ as follows:
[Equation]
The inequality in equation 59 is the general form containing the relation between the upper bound of the probability and ( η, κ, K 2 ) .
While this form is quite complicated and highly technical, we choose not to present all the detailed expressions of equation 59 in the main paper.
Rather than that, we re-write it in a much simpliﬁed form for clarity.
Speciﬁcally, by setting κK 2 / (1 + η ) = 0 .
5 δ and rescaling δ to 2 δ , the aforementioned inequality in equation 59 can be converted into the following simpler form:
[Equation]
Finally, if we set σ = Softmax , then the Lipschitz constant is induced as κ = 1 .
This proof is completed by setting s (cid:63) :
= T .