W ASSERSTEIN A DVERSARIAL R EGULARIZATION (WAR) ON LABEL NOISE
Anonymous authors
Paper under double-blind review
A BSTRACT
Noisy labels often occur in vision datasets, especially when they are obtained from crowdsourcing or Web scraping.
We propose a new regularization method, which enables learning robust classiﬁers in presence of noisy data.
To achieve this goal, we propose a new adversarial regularization scheme based on the Wasserstein distance.
Using this distance allows taking into account speciﬁc relations between classes by leveraging the geometric properties of the labels space.
Our Wasserstein Adversarial Regularization (WAR) encodes a selective regularization, which promotes smoothness of the classiﬁer between some classes, while preserving sufﬁcient complexity of the decision boundary between others.
We ﬁrst discuss how and why adversarial regularization can be used in the context of label noise and then show the effectiveness of our method on ﬁve datasets corrupted with noisy labels: in both benchmarks and real datasets, WAR outperforms the state-of-the-art competitors.
I NTRODUCTION
Deep neural networks require large amount of accurately annotated training samples to achieve good generalization performances.
Unfortunately, annotating large datasets is a challenging and costly task, which is practically impossible to do perfectly for every task at hand.
It is then most likely that datasets will contain incorrectly labeled data, which induces noise in those datasets and can hamper learning.
This problem is often referred to as learning with label noise or noisy labels .
The probability of facing this problem increases when the dataset contains several ﬁne grained classes that are difﬁcult to distinguish (Schroff et al., 2011; Krause et al., 2016; Dubey et al., 2018).
As pointed out in (Zhang et al., 2017), deep convolutional neural networks have huge memorization abilities and can learn very complex functions.
That is why training with noisy data labels can lead to poor generalization (Arpit et al., 2017; Wang et al., 2018; Choi et al., 2018).
Hence in this paper we propose a method tackling the problem of overﬁtting on noisy labels, and this without access to a clean validation dataset.
This problem has been considered in recent literature, mainly in three ways.
First are data cleaning methods : (Vahdat, 2017; Xiao et al., 2015; Li et al., 2017) learn relations between noisy and clean labels before estimating new labels for training.
In (Lee et al., 2018), few human veriﬁed labels were necessary to detect noisy labels and adapt learning.
In (Jiang et al., 2018; Ren et al., 2018), the methods rely either on a curriculum or on meta-gradient updates to re-weight training sets and downweight samples with noisy labels.
Second are Transition probability-based methods : (Sukhbaatar & Fergus, 2014; Patrini et al., 2017; Hendrycks et al., 2018) estimate a probability for each label to be ﬂipped to another class and use these estimations to build a noise transition matrix.
In (Sukhbaatar & Fergus, 2014), the authors add an extra linear layer to the softmax in order to learn the noise transition matrix itself, while (Hendrycks et al., 2018) uses a small set of clean validation data to estimate it.
(Patrini et al., 2017) proposes a forward/backward loss correction method, which exploits the noise transition matrix to correct the loss function itself.
Third are regularization-based methods .
In (Reed et al., 2015; Ma et al., 2018), the authors use a mixture between the noisy labels and network predictions.
In (Tanaka et al., 2018), the regularization is achieved by alternatively optimizing the network parameters and estimating the true labels while the authors of (Han et al., 2018; Yu et al., 2019) propose peer networks feedbacking each other about predictions for the noisy labels.
Song et al. (2019) proposes to replace noisy labels in the mini-batch by the consistent network predictions during training, while Chen et al. (2019) proposes noisy cross-validation to identify samples that have correct labels.
In (Wang et al., 2019; Zhang & Sabuncu, 2018; Ghosh et al., 2017), robust loss functions are proposed to overcome limitations of cross entropy loss function.
In contrast to those works, we propose to regularize predictions in areas of the feature space close to the decision boundary of conﬂicting classes, therefore mitigating the inﬂuence of noisy labels.
To do so, we use the adversarial regularization (AR) framework (Goodfellow et al., 2015;
Miyato et al., 2018) on the noisy label problem.
We use AR to reduce the discrepancy between the prediction of a true input sample and the one obtained by a near-by adversarial sample.
To reduce this discrepancy, we use a loss based on the Wasserstein distance computed with respect to a ground cost encoding class similarities.
This ground cost provides the ﬂexibility to regularize with different strengths pairs of classes.
This strength can depend on semantic relations, classes similarities, or prior knowledge (e.g. on annotators’ mistakes).
This way, the classiﬁer can discriminate non-similar objects robustly under the presence of noise and class overlap.
We name our proposed method
Wasserstein Adversarial Regularization (WAR).
WAR allows incorporating speciﬁc knowledge about the potential degree of mixing of classes through a ground cost that can be designed w.r.t.
the problem at hand.
Nevertheless, this knowledge might be unknown or difﬁcult to craft.
In this paper, we use distances between word embeddings of the class names to derive a semantic ground cost.
Experiments in ﬁve datasets (Fashon-MNIST, CIFAR10, CIFAR100 and real life examples on clothing classiﬁcation and a remote sensing semantic segmentation dataset) under label noise conditions show that WAR outperforms the state-of-the-art in providing robust class predictions.
W ASSERSTEIN A DVERSARIAL R EGULARIZATION FOR LABEL NOISE
Given a set of labeled data { x i , y i } i =1 ,...
N , we are interested in learning a classiﬁer p θ deﬁned by a set of parameters θ .
Data x
i are usually elements of R d , while y i ∈ C are one-hot vectors encoding the belonging to one of C classes.
The empirical risk minimization principle is used to learn p θ .
Given a loss function L , the optimal set of parameters for the classiﬁer is given by arg min θ (cid:80) Ni =
1 L ( x i , y i , p θ ) .
Without loss of generality, we will consider that L is the cross-entropy loss: L CE ( x i , y i , p θ )
= − (cid:80)
c y ( c )
i log p θ ( x i ) ( c ) .
The considered label noise problem arises whenever some elements y i do not match the real class of x i .
Several scenarii exist: in the symmetric label noise, labels can be ﬂipped uniformly across all the classes, whereas in the asymmetric label noise, labels y in the training set can be ﬂipped with higher probability toward speciﬁc classes.
We note that the ﬁrst scenario, while thoroughly studied in the literature, is highly improbable in real situations: for example, it is more likely that an annotator mislabels two breeds of dogs than a dog and a car.
Hence, noise in labels provided by human annotators is not symmetric since annotators make mistakes depending on class similarities (Misra et al., 2016).
To prevent a classiﬁer to overﬁt on noisy labels, we would like to regularize its decision function in areas where the local uniformity of training labels is broken.
To achieve such desired local uniformity, robust optimization can be used.
This amounts to enforce that predicted labels are uniform in a local neighborhood
U i of data point x i .
This changes the total loss function in the following way:
[Equation]
2. A DVERSARIAL R EGULARIZATION
Because the robust optimization problem in equation 1 is hard to solve exactly, adversarial training (Goodfellow et al., 2015; Shaham et al., 2015) was proposed as a possible surrogate.
Instead of solving the max inner problem, it suggests to replace it by enforcing uniformity in the direction of maximum perturbation, called the adversarial direction.
Mainly used for robustness against adversarial examples, adversarial training is however not adapted to our problem, since it can enforce uniformity around a false label.
Following the same reasoning, we propose to cast the problem as a regularization term of the initial loss function.
The total learning loss is then
L tot ( x i , y i , p θ ) =
L CE ( x i , y i , p θ ) + βR AR ( x i , p θ ) , where R AR is a regularization term reading:
R AR ( x i , p θ ) =
D ( p θ ( x i + r ai ) , p θ ( x i )) with r ai = argmax r
[Equation]
Basically, it minimizes an isotropic divergence D between the probability output p θ ( x i + r a ) and p θ ( x i ) .
A sound choice for D can be the Kullback-Leibler (KL) divergence.
R AR can be seen as a (negative) measure of local smoothness , or also as a Local Lipschitz constant in the ε neighborhood of x i with respect to the metric D , hence a measure of complexity of the function.
R AR promotes local uniformity in the predictions without using the potentially noisy label y i : therefore, it reduces the inﬂuence of noisy labels, since it is computed from the prediction p θ ( x i ) that can be correct when the true label is not.
R AR shares strong similarities with the Virtual Adversarial Training (VAT) from Miyato et al.
(2018), at the notable exception that we do not consider a semi-supervised learning problem and that we regularize on the labeled training positions x i , where VAT is applied on unlabeled samples.
It can be shown (proof in the supplementary material A.1) that this regularization acts as a label smoothing technique:
Lemma 1 Let D be the Kullback-Leibler divergence.
Let (cid:15) =
ββ +1 ∈ [0 , 1[ .
Let y ai
= p θ ( x i + r ai ) be the predicted (smooth) adversarial label.
Let H be the entropy.
The regularized learning problem
L tot ( x i , y i , p θ ) is equivalent to :
[Equation]
This leads to the following interpretation: instead of learning over the exact label or over a mix between the exact label and the network prediction, we learn over an interpolation with the adversarial label, while maximizing the entropy of the adversarial label (i.e. blurring the boundaries of the classiﬁer).
Related developments can be found in adversarial label smoothing (ALS)
(Shafahi et al., 2018;
Goibert & Dohmatob, 2019), which aims at providing robustness against adversarial attacks.
Yet, one of the major limit of this approach is that the regularization is conducted with the same magnitude everywhere.
As a consequence, a strong regularization can remove the label noise, but also hinder the ability of the classiﬁer to separate similar classes where a complex boundary is needed.
To overcome this issue, we propose to replace D by a geometry-aware divergence taking into account the speciﬁc relationships between the classes.
2. W ASSERSTEIN A DVERSARIAL R EGULARIZATION (WAR)
To make the divergence aware of speciﬁc relationships between classes, we replace the isotropic divergence D with a Wasserstein distance computed in the labels space.
We name our proposed method Wasserstein Adversarial Regularization.
Frogner et al.
(2015) already used the Wasserstein distance as a loss in a learning system between the output of the model for multi-label learning.
The interest of the Wasserstein distance is to take into consideration the geometry of the label space.
We deﬁne the proposed regularization term R WAR as follows:
[Equation]
OT λ C is an optimal transport (OT) distance ( [Equation] [Equation]
The OT problem seeks an optimal coupling T ∗ ∈
U ( α , β ) =
{ T | T ≥ 0 , T 1 = α , T (cid:62) 1 = β } between two distributions α , β with respect to a ground cost C ∈ R n 1 × n 2 .
U ( α , β ) is the space of joint probability distributions with marginals α and β .
OT distances are classically expressed through the Wasserstein distance
W C ( α , β ) =
min T ∈ U ( α , β ) (cid:104) T , C (cid:105) , where (cid:104) .
(cid:105) is the Frobenius product.
Unfortunately, this distance is expensive to compute (cubical complexity).
In practice, we will use the solution of the sharp entropic variant of the optimal transport problem (Luise et al., 2018):
[Equation]
where H denotes the entropy function and λ the regularization strength.
Using this regularized version has several advantages: i) it lowers the computational complexity to quadratic, ii) it turns the problem into a strongly convex one, for which gradients can be computed efﬁciently and iii) it allows to vectorize the computation of all Wasserstein distances in a batch, which is particularly appealing for training deep neural nets.
Based on (Genevay et al., 2018), we use the AutoDiff framework, which approximates the derivative of this regularization with a ﬁxed number of iterations of the
Sinkhorn algorithm.
Choice of ground cost.
The ground cost C reﬂects the geometry of the label space.
It bridges the gap between AR and WAR.
An uninformative 0-1 ground cost, i.e. 0 over the diagonal andeverywhere, would give the total variation (TV) loss (Remark 2.26 in (Peyr´e & Cuturi, 2019)), which could also be used as D in the AR framework.
Below, we refer to this special case as R WAR.0-1 .
To deﬁne a C matrix encoding class relations, multiple choices are possible.
We could calculate the
Wasserstein distance between classes, but this would be biased as we have noisy labels.
We could set it manually (Tuia et al., 2011), but this becomes unpractical when a large set of classes is present.
In absence of prior information about the nature of the source of labelling errors, we decided to rely on semantic distances based on word embeddings such as word2vec (Mikolov et al., 2013).
Similarities between classes are then deﬁned via Euclidean distances in the embedding space, as proposed in (Frogner et al., 2015).
Finally, as our method requires large values of the cost between similar classes, we apply the function e − m (where m is the Euclidean distance between the two class names)
element-wise and set the diagonal of C to 0.
Function smoothness and ground metric.
Now we discuss how the proposed regularization term regularizes the model p θ with a smoothness controlled by the ground metric C .
It is not possible to extend the label smoothing in Lemma 1 to WAR because the OT distance does not admit a close form solution, but we can still show how R WAR promotes label smoothness.
To this end, we look at the regularization term OT λ C (ˆ p θ ( x ) , p θ ( x + r )) for a given sample x and a pre-computed r. We can prove (see A.2 of the supplementary material) the following lemma:
Lemma 2 Minimizing R WAR with a symmetric cost C such that C i,i = 0 , ∀ i is equivalent to minimizing a weighted total variation (TV) norm between p θ ( x ) and p θ ( x + r ) .
[Equation]
where c k = min i,i (cid:54) =
k c k,i is the minimal off-diagonal cost for row k of C and c = min k c k is a global minimum out of the diagonal.
By minimizing the proposed R WAR regularization with r belonging in a small ball around x , we actually minimize a local approximation of the Lipschitz constant of p θ .
This has the effect of smoothing-out the model around x and makes it more robust to label noise.
One can see the effect of the cost matrix in the center term of equation 4, where the values in the ground metric correspond to a weighting of a total variation, hence controlling the effect of the regularization.
Interestingly, the
Wasserstein distance can be bounded both below ( c ) and above ( ¯ c ) by Total Variation and weighted total variation similarly to the equation above.
Finally, in practice we minimize the expectation of the OT loss, which means that we will penalize areas of high density similarly to a regularization with the Sobolev norm (i.e. penalizing the expected norm of the model gradient (Mroueh et al., 2018)), while keeping a ﬁner control of the class relations, since we use the ground loss C that promotes anisotropy.
Illustration of the effect of R
WAR
We illustrate AR and WAR in a simple toy 3 classes classiﬁcation problem with noise (Figure 1).
Each column of the ﬁgure corresponds to a divergence function D .
The top row illustrates the values predictions adversarial regularization
, we compare the effect of training with the entropy alone (
), R with λ
1 and λ
, as well as R with
TV, and L
D .
For the problem, we generated two close and
), as well as a black), which is far from the others.
Then, we introduced noisy labels (of the class) in the region of the red
On this toy example, CCE overﬁts the noisy black labels, yet is able to distinguish the red and orange classes.
The R AR regularizers, being class agnostic, correct for the noisy black labels in the bottom
[Figure]
part, but smooth the complex decision function between the orange and red classes.
On the contrary,
R WAR uses a different cost per pair of classes, illustrated in the top left panel of Figure 1: the smallest cost is set between the red and black classes, which has the effect of promoting adversarial examples in that direction.
This cost / smoothing relation is due to the fact that our problem is a minimization of the OT loss: in other words, the higher the cost between the classes, the less the binary decision boundary will be smoothed.
Finally, the effect of the global λ parameter can also be appreciated in the classiﬁcation results: while using λ = 0 .
05 , the smoothing of the loss is decreased and the ﬁnal decision boundary between the mixed classes keeps all its complexity.
E XPERIMENTS
We evaluate the proposed approach WAR on both image classiﬁcation and semantic segmentation tasks.
We ﬁrst showcase the performance of WAR on a series of image classiﬁcation benchmarks (Section 3.1), and then consider two real world cases: ﬁrst, the classiﬁcation of clothing images from online shopping websites (Section 3.2) and then the semantic segmentation of land use in sub-decimeter resolution aerial images (Section 3.3).
3. I MAGE CLASSIFICATION ON SIMULATED BENCHMARK DATASETS
Datasets and noisy labels simulations.
We consider three image classiﬁcation benchmark datasets: Fashion-MNIST (Xiao et al., 2017), and CIFAR-10 / CIFAR-100 (Krizhevsky, 2009).
Fashion-MNIST consists of 60 (cid:48) 000 gray scale images of size 28 × 28 with 10 classes.
CIFAR10 and CIFAR-100 consist of 50 (cid:48)
color images of size 32 × 32 covering 10 and 100 classes, respectively.
Each dataset also contains 10 (cid:48) 000 test images with balanced classes.
Since we want to evaluate robustness to noisy labels, we simulated label noise in the training data only.
For all datasets, we introduced 0 % , 20 % and 40 % of noise in the labels.
We considered only asymmetric noise, a class-conditional label noise where each label y i in the training set is ﬂipped into y j with probability P i,j .
As described above, asymmetric noise is more common in real world scenarios than symmetric noise, where the labels are ﬂipped uniformly over all the classes.
For CIFAR-10 and CIFAR-100, we follow the asymmetric noise simulation setting by Patrini et al.
(2017), where class labels are swapped only among similar classes with probability p (i.e. the noise level).
For Fashion-MNIST, we visually inspected the similarity between classes on a t -SNE plot of the activations of the model trained on clean data; we then swapped labels between overlapping classes ( → : one-directional swap, ↔ mutual swap): DRESS → T-SHIRT/TOP, COAT ↔ SHIRT,
SANDAL → SNEAKER, SHIRT → PULLOVER, ANKLE BOOT → SNEAKER.
Baselines.
We compared the proposed WAR with an informative C matrix based on the word2vec embedding ( WAR C ) against several state-of-the-art methods: Unhinged (Rooyen
[Table]
et al., 2015), Bootstrapping (Reed et al., 2015), Forward and Backward loss correction (Patrini et al., 2017), dimensionality driven learning (D2L)
(Ma et al., 2018),
Co-Teaching (Han et al., 2018), and Co-Teaching+ (Yu et al., 2019).
Finally, as a baseline for all the considered methods we also included a categorical cross entropy (CCE) loss function.
All the methods shared the same architecture and training procedures, as detailed in the supplementary material A.4.
Model.
Similarly to other works (Han et al., 2018; Miyato et al., 2018), we employed a 9-layer
CNN architecture, detailed in Table 5 of the supplementary material
A.4.
For WAR C , we set the hyper-parameters β = 10, λ = 0.05, and ε = 0.005 for all the datasets.
The hyper-parameters of the baselines are set according to their original papers.
The noise transition matrix for the Forward and Backward method is estimated from the model trained with cross entropy (Patrini et al., 2017).
The source code of WAR in PyTorch (Paszke et al., 2017) will be released upon publication.
Results.
Classiﬁcation accuracies are reported in Table 1.
Results show that WAR C consistently outperforms the competitors by large margins, across noise levels and datasets.
In particular, WAR C achieved improvements of 4-5% points on fashion-MNIST/CIFAR-10, and 15% on CIFAR-100 at the highest noise level.
This demonstrates that the inclusion of class geometric information during training mitigates the effect of over-ﬁtting to noisy labels.
Besides WAR C , Unhinged and
Co-Teaching also performed well.
The Forward and Backward method performed slightly better than CCE , which is most likely due to the burden in accurately estimating the noise transition matrix.
It is noted that Co-Teaching uses true noise estimate, and the accuracy might drop if the noise ratio is estimated directly from the noisy data.
Furthermore, performance of
Co-Teaching+ 1 is surprising lower than the one of Co-Teaching on two datasets, in contrast to the observations in (Yu et al., 2019).
From our experiments, we observed that Co-Teaching+ does not perform well when the noise is class-dependent and the model considered has a wide capacity.
We provide a detailed discussion about Co-Teaching+ in the supplementary material (section A.7).
Importance of encoding class similarities: Finally, to better assess the signiﬁcance of including class similarities, we compared WAR C with WAR 0 − 1 and AR using KL divergence (Miyato et al., 2018).
For AR , we did not use the hyper-parameters from the original paper, as it would lead to over-ﬁtting on the noisy labels in the later stage of training.
Therefore, we used β =5 and ε = 0.
(similarly to both WAR approaches), and followed the same training procedure as WAR (as detailed in A.4, where a sensitivity analysis for β is also reported in A.5).
Table 2 reports the performance of WAR 0 − 1 and AR , and shows that WAR C is consistently better than AR and WAR 0 − 1 (except in one case), and outperformed AR signiﬁcantly by a 2-3% margin at the highest noise level.
This demonstrates that including priors about class similarities as done in WAR C helps increasing the robustness against label noise.
[Table]
[Table] [Table]
I MAGE CLASSIFICATION WITH REAL - WORLD LABEL NOISE
Dataset.
In this section, we demonstrate the robustness of WAR C on a large scale real world noisy label dataset,
Clothing1M
(Xiao et al., 2015).
The Clothing1M dataset contains 1 million images of clothing obtained from online shopping websites and has 14 classes.
The labels have been obtained from text surrounding the images and are thus extremely noisy.
The overall accuracy of the labels has been estimated to ≈ 61.54%.
The dataset also contains additional manually reﬁned clean data for training (50k samples), validation (14k) and testing (10k).
However, we did not use the clean training and validation data in this work.
Only the testing clean data was used to evaluate the performance of the different approach when learning with label noise.
Experimental setup and Results: Similar to (Patrini et al., 2017; Wang et al., 2019), we used
ResNet-50 pre-trained on ImageNet for a fair comparison between methods (more details in A.4).
For WAR C , the hyperparameters are similar to those of the previous experiment, except ε = 0 .
Results are reported in Table 3, where we compared WAR C against the competitors from (Patrini et al., 2017; Wang et al., 2019), and also reproduced the CCE accuracy by our own experiments.
As shown in Table 3, our method achieved the highest performance compared to all the baselines.
3. S EMANTIC SEGMENTATION OF AERIAL IMAGES
Datasets and noisy labels simulations.
In this experiment we consider the task of assigning every pixel of an aerial image to an urban land use category.
We considered a widely used remote sensing benchmark, the ISPRS Vaihingen semantic labeling dataset 2 .
The data consist of 33 tiles (of varying sizes, for a total of 168 (cid:48) 287 (cid:48)
871 pixels) acquired by an aircraft at the ground resolution of 9 cm.
The images are true orthophotos with three spectral channels (near infrared, red, green).
A digital surface model (DSM) and a normalized digital surface model (nDSM) are also available, making the input space 5 -dimensional.
Among the 33 tiles, we used the initial data split (11 tiles for training, 5 for validation and 17 for testing).
As ground truth, six land cover classes (impervious surfaces, building, low vegetation, tree, car, background/clutter) are densely annotated.
We simulated label noise by swapping labels at the object level rather than ﬂipping single pixels.
An object is the connected component of pixels sharing the same label.
We also focused on plausible labeling errors: for instance, a car could be mislabeled to an impervious surface, but not to a building or a tree.
Following this methodology, a third of the connected components had the label ﬂipped.
An example of the corrupted data is shown in Figure 3 in the supplementary material (A.6).
Model.
We used a U-Net architecture (Ronneberger et al., 2015), modiﬁed to take the 5 channels input data as inputs.
More details about the training procedure are reported in the supplementary material A.4.
Using this methodology, we obtain an overall accuracy on the clean data of 83.89%, which is close to the state of the art for this dataset. [Table]
Results.
We compare WAR C with standard CCE , Bootsoft , Co-Teaching and AR .
The results, computed on the full test ground truth (including boundaries) and averaged over 2 runs, are reported in Table 4.
Note that the classes are unbalanced and, for most of them, the F1-score is improved using WAR C , except for the dominant class (impervious surfaces).
This leads to a much higher average F1-score using WAR C (compared to its competitors)
[Table]
while the overall accuracy is only slightly decreased compared to Bootsoft .
This behavior can be seen in the maps shown in
Figure 2.
We can see in the close-ups that Bootsoft performs poorly in detecting the cars, which are often confused with generic impervious surfaces.
C ONCLUSION AND DISCUSSION
In this paper
[Table]
we proposed Wasserstein Adversarial Regularization (WAR) to address the problem of learning with noisy labels.
Using a ground cost based on class similarites or prior knowledge, we are able to change the geometry of the regularization loss according to class similarities.
We compare
WAR with state of the art algorithms on the Fashion-MNIST, CIFAR-10, CIFAR-100 benchmarks with noisy labels up to 40%.
WAR outperformed state of the art results on all benchmarks.
Furthermore, we proved that WAR performs accurately on real life problems in both classiﬁcation and semantic segmentation problems.
Future works will consider exploring other strategies to deﬁne the ground cost, beyond the current ‘a priori’ setting: such cost could be for instance learned from the data structure itself.
Moreover, even if in this paper we focused on the label noise problem, WAR remains a generic regularization scheme that could be applied to other classical learning problems as enforcing adversarial robustness or semi-supervised learning.
A S UPPLEMENTARY MATERIAL A. L INKS BETWEEN AR AND ADVERSARIAL LABEL SMOOTHING
This part gathers the proof for Lemma 1.
The total learning loss for one sample ( x , y ) is:
[Equation]
where L CE is the cross entropy loss:
[Equation]
We write the R AR regularization as:
[Equation]
where r a = argmax
[Equation]
ε
[Equation]
x
[Equation]
We have that:
[Equation]
where H is the entropy function.
Consequently
[Equation]
the total loss can be rewritten as:
[Equation]
Here β ∈ R + .
Let β = (cid:15) 1 − (cid:15) with (cid:15) ∈ [0 , 1[ .
We have the following equivalence:
[Equation]
i.e. That’s why learning with the total loss is equivalent to learn on an interpolated label and maximizing the entropy of adversarial labels.
A. L INKS BETWEEN WAR AND LABEL SMOOTHING
In this subsection we aim to prove the following relations
[Equation]
First we recall the deﬁnition of the Wasserstein distance
[Equation]
(11) where U ( α , β ) =
{ T | T ≥ 0 , T 1 = α , T (cid:62) 1 = β } .
It is well known and obvious from (Cuturi, 2013; Luise et al., 2018) that the optimal OT matrix of regularized OT T (cid:63)λ (Equation 4) leads to a larger OT loss than the exact OT solution of the problem above T (cid:63) .
this means that
[Equation]
and the relation is strict when λ > 0 .
Now if we suppose that the cost matrix is symmetric and C i,i = 0 and C i,j >
when i (cid:54)
=
j then solving equation 11 means that the maximum amount of mass on the diagonal of T (cid:63) since it leads to a 0 cost.
Under constraints U ( α , β ) this maximum amount is equal to T (cid:63)i,i = min( α i , β i ) , ∀ i .
This implies that for a given row i in T (cid:63) the amount of mass not on the diagonal is for row i is (cid:80)
j (cid:54)
=
i T (cid:63)i,j = max( α i − β i , 0) because of the left marginal constraint in U .
Note that a similar result can be expressed with the column j such that the mass not on the diagonal of column j is (cid:80) i (cid:54) =
j T (cid:63)i,j = max( β j − α j , 0) .
This obviously means that for a given column/row index k
we have (cid:80) i (cid:54) =
k T (cid:63)i,k + T (cid:63)k,i = | α k − β k | .
Let’s write A k = (cid:80) i ≥ k,j ≥ k T (cid:63)i,j C i,j .
We have that W C ( α , β ) =
A 1 .
Now we remark that
[Equation]
We can write that
[Equation]
Since A N = 0 because C N,N = 0 , it turns out that
A 1 = (cid:80) k
=
( A k − A k +1 ) .
Lower bounding every elements of the sum by the previous minoration gives that:
[Equation]
which gives the desired results.
A. A DVERSARIAL SAMPLES COMPUTATION FOR WAR
WAR requires an efﬁcient computation of adversarial samples.
Following Miyato et al.
(2018), we choose the following computation model.
One could use the gradient with respect to the input r but because of differentiability, it vanishes in r = 0 .
When we approximate OT λ C in r = 0 through the second order Taylor expansion, we have
[Equation]
[Equation]
However, computing the hessian H r with respect to r = 0 is costly.
Instead we use the power iteration method (Golub & van der Vorst, 2000) to estimate the dominant hessian’s eigenvector that represent the direction in which the classiﬁcation function will change the most.
The algorithm is repeated k max times, but both the literature and our results suggest that only one iteration is sufﬁcient to achieve state of the art results.
Once the adversarial direction d is deﬁned, one can obtain the adversarial example with r = ε d / (cid:107) d (cid:107) 2 , by projecting onto the ball of radius ε .
A. M ODEL ARCHITECTURE , IMPLEMENTATION DETAILS AND TRAINING PROCEDURE A.4. B ENCHMARK DATASETS
We have used a 9 layer CNN following Han et al.
(2018) for the three image classiﬁcation benchmark datasets: Fashion-MNIST, CIFAR10, and CIFAR100 as shown in Table 5.
Between each layer we use a batch norm layer, a drop-out layer and a leaky-relu activation function with slope of 0 .
We use the Adam optimizer for all our networks with an initial learning rate of 0.001 with coefﬁcient ( β 1 , β 2 ) =
(0 . 9 , 0 . 999) and with mini-batch size of 256.
The learning rate is divided by 10 after epochs 20 and 40 for Fashion-MNIST (60 epochs in total), after epochs 40 and 80 for CIFAR- (120 epochs in total), and after epochs 80 and 120 for CIFAR-100 (150 epochs in total).
While training WAR, we set β = 0 for 15 epochs for faster convergence, as we observed that the network does not overﬁt on noisy labels at early stages of training.
The input images are scaled between [-1, 1] for Fashion-MNIST, and mean subtracted for the CIFAR10, and CIFAR100 datasets before feeding into the network.
The proposed method WAR, AR, and cross entropy loss functions are implemented in PyTorch, and for the Co-teaching 3 , Co-teaching+ 4 method we used the PyTorch code provided by the authors.
For the rest of the state-of-the-art methods (dimensionality driven learning 5 , forward and backward loss correction 6 , and robust loss functions 7 : unhinged and boot strapping) the experiments are conducted using the Keras code provided by respective authors.
We used similar layer initialization for all the methods in Pytorch and Keras.
[Table]
C LOTHING 1M DATASET
Regarding the training procedure for the Clothing1M dataset we give the following details.
Data pre-processing includes resizing the image to 256 x 256, center cropping a 224 x 224 patch from the resized image, and performing mean subtraction.
We used a batch size of 32 and learning rate of 0.0008 to update the network with SGD optimizer with a momentum of 0.9.
The learning rate is divided by 10 after 3 epochs (5 epochs in total).
A.4. V AIHINGEN DATASET
We now give the training procedure for our U-net on the ISPRS Vaihingen semantic labeling dataset.
The network was trained for 300 epochs (with 90 ◦ , 180 ◦ or 270 ◦ rotations and vertical or horizontal ﬂips as data augmentation) using the Adam optimizer with an initial learning rate of 10 − 4 and coefﬁcients ( β 1 , β 2 ) =
After 10 epochs, the learning rate is set to 10 − 5 .
Furthermore, we predict on the full image using overlapping patches (200 pixels overlap) averaged according to a
Gaussian kernel centered in the middle of the patch ( σ = 1).
A. S ENSITIVITY ANALYSIS OF β IN WAR, WAR 0 − 1 , AND AR
We conducted an experimental study to analysis the sensitivity of the trade-off parameter ( β ) between the cross entropy and the adversarial regularization term on CIFAR-10 dataset with 40% noise level.
The experimental results with different values of β are shown in Table 6 and the result reveals that as β increases, AR and WAR are robust to the label noise.
However for the higher β , AR does over-smoothing and decreases the classiﬁcation accuracy.
On the other hand, WAR increases the accuracy as β increases.
This behaviour shows the capability of our proposed method WAR to preserve the discrimination capability between similar classes.
It is noted that WAR points the gradient direction towards the low cost classes, as a result it does not over smooths between the conﬂicting classes, thus maintaining the discrimination ability.
Furthermore, WAR 0 − 1 with uninformative ground cost did not provide better results, and it is mostly similar with different values of β .
This observation reinstates need of having meaningful ground cost to capture the relationship between the classes in the dataset, and to guide gradient direction with respect to the ground cost.
[Table]
A. N OISY LABEL AT OBJECT SCALE FOR V AIHINGEN DATASET
We give the original and the used ground truth for Vaihingen dataset experiments.
A. D ISCUSSION ON C O -T EACHING +
In order to analysis low performance of Co-Teaching+(Yu et al., 2019), we conducted a series of experiments on CIFAR-10 with 40% noise level.
Firstly, we used a similar architecture as the one used in their paper (pairﬂip 45% noise level) and reproduced their results (39%, 43% for CoTeaching and Co-Teaching+ at 120 epochs).
Next we conducted experiments with different settings to analysis sensitiveness of Co-Teaching+, and the results are shown in Table 7.
When the pairﬂip noise and 2-layer CNN (one used in Co-Teaching+) is considered, Co-Teaching+ performs better than Co-Teaching, and it is opposite when the noise setting is changed.
When the wide capacity model is considered, Co-Teaching+ is inferior in both noise settings.
To understand this behaviour, we observed the pure ratio of the selected instances of both methods, and found that the pure ratio of selected instances decreases after few epochs of Co-Teaching+ loss (Co-Teaching+ uses a warm-up strategy with Co-Teaching method for 20 epochs).
For example with CIFAR-10 at 40% noise, the pure ratio of selected instances is approximately around 57%, whereas in Co-Teaching it is around 70%.
It is noted that in Yu et al.
(2019) considered a unrealistic noise simulation (Pairﬂip), which ﬂips the class labels with respect to the successive classes without considering class similarities, where as in our paper we have considered a class dependent noise, which ﬂips the label according to the class similar classes (Patrini et al., 2017).
[Table]