U NIFYING SEMI - SUPERVISED AND ROBUST LEARNING BY MIXUP
A BSTRACT
Supervised deep learning methods require cleanly labeled large-scale datasets, but collecting such data is difﬁcult and sometimes impossible.
There exist two popular frameworks to alleviate this problem: semi-supervised learning and robust learning to label noise.
Although these frameworks relax the restriction of supervised learning, they are studied independently.
Hence, the training scheme that is suitable when only small cleanly-labeled data are available remains unknown.
In this study, we consider learning from bi-quality data as a generalization of these studies, in which a small portion of data is cleanly labeled, and the rest is corrupt.
Under this framework, we compare recent algorithms for semi-supervised and robust learning.
The results suggest that semi-supervised learning outperforms robust learning with noisy labels.
We also propose a training strategy for mixing mixup techniques to learn from such bi-quality data effectively.
I NTRODUCTION
Learning from imperfect data is essential for applying machine learning, especially data-hungry deep learning, to real-world problems.
One approach to handling this problem is semi-supervised learning (SSL) where training data consist of a small amount of labeled data and a large amount of unlabeled data.
Another approach is robust learning to label noise (RLL), wherein all data are labeled but some of them are mislabeled.
SSL leverages large unlabeled data to improve the performance of supervised learning on a limited number of labeled data.
In the context of deep SSL, one effective method is to train neural networks to maintain consistency for a small perturbation of unlabeled inputs (Miyato et al.
; Tarvainen & Valpola (2017); Verma et al.
Oliver et al. (2018) refers these methods as consistency regularization.
In the RLL setting, learners need to enhance their performance using corrupted labels and avoid the performance deterioration caused by such data.
This requirement is particularly important for deep neural networks because they have ample capacity to remember whole samples even if their labels are completely random (Arpit et al.
; Zhang et al.
(2017a)).
To tackle this problem, some methods use a small amount of clean data to estimate noise transition matrix (Xiao et al.
Hendrycks et al.
or to learn to select possibly correctly-labeled samples (Jiang et al.
Jenni & Favaro (2018)).
Although both SSL and RLL aim to alleviate the limited-data problem, they have been studied independently and evaluated using different benchmarks.
However, if only a small amount of clean data is available, they can be regarded as similar problems.
In such as situation, can RLL outperform SSL under the same settings?
This question was our initial motivation to unify these two lines of research.
In this paper, we introduce a generalization of SSL and RLL, based on the concept of trusted data (Charikar et al.
; Hendrycks et al.
(2018)) in the literature of RLL.
More precisely, we assumed that some labels are guaranteed to be clean, and the rest are noisy.
The two learning frameworks can be uniﬁed by controlling the ratio of corrupted labels to all labels and the noisiness of label corruption.
Using the shared evaluation procedure in Oliver et al.
(2018), we compared recent SSL and RLL algorithms using image classiﬁcation task and found that the existing RLL methods using a small amount of clean data cannot outperform SSL under this setting.
This ﬁnding suggests that such RLL algorithms cannot use noisy labels effectively.
Therefore, it is necessary to adaptively use SSL and RLL in a data-driven manner.
As a baseline learning algorithm, we propose combining the mixup losses for SSL (Verma et al.
and RLL (Zhang et al. (2017b)); the results obtained are comparable to those of SSL- and RLL-speciﬁc methods and indicate the effective use of useful information from noisy labels.
L EARNING FROM BI - QUALITY DATA
In this section, we describe the setting of learning from bi-quality data, which is a generalization of
SSL and RLL.
In this formulation, we assume that the given data consist of two parts: trusted data D T (for which the labels are always correct) and untrusted data D U ( for which some labels might be wrong).
Learners are allowed to access the information irrespective of whether each sample is from D T or D U .
Let us denote the ratio of trusted and untrusted data to the entire data p as follows:
[Equation]
We also introduce quality as
[Equation]
[Equation] where p T ( y | x ) and p U ( y | x ) are the conditional probabilities of labels when inputs of trusted data and untrusted data, respectively, are given, and D is a divergence between two probability distributions (e.g., Kullback-Leibler divergence).
In equation 2, q ∈ [0 , 1] .
Obviously, q = 0 if the labels are completely independent of the input features (i.e., completely random), and q = 1 if the labels are clean.
We assume that the quality of untrusted data q U is in [0 , 1] , and that of trusted data q T is 1 .
We believe that this setting is realistic.
Suppose that there are unlabeled data.
Under budget constraint, one strategy to label this dataset as training data is to divide the data to two parts and spend most of the budget on high-quality labeling to acquire D T (e.g., using experts) and the rest on lower-quality labeling to obtain D U (e.g., using crowdsourcing), where usually | D T | ≪ | D U | .
Under this generalized framework, SSL is equivalent to the particular case where q = 0 , that is, labels of untrusted data are entirely random and give no useful information.
In SSL, such labels are usually ignored, and the data are treated as unlabeled.
Meanwhile, RLL without trustable data is another special case, where p = 0 and 0 󰃥 q < 1 .
In some studies on RLL, the setting p >is used explicitly (Charikar et al.
; Li et al.
; Hendrycks et al.
(2018)) and implicitly (Jiang et al.
; Jenni & Favaro (2018)).
Intuitively, when q is relatively high, we can expect some performance gain by utilizing untrusted, but somewhat informative, labels with RLL.
On the contrary, when q is almost zero, untrusted data are not informative anymore, and better generalization may be obtained using only standard SSL.
A critical problem in practice, however, is that we cannot know the exact quality of untrusted data, and, therefore, we cannot decide the best learning strategy in advance.
To handle such data, we need an adaptive mechanism to properly fuse SSL and RLL in a data-driven manner.
To realize this goal, we propose to combine techniques for SSL and RLL.
We use the convex combination of loss functions for SSL L semi and RLL L robust .
That is, with an additional hyperparameter γ , the loss function for untrusted data is deﬁned as follows:
[Equation]
For L robust and L semi , any loss functions for SSL and RLL can be used 1 .
In this study, as a baseline of this setting, we combine mixup techniques for SSL
Prepare trusted set: D T , untrusted set: D U Initialize neural network f
Set hyper parameters: α , β , γ
Set loss function: L ( · , · ) : categorical cross entropy for k ∈ { 0 , 1 , . .
, K − 1 } : Sample λ
[Equation]
Sample ( x i , y i ) ,
[Equation]
x j
[Equation]
) from D T
[Equation]
Sample (
[Equation]
from D U
Predict y ′′ i =
f
( x ′ i ) and y ′′
j = f ( x ′ j )
[Equation]
[Equation]
[Equation]
Update parameters of f with
L T + σ ( k ) L U ⊲ For σ , see Section  MIXMIXUP mixup is a regularization technique, where neural network models are trained on virtual training pairs (˜ x, ˜ y ) , where
[Equation]
[Equation]
Here, ( x i , y i ) and ( x j , y j ) are sampled from training data, and λ ∈ [0 , 1] is sampled from Beta distribution.
Zhang et al.
(2017b) showed that this method alleviates the performance decrease under label corruption.
In Verma et al.
, mixup is used for SSL as consistency regularization (Oliver et al.
Here are learned to λ x U
i + (1 − λ ) x
U j to λ y Ui + (1 − λ ) y Uj , where y U i , y U
j are the predicted outputs corresponding to x U i , x U j .
We show the details in Algorithm 1.
The parameters of neural networks are updated with a combination of losses on trusted data L T and untrusted data L U as L T + σ ( k ) L U .
We use sigmoid scheduling σ ( k ) as Verma et al.
Because this method mixes the mixup losses, hereinafter we refer to our method as mixmixup .
E XPERIMENTS 4. S ETTINGS
We use CIFAR-10 (Krizhevsky (2009)), which has 50,000 of 10-category images.
Following the common protocol of SSL in Oliver et al.
(2018), we split the training dataset into 45,000 images for training and 5,000 for validation.
To simulate bi-quality data, we sample 4,000 examples from the training data as trusted data and the rest as untrusted data.
We randomly replace each label of the untrusted samples with another one with a given probability, which is a common protocol in RLL (Zhang et al.
(2017b)
; Jiang et al.
We use the non-corrupted validation data and test data for hyperparameter tuning and ﬁnal evaluation, respectively.
Following Oliver et al.
(2018) and Verma et al.
(2018), we use WRN-28-2 (Wide ResNet 28-2, proposed in Zagoruyko & Komodakis (2016)) as the image classiﬁer.
To optimize the network, we use SGD with a learning rate of 0 . 1 , a momentum of 0 . 9 , a weight decay of 1 .
× 10 − 4 and a minibatch size of 256 .
We train networks for 1 . 6 × 10 5 iterations, following Verma et al.
We implement the model with PyTorch v1.0 (Paszke et al.
and tune hyperparameters α , β and γ in Algorithm 1 with a Bayesian optimization algorithm in Optuna v0.8 2 .
4. R ESULTS 4.2. L EARNING FROM BI - QUALITY DATA
According to the settings in Section 4.1, we trained WRN-28-2 with bi-quality data of different quality.
We randomly replace 40% and 100% of the labels of the untrusted data to simulate label
[Table] [Table]
corruption. From the deﬁnition of quality (equation 2), the quality of these data is 0.6 and 0.0, respectively.
The former corresponds to RLL with trusted data, and the latter to SSL.
4.2. C OMPARISON WITH OTHER METHODS
Table 2 presents the test accuracy of SSL methods (Verma et al.
; Tarvainen & Valpola (2017);
Miyato et al.
(2018), Table 2 A) and RLL methods with trusted data (Jiang et al.
; Hendrycks et al.
Table 2 B) with the results obtained using mixmixup (Table 2 C).
In all the experiments, the same network (WRN-28-2) and the same dataset split mentioned above are used.
Note that SSL methods in Table
A does not use labels of untrusted data.
Surprisingly, our results suggest that SSL (Table 2 A) can provide better performance than RLL (Table 2 B) with trusted data for identical settings.
This result suggests the difﬁculty of learning robustly from partially corrupted labels preventing overﬁtting.
Further, mixmixup can handle both SSL and RLL.
Moreover, the accuracy when the quality is 0. (corresponding to RLL) is superior to that for quality 0.0 (corresponding to SSL), indicating that mixmixup can effectively use information from corrupted labels.
C ONCLUSION
In this paper, we introduce a novel framework of weakly supervised learning by unifying SSL and RLL, which have been independently studied.
To handle this problem, we propose to mix mixup for SSL and RLL.
This method empirically works well and achieves competitive results with semisupervised and robust learning speciﬁc methodologies.
In addition, our experiments indicate that the performance of some RLL with trusted data might be inferior to that of SSL under identical settings.
This result suggests that the existing RLL methods cannot effectively exploit the information which should be extracted from noisy labels.
Our proposed method does not use the estimated quality; instead some hyperparameters are introduced.
The use of quality estimation may ease hyperparameter tuning, but is still an open question. A CKNOWLEDGEMENT
This work was supported by JSPS KAKENHI Grant Number JP19H04166.