L EARNING TO I MPUTE : A G ENERAL F RAMEWORK FOR S EMI - SUPERVISED L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Recent semi-supervised learning methods have shown to achieve comparable results to their supervised counterparts while using only a small portion of labels in image classiﬁcation tasks thanks to their regularization strategies.
In this paper, we take a more direct approach for semi-supervised learning and propose learning to impute the labels of unlabeled samples such that a network achieves better generalization when it is trained on these labels.
We pose the problem in a learning-to-learn formulation which can easily be incorporated to the state-of-the-art semi-supervised techniques and boost their performance especially when the labels are limited.
We demonstrate that our method is applicable to both classiﬁcation and regression problems including image classiﬁcation and facial landmark detection tasks.
I NTRODUCTION
Semi-supervised learning (SSL) (Chapelle et al., 2009) is one of the approaches to learn not only from labeled samples but also unlabeled ones.
Under certain assumptions such as presence of smooth prediction functions that map data to labels, of low-dimensional manifolds that the high-dimensional data lies (Chapelle et al., 2009), SSL methods provide a way to leverage the information at unlabeled data and lessens the dependency on labels.
Recent work (Tarvainen & Valpola, 2017; Miyato et al., 2018; Berthelot et al., 2019; Xie et al., 2019) have shown that semi-supervised learning by using only a small portion of labels can achieve competitive results with the supervised counterparts in image classiﬁcation tasks ( i.e. CIFAR10, SVHN).
They built on a variation of the well-known iterative bootstrapping method (Yarowsky, 1995) where in each iteration a classiﬁer is trained on the current set of labeled data, the learned classiﬁer is used to generate label for unlabeled data.
However, the generalization performance of this approach is known to suffer from ﬁtting the model on wrongly labeled samples and overﬁtting into self-generated labels (Tarvainen & Valpola, 2017).
Thus, they mitigate these issues by various regularization strategies.
While there exist several regularization (Srivastava et al., 2014) and augmentation (Zhang et al., 2018;
DeVries & Taylor, 2017) techniques in image recognition problems which are known to increase the generalization performance of deep networks, speciﬁc regularization strategies for semi-supervised classiﬁcation are used to estimate correct labels for the unlabeled data by either encouraging the models to produce conﬁdent outputs (Lee, 2013; Grandvalet & Bengio, 2005) and/or consistent output distributions when its inputs are perturbed (Tarvainen & Valpola, 2017; Miyato et al., 2018; Berthelot et al., 2019).
The assumption here is that if a good regularization strategy exists, it can enable the network to recover the correct labels for the unlabeled data and then the method can obtain similar performance with the supervised counterpart when trained on them.
Though this ad-hoc paradigm is shown to be effective, it raises a natural question for a more direct approach: Can we instead encourage the network to label the unlabeled data such that the network achieves better generalization performance when trained with them?
In this paper, we propose a new learning-to-learn method for semi-supervised learning that can be put in a meta-learning framework to address this question.
Our method involves learning an update rule to label unlabeled training samples such that training our model using these predicted labels improves its performance not only on itself but also on a meta-validation set.
Crucially, our method is highly generic and can easily be incorporated to the state-of-the-art methods (Lee, 2013; Berthelot et al., 2019) and boost their performance, in particular, in the regime where the number of available labels is limited.
Moreover, our method is not limited to classiﬁcation problems, we show that it can be extended to semi-supervised regression tasks where the output space is continuous and achieves signiﬁcant performance gains.  R ELATED W ORK Semi-supervised classiﬁcation.
There is a rich body of literature in SSL (Chapelle et al., 2009) for classiﬁcation.
Most of the recent work (Lee, 2013; Tarvainen & Valpola, 2017; Miyato et al., 2018;
Berthelot et al., 2019; Xie et al., 2019) builds on the idea of the bootstrapping technique of Yarowsky (1995) that involves iterative optimization of classiﬁer on a set of labeled data and reﬁnement of its labels for the unlabeled data.
This paradigm is known to overﬁt on the noisy self-generated labels and thus suffer from low generalization performance. To alleviate the sensitivity to inaccurate labeling, researchers introduce various regularization strategies. Grandvalet & Bengio (2005) propose a minimum entropy regularizer that encourages each unlabeled sample to be assigned to only one of the classes with high probability. Lee (2013) instead follow a more direct approach, use the predicted label with the maximum probability for each unlabeled sample as true-label which is called “pseudolabel”.
An orthogonal regularization strategy is to encourage a classiﬁer to be invariant to various stochastic factors and produce consistent predictions for unlabeled data when the noise is added to intermediate representations (Srivastava et al., 2014) and to input in an adversarial manner (Miyato et al., 2018).
In the same vein, Laine & Aila (2017); Tarvainen & Valpola (2017) introduce the Π -model and Mean-Teacher that are regularized to be consistent over previous training iterations by using a temporal ensembling and teacher/student networks respectively.
Recently Berthelot et al.
(2019) introduced MixMatch algorithm that further extended the previous work by unifying the idea of the consistency regularization and augmentation strategies (Zhang et al., 2018).
While the recent techniques are shown to be effective in several classiﬁcation benchmarks, the idea of consistency regularization is still implicit in terms of generalization performance.
Here we take an orthogonal and more direct approach to this approach and learn to impute the labels of the unlabeled samples that improve the generalization performance of a classiﬁer.
We also show that our method can be used with the recent SSL techniques.
Semi-supervised regression.
Some of the recent techniques in semi-supervised classiﬁcation are shown to be applicable to regression problems.
Jean et al.
(2018) have adapted various existing
SSL methods such as label propagation, VAT (Miyato et al., 2018) and Mean Teacher (Tarvainen & Valpola, 2017) and studied their performance in regression.
The same authors also proposed a Bayesian SSL approach for SSL regression problems which is based on the recent deep kernel learning method (Wilson et al., 2016) based approach for semi-supervised learning that aims at minimizing predictive variance of unlabeled data.
As in the classiﬁcation task, these methods are typically ad-hoc and does not aim to generate labels for the unlabeled data that are optimized to improve the generalization in regression.
Meta-learning.
Our method is also related to meta-learning (Schmidhuber, 1987; Bengio et al., 1992) and inspired from the recent work (Andrychowicz et al., 2016;
Finn et al., 2017) where the goal is typically to quickly learn a new task from a small amount of new data.
Andrychowicz et al.
(2016); Finn et al.
(2017) propose a learn gradient through gradient approach to train a model which has a strong generalization ability to unseen test data and can be easily adapted to new/unseen tasks.
Sung et al.
(2018) introduce the relation network to learn an additional metric such that the learned model’s feature embedding can be generalized for unseen data and unseen tasks.
Ren et al.
(2018b) adopt the meta learning to learn the weight of samples to tackle the sample imbalance problem.
Lorraine & Duvenaud (2018); MacKay et al.
(2019) employ meta learning for automatically learning the hyper-parameters of the deep neural networks.
Meta-learning has recently been applied to unsupervised learning (Hsu et al., 2019) and SSL for few shot learning (Ren et al., 2018a).
Ren et al.
(2018a) adapts the prototypical network to use unlabeled examples when producing prototypes, enabling the prototypical network to exploit those unlabeled data to assist few shot classiﬁcation.
Antoniou & Storkey (2019) propose to learn a label-free loss function, parameterized as a neural network that enables the classiﬁer to leverage the information from a validation set and achieves better performance in few-shot learning.
(Li et al., 2019) propose a meta-learning technique to initialize a self-training model and to ﬁlter out noisy pseudo labels for semi-supervised few-shot learning.
Similarly, our work also builds on the ideas of optimizing for improving the generalization for unseen samples.
However, in contrast to the existing meta-learning methods that is proposed for few-shot learning problems, we focus on semi-supervised learning in general classiﬁcation and regression problems where the number of samples are not limited to few.
M ETHOD
Consider a dataset D that consists of |L| labeled samples L = { ( x 1 , y 1 ) , ( x 2 , y 2 ) , . .
, ( x |L| , y |L| ) } which is further split into a training T and meta-validation set V with |T | and |V| samples respectively.
We also have a set of unlabeled samples U
= { x u } i =1 ,..., |U| .
Further, we let Φ θ denote a model (a function parameterized for instance, as a deep neural network) with parameters θ , that is trained to predict labels y from given samples x as y =
Φ θ ( x ) .
We are interested in imputing missing labels z of the unlabeled samples that are not only accurate but also improve performance at inference time, when included in our training set for optimizing model parameters θ .
A straightforward approach to this problem is to ﬁrst train a model to optimize the following cost function on the training set T :
[Equation]
where (cid:96) is a task speciﬁc loss, such as a soft-max followed by a cross-entropy loss for classiﬁcation or squared loss for regression tasks.
One can then use the trained model Φ θ to impute labels for samples in the unlabeled set U as z =
Φ θ (˜ x u ) as in Tarvainen & Valpola (2017); Berthelot et al. (2019), where ˜ x u denotes randomly perturbed input image x u ( e.g. random crops). Note that the label imputation procedure can be replaced with other pseudo-label prediction techniques.
Now we can expand T by adding the unlabeled samples with their imputed pseudo-labels i.e. A = T ∪ U and further train θ by using Eq.
However, as many pseudo-labels z will likely be noisy, there is no guarantee that training on the augmented train set will improve the performance of the model on the meta-validation set V :
[Equation]
Note that in all our experiments V is randomly sampled from T which ensures that our method is not trained using any additional data.
Our goal is also to minimize the expected value of the loss in Eq.
(1) with respect to the model parameters θ via an algorithm such as stochastic gradient descent (SGD).
Here, we go a step further and consider how the label imputation affects the ability of SGD to optimize the loss in Eq.
We pose this as a meta-learning problem, which is derived next.
We start by considering the loss value of the model Φ θ on the augmented training set A :
[Equation]
We ﬁrst simulate a step of SGD using the loss in Eq.
(3) to drive the update of the model parameters θ .
At step t , SGD updates the parameters for minimizing Eq. (3) as follows:
[Equation]
where ∇ θ is the gradient operator.
We then wish to ﬁnd z that minimizes the meta-validation objective C V (ˆ θ t +1 ) in Eq.
(2) with ˆ θ t +1 .
This corresponds to the following bilevel optimization problem min z
[Equation]
Our goal is to learn predicted pseudo-labels that minimize the meta-validation loss in Eq. To this end, we propose two options, denoted as Option 1 and Option 2 . In the former, we treat z as latent (or learnable) parameters and compute the gradients of the meta-validation loss w.r.t z to update z , which is used to update model parameters θ .
In the latter, z is considered as the output of network Φ θ and the gradients are thus computed w.r.t the model parameters θ for updating θ . Optimization. The gradients of C V w.r.t z for Option 1 and w.r.t θ for Option 2 are respectively be written as:
[Equation]
where I is identity matrix and ∇ θ ∇ θ indicates a second order derivative which is supported by standard deep learning libraries such as Pytorch Paszke et al. Tensorﬂow Abadi et al. We then use the gradients in Eq.
(6) or Eq.
(7) to update the model parameters such that the updated model can predict pseudo-labels that minimize the meta-validation loss. In Option 1, we update the pseudo-labels as ˆ z = z − η z d C V d z and the updated pseudo-labels are then used to update the model parameters θ by minimizing the loss C U =
(cid:80) x u ∈U (cid:96) (Φ θ t ( x ui ) , ˆ z i ) . In Option 2, the gradient in Eq.
{+(7) is used to update the model parameters as θ t +
+}
=
θ t − η ∂ C V
∂θ t We further depict the optimization details of Option 1 and 2 in Alg.
We ﬁrst estimate the z with the current model θ t , update the model to optimize the loss C A and then re-estimate z with ˆ θ t .
This part is so far similar to the self-labeling method of Lee (2013).
The loss function C A can trivially be replaced by those used in recent work such as Mean-Teacher (Tarvainen & Valpola, 2017) or MixMatch (Berthelot et al., 2019); we applied our method in conjunction with all three loss functions and report results in Section 4.
Next we apply the updated model to a mini-batch of unlabeled samples, compute the loss and simulate a SGD step w.r.t this loss.
In Option 1, we initialize z with the output of Φ θ (˜ x u ) and update it by using the gradient from Eq. In Option 2, the meta-update is computed w.r.t θ and used to update the model parameters by using the gradient from Eq. Input :
[Equation]
U
[Equation]
V (cid:46) training set
, unlabeled and meta-validation set
resp.
[Table]
for t = 0 , · · · , K − 1 do z = { z i } i =1 , ··· , |B U t | = { Φ θ t (˜ x ui ) } i =
[Equation]
··· , |B U t | (cid:46) estimate pseudo labels for unlabeled samples, ˜ x ui is randomly perturbed input x ui
C
[Equation]
(cid:46) update the model and move to the meta update
[Equation]
estimate pseudo labels using the updated model
[Equation]
simulate a SGD step
C
[Equation]
(ˆ
[Equation]
) (cid:46) evaluate on validation data if option 1 then ˆ z
[Equation]
(cid:46) correct pseudo labels
C
[Equation]
) θ[-t +
 = ˆ θ t − η-]
[Equation]
∂ ˆ θ t (cid:46) update the model with corrected pseudo labels else if option 2 then θ t +
=
ˆ θ t − η ∂ C V
∂
ˆ θ t (cid:46) update the model directly with meta-gradient
Output : θ K end
[-Option 1: z as latent parameters. In this case, we create a z vector for each x u , initialize z t = with the output of Φ t =
 θ ( x ) and update it by using the gradient from Eq. In case of classiﬁcation, the pseudo-labels are obtained after applying a softmax on jittered input ( e.g. random cropping) or a gumbel-softmax Jang et al.-]E XPERIMENTS
We evaluate the performance of our method on multiple classiﬁcation and regression benchmarks, and analyze the results below.
Note that we use the validation sets only for tuning hyperparameters
[Figure]
such as learning rates and for early stopping. The meta-validation sets ( C V in Eq. (5)) are sampled from the training set in all our experiments as in Rajeswaran et al.
Finn et al. (2017); Vinyals et al. ; Sung et al. During training, we sample two mini-batches of data from the same training set at each iteration: one acts as training and the other one acts as the meta-validation set to ensure that our method is not trained on more data than the baselines.
We train the model on the former and optimize the pseudo labels on the latter.
4. C LASSIFICATION RESULTS
Toy datasets.
We ﬁrst validate our method on a synthetic binary classiﬁcation dataset, two noisy concentric circles 2 , where each circle corresponds to a class.
To this end, we generate 10,000 samples and randomly pick 50, 2500 and 1000 for training, testing and validation sets respectively and use the rest of the samples as unlabeled data.
Figure 1 illustrates the dataset where the labels for both test (top row) and unlabeled data (bottom row) are indicated in blue and orange colors.
The labeled training samples for two classes are shown in green and pink in GT.
Experiment 1.
In Fig.
1 we depict the predictions of two baselines: supervised learning (SL) which is trained only on the labeled training set ( i.e. green and pink samples), Pseudo-Labeling (Lee, 2013) (PL) which iteratively ﬁrst trains a network on the labeled and unlabeled data and then re-labels the unlabeled ones, and two variants of our method.
For all the methods, we use a shallow small capacity network containing two fully-connected layers ( 2 fc −→ 4 fc −→ 2 ), one leaky ReLU activation layer in-between and a sigmoid function at the end.
We ﬁrst observe that SL misclassiﬁes most of the outer circle points and predicts them as inner circle labels.
This is due to the limited labeled data and its non-uniform spread over the input space.
We see that PL signiﬁcantly improves over SL by leveraging the unlabeled data.
However many of its predictions are still inaccurate, especially in the regions of low label density and high ambiguity.
This occurs because the iterative relabeling procedure is sensitive to the initial labeling of the unlabeled samples and thus it gets stuck in a non-optimal minimum.
Both the variants of our method largely overcome the bias in the training data by labeling the unlabeled samples in a way that they lead to accurate predictions on the meta-validation set.
This shows that the meta-updates are successful to correct the imputed labels based on the signal from the meta-validation set (see Fig.
1 bottom row)
and to prevent our method to simply overﬁt into the pseudo-labels.
While both variants agree on the most of the samples, they slightly differ in the low-density regions ( i.e. the area between the circles).
CIFAR-10 & -100.
We also evaluate our method on the CIFAR image classiﬁcation benchmarks (Krizhevsky et al., 2009) that are commonly used for evaluation of both supervised and semi-supervised classiﬁcation.
Both datasets contain 50,000 and 10,000 training and testing samples respectively.
In our experiments, we strictly follow the training and testing protocol for semi- supervised learning which is proposed by the previous work (Oliver et al., 2018; Berthelot et al., 2019) where 5000 of training samples are used as validation data, the remaining 45,000 training samples are split into labeled and unlabeled training sets.
As in Oliver et al.
(2018), we randomly pick |T | samples as labeled data and the rest ( |U| = 45 , 000 − |T | samples) are unlabeled data.
We report results for multiple training data-regimes |T | = 250 , 500 , 1000 , 2000 , 4000 on CIFAR-10 and |T | = 1000 , 2000 , 3000 , 4000 , 5000 on CIFAR-100.
Note that we do not use the validation data in training of the model parameters but only for hyperparameter selection.
Experiment 2.
First we compare our method to the SL and PL baselines for various number of training samples and report their performance in Table 1.
We use a 13-layer conv-net as the classiﬁer for all the methods (see Appendix A for details).
First we observe that PL improves over SL by leveraging the unlabeled data in all the settings.
Both our methods (option-1/2) that builds on PL achieves substantial performance gains over PL.
Interestingly the relative improvements are higher in the more challenging case of when the labeled data is limited.
We also observe that option-2, which involves updating the model parameters with meta-gradient, is a better strategy for these benchmarks. [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] PL+ [Table] Experiment 3.
Here we show that our method can be incorporated to the state-of-the art methods such as Mean Teacher (MT) (Tarvainen & Valpola, 2017) and MixMatch (MM) (Berthelot et al., 2019) that use more sophisticated backbone networks, augmentation and regularization strategies, and also boost their performance.
For this experiment, we follow the implementation of Berthelot et al.
(2019) – we adopt a more competitive backbone, WideResNet-28-2, use the Adam optimizer along with standard data augmentation techniques (see Appendix A for more details). [Table] [Table] [Table]
[Table]
[Table] [Table] [Table] Table 2 depicts the classiﬁcation error rate for the several state-of-the-art techniques including Π model (Laine & Aila, 2017), PL (Lee, 2013), VAT (Miyato et al., 2018), MT (Tarvainen & Valpola, 2017) and MM (Berthelot et al., 2019).
Note that some methods do not report results on CIFAR-100.
All the results except MT and MM are taken from the original papers.
As our methods are built on
MM and MT, we show the results of our own implementation which are on par with the published ones.
From the table, we see that our method achieves signiﬁcant improvement over MT baseline, especially in the low label regime, up-to 11 points in case of 250 labels in CIFAR-10.
Again our second variant consistently outperforms the ﬁrst one when used with the MT in both CIFARand -100, whereas the more competitive MM baseline already produces accurate pseudo-labels in
CIFAR-10 and so the two options perform comparably.
The reason is, in option2, we obtain the meta gradient on the model parameters and update the model directly, whereas in option1, we ﬁrstly update the pseudo labels and then train the model on them.
Though updating the pseudo labels can improve the performance (e.g. MT + option1 vs MT), some of the labels can still be noisy after the update and optimizing the model on the noisy ones may degrade the performance.
In contrast, in option2, the meta gradient is applied to the network parameters directly, which alleviates the potential pseudo-label noise.
In case of MM, our method is able to boost its performance only in case of few labels (250 for
CIFAR-10 and 1000 for CIFAR-100) and does comparable and slightly worse in case of more labels.
We believe that it is harder to improve the performance of MM in the CIFAR, as its performance approaches to the supervised counterpart quickly.
We leave the evaluation of MM on a more challenging benchmark as future work and below we show results for tasks where MM is not applicable.
4. R EGRESSION RESULTS
AFLW.
Next we move to a regression task and use the Annotated Facial Landmarks in the Wild (AFLW) dataset (Koestinger et al., 2011; Zhang et al., 2015) where we aim at predicting 5 landmarks’ location of faces in images.
The AFLW is originally designed for supervised facial landmark detection.
We use the ofﬁcial train and test splits, randomly pick 10% of samples of the original training set as the validation set only for hyperparameter tuning and early stopping, and use the rest of this data as labeled and unlabeled data in our experiments.
We evaluate the baselines and our method for 1%, 2%, 5%, 10% of training data as labeled data and report the standard Mean Square Error (MSE) normalized by the inter-ocular distance as in Zhang et al.
Experiment 4.
For the regression task, we adopt the TCDCN backbone architecture in Zhang et al.
(2015) and
SGD optimizer, use standard data augmentation.
We train all methods for 22500 steps. and set the initial learning rate to 0 .
03 and reduce it by 0.1 every 750 steps.
The momentum and the weight decay are set to 0.9 and 0.0005.
Here we use SL, PL and MT as the baselines and also build our method on both the PL and MT.
Note that MM is not applicable to this task, as mixing up two face images doubles the number of landmarks.
Table 3 depicts the results for the baselines and ours in terms of mean error rate.
[Table]
First we observe that the supervised learning on 1% of the labels is very challenging and obtains only 16.31% which is on par with the performance of simply taking the mean of each facial landmark over all training samples (16.58%).
As expected, using the unlabeled face images is beneﬁcial and both PL as well as MT methods signiﬁcantly improves over SL.
Our method achieves consistent improvement over PL and MT for different portions of labels.
This strongly suggests that our method is able to reﬁne the estimated landmarks of PL and MT on the unlabeled images and further improve its performance.
We also analyze the effect of meta-updates during training our model in Fig.
To this end, we ﬁrst plot the regression loss on the meta-validation batch before and after the meta-update in the left ﬁgure.
This clearly shows that updating the pseudo landmark positions in the unlabeled images lead to a better accuracy on the meta-validation samples.
Second we show the test
Though both option-1 and 2 improve over the baselines, here option-1 outperforms option-2 on the regression problem which is in contrast to the classiﬁcation tasks above.
This is possibly due to the fact that the output space of the landmarks is continuous and less constrained than the label space for classiﬁcation which makes the option-2 more prone to overﬁt to the validation set.
A promising direction which is worth investigating in future is to alleviate the overﬁtting by using a regularizer to enforce the structure in the output space.
[Figure]
We also illustrate success/failure cases on the test images in Fig.
3 and depict the ground-truth, predicted landmarks of MT and our method when trained with 1% of the labeled data.
The performance difference is visually signiﬁcant and our method outputs more accurate landmarks than MT.
The bottom row shows the cases of extreme pose variation and occlusion where both MT and ours fail to achieve accurate predictions.
C ONCLUSION
In this paper we have propose a general semi-supervised learning framework, it learns to impute the labels of the unlabeled data such that the training a deep network on these labels improves its generalization ability.
Our method can easily be used in conjunction with several state-of-theart semi-supervised methods and extended to multiple classiﬁcation and regression tasks such as image classiﬁcation and facial landmark detection.
We show that our method achieves signiﬁcant performance gains over competitive baselines in challenging benchmarks, especially when the labeled data is scarce.
As future work, we plan to extend our method to semi-supervised learning in structured output problems.
A A PPENDIX
To evaluate our method with existing semi-supervised learning, we implement our method and other methods in Pytorch.
A. T OY E XPERIMENTS
Experiment 1.
To conduct experiments on the concentric circles dataset, we use a small network
containing
two fully-connected layers ( 2 fc −→ 4 fc −→ 2 ) and one leaky relu activation layer.
We compare our option 1 and option 2 to the Pseudo Label (PL) Lee (2013) and supervised learning approach and we train all methods for 20 × 20 steps ( i.e. 20 epochs and 20 steps per epoch).
We use Adam as the optimizer and the learning rate is 0.03.
The maximum of the unsupervised loss weight λ for both PL and Ours is set to 1 and we adopt the linear-schedule for increasing the weight on the unsupervised loss.
Speciﬁcally, the λ is initialized at 0 and increases to 1 gradually in 20 × 5 steps by linear-schedule as Berthelot et al.
During the meta update in both our option1 and option2, we estimate the pseudo labels of unlabeled samples by gumbel softmax and compute the Mean Square
Error (MSE) between prediction and pseudo labels as the unlabeled loss.
η z is 1 for all experiments.
A. C LASSIFICATION E XPERIMENTS ON CIFAR-10 & -
For all experiments on both CIFAR-10 and CIFAR-100, we follow the optimizing strategy in Berthelot et al.
(2019), i.e. we adopt Adam as the optimizer and ﬁx the learning rate.
We evaluate the model using an exponential moving average of the learned models’ parameters with a decay rate of 0.999.
In addition, the weight decay is set to 0.02 for all methods.
We report the median error of the last 20 epochs for comparisons.
For image preprocessing, we use standard data augmentation such as standard normalization and ﬂipping, random crop as Berthelot et al.
During the meta update in both our option1 and option2, we predict the pseudo labels of unlabeled images by applying softmax on the augmentation of the original unlabeled image and compute the Mean Square Error (MSE) between prediction on the original image and pseudo labels as the unlabeled loss as in Berthelot et al.
Experiment 2.
We use a 13-layer conv-net Tarvainen & Valpola (2017); Laine & Aila (2017) in the experiments where we compare our method to the SL and PL on both CIFAR-10 and CIFAR-100.
The architecture of the network is illustrated in Fig.
In addition to this, we apply batch normalization on each convolutional and fully connected layers.
We use Leaky Relu with negative slope ( α = 0 .
1 ) as the nonlinear activation function on each convolutional layers.
On CIFAR-10, the batch size and learning rate is set to 50 and 0.003, respectively, as in Tarvainen & Valpola (2017) while on
CIFAR-100, the batch size is 128.
We train each method for 40 × 1000 steps.
For unsupervised loss weight λ , we use the linear-schedule and it increases gradually to 75 in 40 × (1000 × 0 . 4) steps (for both the PL and Ours).
𝐂 𝐨 𝐧 𝐯 ( 𝐊 𝐞 𝐫 𝐧 𝐞 𝐥 : 𝟑 , 𝐅 𝐢 𝐥 𝐭 𝐞 𝐫 : 𝟏𝟐𝟖 ) 𝐋 𝐞 𝐚 𝐤 𝐲 𝐑 𝐞 𝐥 𝐮 ( 𝛂 = 𝟎 . 𝟏 ) 𝐌 𝐚 𝐱 𝐏 𝐨𝐨 𝐥 ( 𝟐 ×)
Input Image 32×32× 𝐂 𝐨 𝐧 𝐯 ( 𝐊 𝐞 𝐫 𝐧 𝐞 𝐥 : 𝟑 , 𝐅 𝐢 𝐥 𝐭 𝐞 𝐫 : 𝟏𝟐𝟖 ) 𝐋 𝐞 𝐚 𝐤 𝐲 𝐑 𝐞 𝐥 𝐮 ( 𝛂 = 𝟎 . 𝟏 ) 𝐂 𝐨 𝐧 𝐯 ( 𝐊 𝐞 𝐫 𝐧 𝐞 𝐥 : 𝟑 , 𝐅 𝐢 𝐥 𝐭 𝐞 𝐫 : 𝟏𝟐𝟖 ) 𝐋 𝐞 𝐚 𝐤 𝐲 𝐑 𝐞 𝐥 𝐮 ( 𝛂 = 𝟎 . 𝟏 ) 𝐃 𝐫 𝐨 𝐩 𝐨 𝐮 𝐭 ( 𝐩 = 𝟎 .) 𝐂 𝐨 𝐧 𝐯 ( 𝐊 𝐞 𝐫 𝐧 𝐞 𝐥 : 𝟑 , 𝐅 𝐢 𝐥 𝐭 𝐞 𝐫 : 𝟐𝟓𝟔 ) 𝐋 𝐞 𝐚 𝐤 𝐲 𝐑 𝐞 𝐥 𝐮 ( 𝛂 = 𝟎 . 𝟏 ) 𝐌 𝐚 𝐱 𝐏 𝐨𝐨 𝐥 ( 𝟐 ×) 𝐂 𝐨 𝐧 𝐯 ( 𝐊 𝐞 𝐫 𝐧 𝐞 𝐥 : 𝟑 , 𝐅 𝐢 𝐥 𝐭 𝐞 𝐫 : 𝟐𝟓𝟔 ) 𝐋 𝐞 𝐚 𝐤 𝐲 𝐑 𝐞 𝐥 𝐮 ( 𝛂 = 𝟎 . 𝟏 ) 𝐂 𝐨 𝐧 𝐯 ( 𝐊 𝐞 𝐫 𝐧 𝐞 𝐥 : 𝟑 , 𝐅 𝐢 𝐥 𝐭 𝐞 𝐫 : 𝟐𝟓𝟔 )
𝐋 𝐞 𝐚 𝐤 𝐲 𝐑 𝐞 𝐥 𝐮 ( 𝛂 = 𝟎 . 𝟏 )
𝐃 𝐫 𝐨 𝐩 𝐨 𝐮 𝐭 ( 𝐩 = 𝟎 .)
𝐂 𝐨 𝐧 𝐯 ( 𝐊 𝐞 𝐫 𝐧 𝐞 𝐥 : 𝟑 , 𝐅 𝐢 𝐥 𝐭 𝐞 𝐫 : 𝟓𝟏𝟐 ) 𝐋 𝐞 𝐚 𝐤 𝐲 𝐑 𝐞 𝐥 𝐮 ( 𝛂 = 𝟎 . 𝟏 )
A v g 𝐏 𝐨𝐨 𝐥 ( 𝟔 ×) 𝐂 𝐨 𝐧 𝐯 ( 𝐊 𝐞 𝐫 𝐧 𝐞 𝐥 : 𝟑 , 𝐅 𝐢 𝐥 𝐭 𝐞 𝐫 : 𝟐𝟓𝟔 ) 𝐋 𝐞 𝐚 𝐤 𝐲 𝐑 𝐞 𝐥 𝐮 ( 𝛂 = 𝟎 . 𝟏 ) 𝐂 𝐨 𝐧 𝐯 ( 𝐊 𝐞 𝐫 𝐧 𝐞 𝐥 : 𝟑 , 𝐅 𝐢 𝐥 𝐭 𝐞 𝐫 : 𝟏𝟐𝟖 ) 𝐋 𝐞 𝐚 𝐤 𝐲 𝐑 𝐞 𝐥 𝐮 ( 𝛂 = 𝟎 . 𝟏 )
𝐅 𝐮 𝐥𝐥 𝐲 𝐂 𝐨 𝐧𝐧 𝐞 𝐜 𝐭 𝐞 𝐝 𝐋 𝐚 𝐲 𝐞 𝐫 (× 𝐧 𝐮 𝐦 _ 𝐜 𝐥 𝐚 𝐬𝐬 )
Effect of Meta-validation Batch Size.
To study the inﬂuence of the meta-validation batch size, we set the batch size of the meta-validation mini-batch to 25, 50, 100 (Note, in experiment 2 on
CIFAR-10, the batch size of the labeled, unlabeled and meta-validation data are set to 50 as in
Tarvainen & Valpola (2017); Laine & Aila (2017))
and report the results of two variants of our method applied to PL for 250 labels case on CIFAR-10 in Fig.
More speciﬁcally, our option using 25, 50, 100 samples as meta-validation data at each training iteration obtain 43.4 %, 43.
% and 43.72 %, respectively while option2 attains 42.7 %, 42.51 % and 42.47 %.
These results again strongly verify that both option1 and option2 achieve consistent improvements over the PL.
In addition, it is clear that the performance of our method is not sensitive to the batch size of the meta-validation mini-batch ( i.e. less than 0.4 %).
[Figure]
Experiment 3.
We adopt the WideResNet-28-2 as the network for all methods as Berthelot et al.
The batch size on CIFAR-10 is set to 32 and it is 128 on CIFAR-100.
The learning rate is initialized as 0.002 and ﬁxed.
We use the same consistency weight and ramp-up schedule (linear-schedule) to increase the unsupervised loss weight λ as Berthelot et al.
A. R EGRESSION EXPERIMENTS ON AFLW
Experiment 4.
We adopt the TCDCN proposed in Zhang et al.
(2015) as the network and SGD as optimizer.
An illustration of the TCDCN’s architecture is shown in Fig.
We adapt the Pseudo Label (PL), Mean Teacher (MT) and the supervised learning methods as the baselines and our method is built on PL as well as MT.
To estimate the loss on an unlabeled image, we ﬁrstly crop an image from the original image by moving the cropping window a random number of pixels.
We then estimate the location of landmarks on the augmented image and subtract the number of moving pixels, resulting in the pseudo label for the original image.
We then apply MSE to the prediction of the original image and the pseudo labels to estimate the loss.
We use the linear-schedule to update the unsupervised loss weight λ to 1 in 9000 steps. 𝐂 𝐨 𝐧 𝐯 ( 𝐊 𝐞 𝐫 𝐧 𝐞 𝐥 : 𝟓 , 𝐅 𝐢 𝐥 𝐭 𝐞 𝐫 : 𝟐𝟎 ) 𝐇 𝐚 𝐫 𝐝 𝐓 𝐚 𝐧 𝐡 𝐌 𝐚 𝐱 𝐏 𝐨𝐨 𝐥 (× 𝟐 ) 𝐂 𝐨 𝐧 𝐯 ( 𝐊 𝐞 𝐫 𝐧 𝐞 𝐥 : 𝟓 , 𝐅 𝐢 𝐥 𝐭 𝐞 𝐫 : 𝟒𝟖 ) 𝐇 𝐚 𝐫 𝐝 𝐓 𝐚 𝐧 𝐡 𝐌 𝐚 𝐱 𝐏 𝐨𝐨 𝐥 (× 𝟐 ) 𝐂 𝐨 𝐧 𝐯 ( 𝐊 𝐞 𝐫 𝐧 𝐞 𝐥 : 𝟑 , 𝐅 𝐢 𝐥 𝐭 𝐞 𝐫 : 𝟔𝟒 ) 𝐇 𝐚 𝐫 𝐝 𝐓 𝐚 𝐧 𝐡 𝐌 𝐚 𝐱 𝐏 𝐨𝐨 𝐥 (× 𝟐 ) 𝐂 𝐨 𝐧 𝐯 ( 𝐊 𝐞 𝐫 𝐧 𝐞 𝐥 : 𝟑 , 𝐅 𝐢 𝐥 𝐭 𝐞 𝐫 : 𝟖𝟎 ) 𝐇 𝐚 𝐫 𝐝 𝐓 𝐚 𝐧 𝐡 𝐅 𝐮 𝐥𝐥 𝐲 𝐂 𝐨 𝐧 𝐧 𝐞 𝐜 𝐭 𝐞 𝐝 𝐋 𝐚 𝐲 𝐞 𝐫 ( 𝟕𝟐×) 𝐅 𝐮 𝐥𝐥 𝐲 𝐂 𝐨 𝐧 𝐧 𝐞 𝐜 𝐭 𝐞 𝐝 𝐋 𝐚 𝐲 𝐞 𝐫 (× 𝟏𝟎 ) [Figure]