S EMI - SUPERVISED L EARNING BY C OACHING
Anonymous authors
Paper under double-blind review
A BSTRACT
Recent semi-supervised learning (SSL) methods often have a teacher to train a student in order to propagate labels from labeled data to unlabeled data.
We argue that a weakness of these methods is that the teacher does not learn from the student’s mistakes during the course of student’s learning.
To address this weakness, we introduce Coaching, a framework where a teacher generates pseudo labels for unlabeled data, from which a student will learn and the student’s performance on labeled data will be used as reward to train the teacher using policy gradient.
Our experiments show that Coaching signiﬁcantly improves over state-of-the-art
SSL baselines.
For instance, on CIFAR-10, with only 4,000 labeled examples, a
WideResNet-28-2 trained by Coaching achieves 96 .
11% accuracy, which is better than 94 .
9% achieved by the same architecture trained with 45,000 labeled.
On ImageNet with 10% labeled examples, Coaching trains a ResNet-50 to 72 .
94% topaccuracy, comfortably outperforming the existing state-of-the-art by more than 4% .
Coaching also scales successfully to the high data regime with full ImageNet.
Speciﬁcally, with additional 9 million unlabeled images from OpenImages, Coaching trains a ResNet-50 to 82 .
34% top-1 accuracy, setting a new state-of-the-art for the architecture on ImageNet without using extra labeled data.
I NTRODUCTION
Professional players in competitive sports such as chess, tennis, or swimming often have coaches to help improving their performance.
Although coaches typically do not play as well as the players, they observe the players and provide instructions to improve the players’ performance.
Modern semi-supervised learning (SSL) algorithms do not follow this strategy.
They instead have a teacher model that generates pseudo labels for unlabeled data, from which a student model learns by imitation ( e.g. , Lee (2013); Tarvainen & Valpola (2017); Laine & Aila (2017)).
A weakness of these methods is that the teacher does not adjust itself based on the student’s performance and cannot adapt to make the student better over time, unlike professional sport coaches develop their players.
Here, we propose a new semi-supervised learning method, called Coaching as shown in Figure 1, where the teacher learns throughout the course of student’s training.
In our method, a teacher generates pseudo labels for unlabeled data, from which the student will learn.
The student’s performance on labeled data will be used as reward to train the teacher with policy gradient .
Experiments show that our method achieves signiﬁcant improvements over state-of-the-art semisupervised learning baselines and can be up to 10 × more data efﬁcient than supervised learning.
For instance, with CIFAR-10, only using 4,000 labeled examples, a WideResNet-28-2 can be coached to 96 .
11% accuracy, outperforming the same model trained with 45,000 labeled examples which achieves 94 .
Meanwhile, on ImageNet, using ResNet-50 with only 10% labeled examples, our method achieves 72 .
94% top-1 accuracy, outperforming all existing semi-supervised learning methods with the same amount of labeled data, and approaching the top-1 accuracy of 76 .
3% of the same ResNet-50 trained with all labels.
Coaching also scales to the high data regime.
In particular, with all 1.28 million labeled examples from ImageNet, plus 9 million unlabeled and potentially out-of-distribution data from OpenImages (Kuznetsova et al., 2018), a ResNet-50 can be coached to the accuracy of 82 .
34% , which is a new state-of-the-art for the architecture without using extra labeled data.
M ETHOD
Notations.
Let T , S respectively be the teacher network and the student network in Coaching, and θ T , θ S be their corresponding parameters.
Since we work with both labeled data and unlabeled data, we use ( x lab , y lab ) to refer to a pair of an input and its corresponding label, and use x unl to refer to an unlabeled example.
In addition, we use (cid:96) ( x, y ; θ ) to denote the cross entropy loss computed on input x by with parameter θ on label y .
As shown in Figure 1, each training step in Coaching consists of two phases:
Phase 1: The student learns from data pseudo labeled by the teacher.
In this phase, the teacher
T ﬁrst performs a forward pass on x unl to compute the class distribution P ( ·| x unl ; θ T ) .
From this distribution, the teacher samples a pseudo label ˆ y unl ∼ P ( ·| x unl ; θ T ) .
The pair x unl , ˆ y unl is then shown to the student S to make an update on its parameters θ S .
The update is based on the gradient computed by back-propagating from the cross entropy loss.
For instance, if θ S is updated using SGD, then:
[Equation]
where η is the learning rate.
Phase 2: The teacher learns from the student’s loss.
After the student updates its parameters as in Equation 1, its parameters θ ( t +1) S is evaluated on a labeled example x lab , y lab using the cross entropy loss.
The goal of the teacher in Coaching is to give the pseudo labels ˆ y unl such that if the student is updated as in Equation 1, then the cross entropy loss (cid:96) ( x lab , y lab ; θ ( t +1) S ) will be minimized .
Clearly, (cid:96) ( x lab , y lab ; θ ( t +1) S ) depends on θ ( t +1) S , which in turn depends on the pseudo label ˆ y unl that the teacher samples.
From the perspective of reinforcement learning, ˆ y unl can be treated as an onpolicy action of the teacher, which leads to the reward of − (cid:96) ( x lab , y lab ; θ ( t +1) S ) .
In this perspective, we propose to train θ T to minimize the value of (cid:96) ( x lab , y lab ; ¯ θ ( t +1) S ) , where ¯ θ ( t +1) S is the expected destination that the teacher will guide the student to.
This expectation is taken over all possible pseudo labels ˆ y unl .
Formally,
[Equation]
R ( θ T )
where R
[Equation]
θ T
[Equation]
To ﬁnd θ ∗ T
[Equation]
we differentiate R ( θ T ) in Equation 2 with respect to θ T .
Here
[Equation]
we present the resulting gradient g ( t ) T , which has the form
[Equation]
The full derivation can be found in Appendix A, but intuitively, the differentiation depends on two tools.
The ﬁrst tool is the is the chain rule, which we leverage to differentiate R ( θ T ) with respect to θ T .
The second tool is the REINFORCE equation (Williams, 1992), which we leverage to establish the relationship between E ˆ y unl (cid:104) θ ( t +1) S (cid:105) and θ T .
Coaching combines the two steps above in an SGD step.
We summarize the method in Algorithm 1.
Generalize to an arbitrary batch size.
Above, we have only discussed Coaching for a single unlabeled data x unl and a single labeled data x lab , y lab .
Now, we describe how to scale Coaching to an arbitrary batch size.
Scaling x lab , y lab to a minibatch of labeled example, X lab , Y lab is straightforward, as we can simply replace all computations of the cross entropy (cid:96) ( x lab , y lab ; θ ( t +1) S ) with the average cross entropy on the minibatch (cid:96) ( X lab , Y lab ; θ ( t +1) S ) .
To scale a single unlabeled example x unl to a minibatch of unlabeled examples X unl = { x (1) unl , x (2) unl , ..., x ( B ) unl } , we treat each batch of pseudo labels ˆ Y unl as a compound action sampled from the joint distribution
[Equation]
Since every pseudo label ˆ y ( i ) unl is sampled independently, applying REINFORCE as in Equation 3 simply factors the per-instance cross entropy (cid:96) ( x unl , ˆ y unl ; θ T ) into the batch cross entropy (cid:80) Bi
=1 (cid:96) ( x ( i ) unl , ˆ y
( i ) unl ; θ T ) .
E XPERIMENTS
Compared to other methods that use both labeled data and unlabeled data, Coaching has three main advantages:
The teacher does not only demonstrate its knowledge to the student but also adjusts its teaching strategy in an adaptive manner with the student, throughout the course of the student’s learning.
The teacher in Coaching can beneﬁt from advanced SSL techniques such as consistency regularization.
The student in Coaching never learns directly from labeled data .
This does not only prevent overﬁtting when limited labeled data is available, but also allows us to ﬁnetune the trained student in Coaching directly on labeled data to further boost the student’s performance.
We perform experiments to verify the strength of Coaching.
In Section 3.1, we consider the low data regime with typical benchmarks for SSL methods.
After that, in Section 3.2, we consider the high data regime which contains potentially out-of-distribution data.
Model Architectures.
In our experiments, our teacher model and our student model always have the same architecture but with different weights.
For CIFAR-10 and SVHN, we use the WideResNet28-2 (Zagoruyko & Komodakis, 2016), which has 1.45 million parameters.
For ImageNet, we use a
ResNet-50 (He et al., 2016), which has 25.5 million parameters.
For experiments that train only one model, we apply exponential moving average with a decay rate of 0 .
99 on the weights of the model.
For experiments that have a teacher model and a student model, we apply this exponential moving average on the weights of the student model only.
Additional Implementation Details.
To improve the stability and accuracy of the method, we apply a few minor enhancements to the teacher :
Use cosine distance instead of dot product.
As the dot product h ( t ) in Equation 3 has a large value range, in order to stabilize training, we compute h ( t ) using the gradients’ cosine distance.
Use a baseline for h ( t ) .
To further reduce the variance of h ( t ) , we maintain a moving average b of h ( t ) and subtract b from h ( t ) every time we compute g ( t ) T as in Equation 3. 3.
Additional supervised loss for the teacher.
We ﬁnd that adding the supervised loss (cid:96) ( x lab , y lab ; θ T ) to the teacher’s objective results in a faster learning and better student.
Consistently regularize the teacher.
In the low data regime, consistency regularization improves the teacher and the student.
More details are in Section 3.1. 5.
Pre-training the teacher.
When the number of classes is large, it is beneﬁcial to initialize the teacher with a trained model so that the pseudo labels are better than random at the beginning of the student’s learning.
If we pre-train the teacher, Point 3 has minimal effect.
Finetuning the student.
Since the student in Coaching only learns from unlabeled data and pseudo labels generated by the teacher, ﬁnetuning a converged student on labeled data often improves the student’s performance.
The details mentioned above are mutually orthogonal.
Since (1) and (2) are crucial to stabilize the
Coaching process, they are always used in our experiments.
In addition, we apply (3) and (4) to the low data regime, and apply (5) for the high data regime for computational efﬁciency and strong performance.
We will explain these decisions in the corresponding sections.
3. R ESULTS ON L OW D ATA R EGIME
Datasets.
We consider three datasets with reduced numbers of labeled instances: CIFAR10 (Krizhevsky, 2009) with 4,000 labeled examples, SVHN (Netzer et al., 2011) with 1,000 labeled examples, and ImageNet (Russakovsky et al., 2015) with 128,000 labeled examples, which is approximately 10% of the whole ImageNet.
All images in these datasets are used as unlabeled examples, which means that even the labeled images can be used as unlabeled examples.
We use the image size of 32 × 32 for CIFAR-10 and SVHN, and the image size of 224 × 224 for ImageNet.
These datasets, label reductions, and image sizes are standard for low data image classiﬁcation.
Baselines.
We compare Coaching against 3 baseline training algorithms Purely Supervised, PseudoLabel (Lee, 2013), and Unsupervised Data Augmentation (UDA; Xie et al.
We discuss these baselines more in Section 4.
We choose these baselines for three reasons.
First, the purely supervised baseline serves to verify our implementation and to demonstrate the overﬁtting of our models when labeled data is scarce.
Second, comparing Coaching with Pseudo-Label conﬁrms the beneﬁts of continuing to train the teacher throughout the course of the student’s learning.
Finally, we compare against UDA because is the state-of-the-art on the datasets that we consider.
To ensure a fair comparison, we re-implement these baselines in our environment.
We follow Oliver et al.
(2018)’s train/eval/test splitting, and we use the same amount of resources to tune hyperparameters for our baselines as well as for Coaching.
More details are in Appendix C.
Additional baselines.
In addition to the three main baselines discussed above, we also include four other baselines: Temporal Ensemble (Laine & Aila, 2017), Mean Teacher (Tarvainen & Valpola, 2017), VAT (Miyato et al., 2018), LGA (Jackson & Schulman, 2019), ICT (Verma et al., 2019), and MixMatch (Berthelot et al., 2019).
We use results reported by Oliver et al.
Since these methods do not share the same controlled environment, the comparison to them is not direct, and should be contextualized as suggested by Oliver et al.
Data augmentations.
In our implementation of UDA and Coaching, we use RandomAugment, which is a randomized augmentation strategy over all the operations in the search space of AutoAugment (Cubuk et al., 2019).
We use RandomAugment because it is simple to implement, requires no expensive search
[Table]
and achieves similar performance compared to UDA with AutoAugment.
More details of RandomAugment can be found in Appendix C.
[Table]
Main results.
In Table 1, we present our main results before and after ﬁnetuning the student on labeled data.
The results conﬁrm that Coaching signiﬁcantly outperforms UDA and other strong baselines in semi-supervised learning.
On CIFAR-10 and SVHN, compared to the state-of-the-art UDA, Coaching’s error rate reduction are roughly 30% and 10% .
As UDA’s accuracy is already relatively high, such error reductions are signiﬁcant.
On CIFAR-10, Coaching is also the ﬁrst approach to exceed supervised learning on the all labels by using merely 4,000 labeled examples.
Meanwhile, on ImageNet-10%, Coaching outperforms UDA by almost 5% in top-1 accuracy, going from 68 .
07% to 72 .
Even prior to ﬁnetuning on labeled data, Coaching still outperforms UDA and other baselines.
Comparing to existing state-of-the-art methods.
To the best of our knowledge, Coaching has achieved new state-of-the-art performances among the same model architectures on three datasets considered in this section.
For CIFAR-10 and SVHN, all existing better results use a larger model and more advanced regularization techniques.
For instance, Xie et al.
(2019) reports 97 .
3% with UDA (Xie et al., 2019)
, but their backbone model is PyramidNet, which has 18 × more parameters than WideResNet-28-2 and they train with Shake-Drop regularization (Yamada et al., 2018).
Similarly, for ImageNet-10%, the only better published result is 73 .
21% top-1 accuracy, achieved by MOAM- S 4 L (Zhai et al., 2019).
This accuracy is only slightly better than Coaching’s 72 .
94% , but uses a 4 × wider ResNet-50.
We believe that the enhancements in architectures, regularization techniques, and model sizes, can be applied to Coaching to further improve our results.
3. R ESULTS ON H IGH D ATA R EGIME
We have seen Coaching achieves strong performance for low data image classiﬁcation tasks.
Another aspect of these tasks is that the unlabeled data also come from the same domain as the labeled data, which is a restricted assumption.
In this section, we show that Coaching also excels in the regime where we have a large labeled dataset and an order of magnitude more unlabeled data.
In this regime, we also test the performance of our method when the unlabeled set may have out-of-domain images, i.e., the images belong to categories that do not exist in ImageNet.
Datasets.
We experiment with all labeled examples in ImageNet.
Additionally, we take unlabeled images from the entire 4 th version of OpenImages dataset (Kuznetsova et al., 2018), which hasmillion natural images.
A few samples from OpenImages can be found in Figure 2.
Unless otherwise speciﬁed, for both datasets, we use the image size of 224 × 224 .
Baselines.
Since this regime of high data has not been extensively studied, we are only aware of two relevant, strong baselines.
Our ﬁrst baseline is Billion-scale Semi-supervised Learning (Billion-scale SSL; Yalniz et al.
Billion-scale SSL uses unlabeled data from the YFCC100M dataset (Thomee et al., 2015), studies several self-training settings, with various model architectures for teachers and students.
Here, we restrict our comparison to the settings that use ResNet-50 for both the teacher and the student.
Our second baseline is UDA (Xie et al., 2019), for which the authors select unlabeled images algorithmically from the JFT dataset.
Other than these baselines, we compare Coaching to techniques that enhance supervised learning, such as DropBlock (Ghiasi et al., 2018), CutMix (Yun et al., 2019), and FixRes (Touvron et al., 2019).
Implementation details.
We implement Coaching the same as in Section 3.1, except for one part: Instead of directly training and consistently regularizing the teacher, we initialize the teacher using a pre-trained ResNet-50 (pre-trained on full ImageNet).
Then, throughout the course of the student’s learning, we only train the teacher to minimize the student’s cross entropy loss.
We do not use additional supervised loss for the teacher because because once the teacher is pre-trained, adding another loss to the teacher has minimal effect.
We do not consistently regularize the teacher because Xie et al.
(2019) has found that consistency regularization requires in-domain data, while we do not ﬁlter our unlabeled images from OpenImages.
[Table]
Results.
We present our results in Table 2.
As can be seen, Coaching outperforms all relevant SSL baselines.
Speciﬁcally, for the image size of 224 , Coaching outperforms Billion-scale SSL by about 1% top-1 accuracy, even though Billion-scale SSL uses 10 times more unlabeled data.
Meanwhile, for the image size of 331 , Coaching achieves the top-1 accuracy of 79 .
86% , comfortably outperforming the top-1 accuracy of 79 .
04% by UDA.
This improvement is particularly signiﬁcant, since Coaching simply uses all data from OpenImages, while UDA has to select and balance the class distribution of their unlabeled data using a pre-trained teacher.
This difference suggests that the teacher in Coaching can give helpful pseudo labels to the student, even on potentially out-of-distribution data.
It is worth mentioning that Coaching also outperforms the strong supervised baselines of DropBlock and FixRes, and is on par with FixRes+CutMix.
However, DropBlock and CutMix are both regularization techniques orthogonal to Coaching.
Similar to consistency regularization in Section 3.1, these techniques can be incorporated into the teacher in Coaching to improve performance.
Comparing to state-of-the-art SSL results.
Yalniz et al.
(2019) reports the top-1 accuracy of 81 . 2% for a ResNet-50 student.
However, they need to pre-train a much bigger network ResNext-101-32x teacher (829 million parameters, 32x larger than ResNet-50) on 1 billion Instagram images with weak labels (Mahajan et al., 2018).
Then, they use the pseudo-labels from this teacher to train a ResNetstudent for 2 billion steps.
The fact that they use weakly labeled data from Instagram, much bigger architecture in ResNext-101-32x48 makes their results not directly comparable to ours.
Meanwhile, without the need of a much bigger dataset and architecture as used in Yalniz et al.
Coaching achieves almost as good top-1 accuracy.
To achieve this, we iterate the process of Coaching by turning the student into the teacher after convergence.
After 17 iterations, our ﬁnal student achieves 82 .
34% top-1 accuracy on ImageNet, outperforming Yalniz et al.
(2019)’s
2% , even though we do not have the weakly labeled data from Instagram.
Insights about Coaching on OpenImages.
Figure 2 shows ﬁve images taken from OpenImages, along with their OpenImages tags and the top 5 classes predicted by a teacher trained on ImageNet.
From the ﬁgure, we can see that there are non-trivial overlapping contents between the OpenImages tags and the ImageNet top classes, such as sunglasses in the ﬁrst image.
We also see that for the images whose contents match stronger with an ImageNet class, such as the ﬁrst and the third image, the entropy of the teacher’s prediction is smaller.
As a result, when the teacher samples a pseudo label from these distribution, contents similar to an ImageNet class will receive more consistent labels, while content alien to ImageNet will have higher entropy on their labels.
We suspect this is why a teacher trained on ImageNet can teach a student via pseudo labels on OpenImages.
person, sunglasses, shorts moving van neck brace sunglasses miniskirt school bus wheel,van,
land vehicle grille car wheel golfcart convertible beach wagon squirrel macaque marmoset squirrel monkey titi fox squirrel person,tree, man curly-coated retriever black stork stone wall valley lakeside man, human body balance beam gondola confectionery toyshop paddlewheel
3. A NALYSIS
Ablation Study of Implementation Details.
To understand the contribution of each implementation detail of Coaching, we study their contributions on top of a purely supervised model.
We conduct this study on ImageNet-10% and visualize the results in Figure 3.
From the ﬁgure we see that
RandomAugment and UDA both improve the ﬁnal accuracy signiﬁcantly, respectively by 3 . 13% and 7 .
19% top-1 accuracy.
On top of UDA, Coaching delivers a smaller improvement of 4 .
32% topaccuracy.
However, since UDA’s accuracy is already high, we believe that the improvement of 4 .
32% top-1 accuracy is signiﬁcant.
Finally, ﬁnetuning only slightly improves over Coaching.
However, this extra boost is a unique advantage of Coaching: it is possible for the student in Coaching to ﬁnetune on labeled data because the student never directly learns from these labeled data.
Coaching overﬁts less than Supervised Learning.
In our Coaching framework, the student never directly learns from labeled data.
This behavior is helps the student to avoid overﬁtting, especially when labeled data is scarce.
In Figure 4, we visualize the training accuracy of Coaching and
Supervised Learning on CIFAR-10 with 4,000 labels and on ImageNet with 10% labels.
As shown, the training accuracy of both the teacher and the student of Coaching stay relatively low.
Meanwhile, the training accuracy of the supervised model eventually reaches 100% and causes overﬁtting.
[Figure] [Figure]
R ELATED W ORK
Pseudo-Label.
Pseudo-Label (Lee, 2013) is one of the simplest semi-supervised learning algorithms: First, a teacher model is trained on labeled data.
Then, the converged teacher model generates pseudo labels for unlabeled data.
These unlabeled data and their pseudo labels are combined with the labeled data to train another model, which is called the student model.
An inherent weakness of
Pseudo-Label is that once the teacher generates an incorrect pseudo label for an unlabeled datum, the student can only naively learn from this wrong label.
This phenomenon is called the conﬁrmation bias.
Arazo et al.
(2019) addressed the conﬁrmation bias by generating soft labels from the teacher and by adding noise to these labels.
However, this is a manual ﬁx from an outside model designer.
The main difference between Pseudo-Label and Coaching is that in Coaching, the teacher is trained along with the student throughout the course of training.
This allows wrong knowledge learned by the teacher to be ﬁxed in an end-to-end manner, leading to stronger performances.
Semi-supervised Learning (SSL).
Pseudo-Label belongs to a more general group of algorithms known as Semi-supervised Learning.
Unlike Pseudo-Label, typical SSL methods combine both labeled and unlabeled data to train a single model .
Hence, the objective function of SSL is typically the sum of a supervised loss and an unsupervised loss.
The supervised loss is often the cross-entropy computed on the labeled data.
Meanwhile, the unsupervised loss can be a self-supervised loss (Rasmus et al., 2015; Noroozi & Favaro, 2018;
Gidaris et al., 2018), or consistency regularization (Laine & Aila, 2017; Tarvainen & Valpola, 2017; Miyato et al., 2018; Berthelot et al., 2019; Xie et al., 2019).
Self-supervised losses typically encourage the model to develop a common sense about the images.
Meanwhile, consistency regularization enforces that the model is invariant against certain transformations of the data.
The main difference between Coaching and SSL methods is that the student in Coaching never learns directly from labeled data.
This helps the student in Coaching to avoid overﬁtting to labeled data, especially when labeled data is limited.
Meta Learning.
In Meta Learning, there is typically an outer loop that optimizes the performance of a model trained in an inner loop (Finn et al., 2017; Metz et al., 2019).
Meta Learning has been applied to perform self-training and SSL in the low data regime (Agarwal et al., 2019;
Ren et al., 2018; Boney & Ilin, 2018;
Hsu et al., 2019).
A crucial difference between Coaching and Meta
Learning is that in Coaching, the pseudo labels are chosen to improve the student, and hence there is no need for an outer loop.
We suspect this is an advantage of our method, since gradients to be very powerful for models to navigate in the parameter space.
C ONCLUSION
In this paper, we proposed the Coaching method for semi-supervised learning.
Key to Coaching is the idea that the teacher learns from the student’s loss and improves itself to generate pseudo labels in a way that helps student’s learning the most.
The learning process in Coaching consists of two main updates: updating the student based on the pseudo labeled data produced by the teacher and updating the teacher based on the student’s performance.
Experiments on standard CIFAR-10 and
SVHN show that Coaching is much better than supervised learning and consistenly better than other semi-supervised learning methods.
Coaching scales well to large problems, and successfully uses out-of-domain data to improve ImageNet classiﬁcation.
A D ERIVATION OF THE T EACHER ’ S U PDATE R ULE
In this section, we present the detailed derivation of the Teacher’s update rule in Equation 3 from Section 2. Mathematical Notations and Conventions. Since we will work with the chain rule, we use the standard mathematical notations.
3 Speciﬁcally, for a differentiable function f : R m → R n , and for a vector x ∈ R m , we use the notation ∂f∂x ∈ R n × m to denote the Jacobian matrix of f , whose dimension is n × m .
Additionally, when we mention the Jacobian of a function f at multiple points such as x 1 and x 2 , we will use the notations of ∂f∂x (cid:12)(cid:12)(cid:12) x
= x 1 and ∂f∂x (cid:12)(cid:12)(cid:12) x = x 2 .
Furthermore, by mathematical conventions, a vector v ∈ R n is treated as a column matrix – that is, a matrix of size n × 1 .
For this reason, the gradient vector of a multi-variable real-valued function is actually the transpose of of its Jacobian matrix. Finally, all multiplications in this section are standard matrix multiplications. If an operand is a vector, then it as discussed in the previous paragraph, the operand is treated as a column matrix. Dimension Annotations.
Understanding that these notations and conventions might cause confusions, in the derivation below, we annotate the dimensions of the computed quantities to ensure that there is no confusion caused to our readers. To this end, we respectively use | S | and | T | to denote the dimensions of the parameters θ S , θ T . That is, θ S ∈ R | S |× 1 and θ T ∈ R | T |× 1 . We now present the derivation.
We need to compute:
[Equation]
To simplify our notation, let us deﬁne
[Equation]
Then, by the chain rule
[Equation]
we have
[Equation]
The ﬁrst factor in Equation 7 can be simply computed via back-propagation.
We now focus on the second term.
We have
[Equation]
∂θ
[Equation]
Note that in Equation 8 above, the Jacobian of (cid:96) ( x unl , ˆ y unl ; θ S ) , which has dimension 1 × | S | , needs to be transposed to match the dimension of θ ( t ) S , which, as we discussed above, conventionally has dimension | S | × 1 .
Now, since θ ( t ) S in Equation 8 does not depend on θ T , we can leave it out of subsequent derivations.
Also, to simplify notations, let us deﬁne the gradient
[Equation]
Then, Equation 8 becomes
[Equation]
Since g ( t ) S (ˆ y unl ) has no dependency on on θ T , except for via ˆ y unl , we can apply the REINFORCE equation (Williams, 1992) to achieve
[Equation]
Here, the last equality in Equation 11 is is due to the deﬁnition of the cross entropy loss, which is the negative of the log-prob term in the previous line. Now [Equation]
we can substitute Equation 11 into Equation 7 to obtain [Equation] (cid:124) (cid:123)(cid:122) (cid:125) 1 ×| S | · E [Equation]
y
[Equation]
P ( ·| x
[Equation]
θ T ) (cid:104) g
( t
[Equation]
S (
[Equation]
unl ) (cid:124) (cid:123)(cid:122) (cid:125) | S |×· ∂(cid:96) ( x unl , ˆ y unl
[Equation]
θ T )
∂θ T (cid:124) (cid:123)(cid:122) (cid:125) 1 ×| T | (cid:105) =
η
[Equation]
E
[Equation]
y unl
[Equation]
P
[Equation]
·| x unl
[Equation]
θ
[Equation]
)  (cid:16) ∂(cid:96) ( x
[Equation]
[Equation]
∂θ S (cid:12)(cid:12)(cid:12)(cid:12) θ S =
[Equation]
θ ( t
[Equation]
1) S (cid:124) (cid:123)(cid:122) (cid:125) 1 ×| S | · g ( t ) S (ˆ y unl )
(cid:124) (cid:123)(cid:122) (cid:125) | S |×(cid:17) · ∂(cid:96) ( x unl , ˆ y unl ; θ T )
∂θ T (cid:124) (cid:123)(cid:122) (cid:125) 1 ×| T | 
 (12)
Here, we could factorize the term ∂(cid:96)
( x lab ,y lab ; θ S )
∂θ S (cid:12)(cid:12)(cid:12) θ S
=
¯ θ ( t +1) S into the expectation because it does not depend on θ T .
Finally, if we use Monte Carlo approximation for every term in Equation 12 using the sampled ˆ y unl , then we have Equation 3 from Section 2.
Note that in Section 2, we use the gradient notation, which results in the transposes.
B T RAINING S PEED
Coaching performs up to 5 forward passes and 3 backward passes.
Compared to vanilla backpropagation training, this is more 3 than times more expensive in FLOPs.
However, many computa- tions in Coaching are parallelizable.
For example, the forward pass of the student and the forward pass for the teacher on unlabeled data (the top half of Figure 1), can be run in parallel since they do not depend on each other.
Therefore, on computing hardware with sufﬁcient memory, we ﬁnd
Coaching to be between 2 and 2 .
5 times slower than standard back-propagation training.
C E XPERIMENTAL D ETAILS C. D ATASET S PLITS
We describe how we select the reduced datasets for the experiments on low data image classiﬁcation in Section 3.1.
For CIFAR-10, we download the ﬁve training data batch ﬁles from www.cs.toronto.edu/ ~kriz/cifar.html .
Then, we load all the images into a list of 50,000 images, keeping the order as downloaded.
The ﬁsrt 5,000 images ares reserved for validation.
The next 4,000 images are used as labeled data.
For SVHN, we download the data from the mat ﬁles on ufldl.stanford.edu/ housenumbers/ , and follow the same procedure as with CIFAR-10.
We note that this selection process leads to a slight imbalance in the class distribution for both CIFAR-10 and SVHN, but the settings are the same for all of our experiments.
For ImageNet, we follow the procedure in github.com/tensorflow/models/blob/ master/research/inception/inception/data/download_and_preprocess_ imagenet.sh .
This results in 1,024 training TFRecord shards of approximately the same size.
The order of the images in these shards are deterministic.
For ImageNet-10%, we use the ﬁrstshards; for ImageNet-20%, we use the ﬁrst 204 shards; and so on.
The last 20 shards, corresponding to roughly 25,000 images, are reserved for hyper-parameters tuning.
C. R ANDOM A UGMENT : A DATA AUGMENTATION POLICY
We develop a data augmentation policy that achieves similarly high performance with AutoAugment (Cubuk et al., 2019) in a few cases that we consider, but which does not require learning a controller to generate policies.
We names our policy RandomAugment.
Our goal when developing
RandomAugment is not to outperform AutoAugment, which is why we do not conduct extensive experiments with RandomAugment.
Instead, we simply want to avoid indirectly using labeled data for our experiments, especially for the low data regime experiments in Section 3.1.
Each policy of RandomAugment consists of two operations that applied sequentially on an image.
Each operation applies a uniformly sampled transformation with probability 0 .
5 , and with a level uniformly chosen between 1 and 10 .
For a more comprehensive discussion of the probability and the level of a transformation, we refer readers to the AutoAugment paper (Cubuk et al., 2019). [Figure] [Table] [Figure]
[Table]
We manually design the set of transformations for each of our datasets: CIFAR-10, ImageNet, and
SVHN.
The set of transformations for CIFAR-10 and for ImageNet are the same, and are slightly different from the set of transformation for SVHN.
This is because the numbers in the SVHN have a different requirement for invariant.
For instance, numbers should not be invariant against rotations like 6 and 9, and should not be invariant against horizontal translation like 3 and 8.
Table 3 presents the transformation for our dataset.
In addition to these operations, we only allow RandomAugment to select the three transformations AutoContrast, Brightness, and Invert in the ﬁrst augmenting transformation.
This is to avoid a few degenerating cases.
For instance, when Brightness is applied twice on an image, both times with small levels, the image will become almost black.
In our experiments, RandomAugment’s performance is not far behind compared to AutoAugment.
For example, on full ImageNet with ResNet-50, RandomAugment achives 77 .
98% top-1 accuracy, which is close to the top-1 accuracy of 77 .
6% reported by Cubuk et al.
C. H YPER - PARAMETERS
To tune hyper-parameters, we follow Oliver et al.
and allow each method to have 128 trials of hyper-parameters.
When we tune, we let each model train for up to 50,000 steps.
The optimal hyper-parameters are then used to run experiments that last for much more steps, as we report below.
In our experiments with Coaching, training for more steps typically leads to stronger results.
We stop at 1 million steps for CIFAR-10 and SVHN, and at 0.5 million steps for ImageNet simply because otherwise, these experiments will take too long.
Meanwhile, in our experiments with purely supervised learning, Pseudo-Labels, and UDA, training for more steps overﬁts the models, and we have to employ early stopping.
We report the hyper-parameters for our baselines and for Coaching in Section 3.1.
For the highresource experiments in Section 3.2, we use the same hyper-parameters, because tuning them is too expensive.
Our hyper-parameters can be found in Table 4, 5, 6.
We note that our settings for UDA is different from originally reported by Xie et al.
In their work, Xie et al.
(2019) use a much larger batch size for their UDA objective.
In our implementation of UDA, we keep these batch sizes the same.
This leads to a much easier implementation of data parallelism in our framework, TensorFlow (Abadi et al., 2016) running on TPU big pods.
To compensate for the difference, we train all UDA baselines for much longer than Xie et al.
During the training process, we also mask out the supervised examples with high conﬁdence.
Effectively, our UDA model receives roughly the same amount of training with labeled examples and unlabeled examples as the models in Xie et al.
We have also veriﬁed that on ImageNet-10% with the augmentation policy from AutoAugment (Cubuk et al., 2019), our UDA implementation achives 68 .
77% top-1 accuracy, which is similar to 68 .
66% that Xie et al.
(2019) reported.
[Figure] [Table]
[Table]