Semi-supervised Vision Transformers at Scale
Abstract
We study semi-supervised learning (SSL) for vision transformers (ViT), an underexplored topic despite the wide adoption of the ViT architectures to different tasks.
To tackle this problem, we use a SSL pipeline, consisting of first un/self-supervised pre-training , followed by supervised fine-tuning , and finally semi-supervised finetuning .
At the semi-supervised fine-tuning stage, we adopt an exponential moving average (EMA)-Teacher framework instead of the popular FixMatch, since the former is more stable and delivers higher accuracy for semi-supervised vision transformers.
In addition, we propose a probabilistic pseudo mixup mechanism to interpolate unlabeled samples and their pseudo labels for improved regularization, which is important for training ViTs with weak inductive bias.
Our proposed method, dubbed Semi-ViT , achieves comparable or better performance than the
CNN counterparts in the semi-supervised classification setting.
Semi-ViT also enjoys the scalability benefits of ViTs that can be readily scaled up to large-size models with increasing accuracies.
For example, Semi-ViT-Huge achieves an impressive 80% top-1 accuracy on ImageNet using only 1% labels, which is comparable with Inception-v4 using 100% ImageNet labels.
The code is available at https://github.com/amazon-research/semi-vit .
Introduction
[Figure]
ImageNet 10% Labels
Semi-ViT
SimCLRv2PAWSFixMatchMPL (b)        1% labels 10% labels 100% labels
I m a g e N e t t o pa cc u r a c y
S e
m i - V i T - H
R e s N e tI n c e p
t
i o n - v
C o n v N e
X t - L
S w i n - B
E ff i c
i e n t
N e t - L
S e
m i - V i T - H (c)
In the past few years, Vision Transformers (ViT) [18], which adapt the transformer architectures [64] to the visual domain, have achieved remarkable progresses in supervised learning [63, 44, 73], un/self-supervised learning [16, 12, 25], and many other computer vision tasks [11, 19, 1, 58] (with architecture modifications).
However, ViTs have yet to show the same advantage in semi-supervised learning (SSL), where only a small subset of the training data is labeled, a problem in the middle between supervised and un/self-supervised learning.
Although several recent methods in SSL have significantly advanced the field [39, 62, 7, 55, 70, 10, 53], the transfer of these methods from
Convolutional Neural Networks (CNN) to ViT architectures has yet to show much promise.
For example, as discussed in [68], the direct application of FixMatch [55], one of the most popular SSL methods, to ViT leads to an inferior performance (about 10 points worse) than when used with a CNN architecture.
The challenge could be potentially caused by the fact that ViTs are known to require more data for training and to have a weaker inductive bias than CNNs [18].
However, in this paper we show that semi-supervised ViTs can outperform the CNN counterparts when trained properly, suggesting promising potential to advance SSL beyond CNN architectures.
To achieve such success, we use the following SSL pipeline: 1) un/self-supervised pre-training on all data (both labeled and unlabeled), followed by 2) supervised fine-tuning only on labeled data, and finally 3) semi-supervised fine-tuning on all data.
This pipeline is stable and helps reduce the sensitivity of hyperparameter tuning when training ViTs for SSL in our experiments.
At the final stage of semi-supervised fine-tuning , we adopt the EMA-Teacher framework [62, 10], an improved version over the popular FixMatch [55].
Unlike FixMatch that often fails to converge when training semi-supervised ViT, EMA-Teacher shows more stable training behaviors and better performance.
In addition, we propose probabilistic pseudo mixup for the pseudo-labeling based SSL methods, which interpolates the unlabeled samples coupled with pseudo labels for enhanced regularization.
In the standard mixup [75] the mixup ratio is randomly sampled from a Beta distribution.
In contrast, in the probabilistic pseudo mixup the ratio depends on the respective confidences of two mixed-up samples, such that the sample with higher confidence will weigh more in the final interpolated sample.
This new data augmentation technique brings non-negligible gains since ViT has weak inductive bias, especially for scenarios where the training is more difficult, e.g. , without un/self-supervised pre-training or on data regimes with very few labeled samples ( e.g. , 1% labels).
We call our method
Semi-ViT .
Notice that Semi-ViT is built on exactly the same design of ViTs ( i.e. , there are neither additional parameters nor architectural changes).
Semi-ViT achieves promising results from different aspects (Figure 1).
For the first time, we show that pure ViTs can reach comparable or better accuracy than CNNs on SSL 1 . 2)
Semi-ViT can be readily scaled up under the SSL setting.
This is illustrated in Figure 1 (a) and (b) on ViT architectures at different scales, ranging from ViT-Small to ViT-Huge, and Semi-ViT outperforms the prior art such as SimCLRv2 [15].
3) Semi-ViT has shown the potential for a substantial reduction of labeling cost.
For example, as seen in Figure 1 (c), Semi-ViT-Huge with 1% (10%) ImageNet labels achieves a comparable performance of a fully-supervised Inception-v4 [59] (ConvNeXt-L [45]).
This implies a 100 × ( 10 × ) reduction in human annotation cost.
4) Semi-ViT achieves the state-of-the-art SSL results on ImageNet, e.g. , 80.0% (84.3%) top-1 accuracy with only 1% (10%) labels.
In addition, the substantial boost in performance by Semi-ViT is not isolated on ImageNet: we find an increase of 13%-21% (7%-10%) top-1 accuracy with 1% (10%) labels over the supervised fine-tuning baselines, for other datasets including Food-101 [9], iNaturalist [30] and GoogleLandmark [52].
Semi-supervised Vision Transformers 2.1 Pipeline
Some pipelines for semi-supervised learning exist in the literature.
For example: 1) The model is directly trained from scratch using SSL techniques, e.g. , FixMatch [55]; 2)
The model is un/selfsupervised pretrained first and finetuned on the labeled data later [26, 14, 22]; 3)
The model is self-supervised pretrained first and then semi-supervised finetuned on both labeled and unlabeled data [10].
In this paper, we instead adopt the following pipeline: at first, optional self-supervised pre
-training on all the data without using any labels; next, standard supervised fine-tuning on the available labeled data; and finally, semi-supervised fine-tuning on both labeled and unlabeled data.
This procedure is similar to [15], with the difference that they use knowledge distillation [29] in their final stage.
We find that this training pipeline is stable to train semi-supervised vision transformers and achieves promising results, with possibly less hyperparameter tuning.
[Figure]
logit
[Figure]
pseudo
[Figure]
C
[Figure]
l o
[Figure]
2.2 EMA-Teacher Framework
FixMatch [55] emerged as a popular SSL method in the past few years.
As discussed in [10], it can be interpreted as a student-teacher framework, where the student and teacher models are identical, as seen in Figure 2 (a).
However, FixMatch has unexpected behaviors, especially when the model consists of batch normalization (BN) [35].
Although ViT uses Layer Normalization (LN) [4] instead of BN as normalization, we still found that FixMatch with ViT underperforms the CNN counterparts and often does not converge.
This phenomenon was also observed in [68].
A potential reason to this is that the student and the teacher models are identical in FixMatch, which could easily lead to model collapse [26, 22].
This instability of the identical student-teacher framework has also been observed in other areas, e.g. semi-supervised speech recognition [42, 48, 28].
As suggested in [10], the EMA-Teacher (shown in Figure 2 (b)) is an improved version of FixMatch, thus we adopt it for our Semi-ViT.
In the EMA-Teacher framework, the teacher parameters θ ′ are updated by exponential moving average (EMA) from the student parameters θ ,
[Equation]
where the momentum decay m is a number close to 1, e.g. , 0.9999.
The student parameters are updated by standard learning optimization, e.g. , SGD or AdamW [46].
The other components are exactly the same as FixMatch, as seen in Figure 2.
This temporal weight averaging can stabilize the training trajectories [3, 36] and avoids the model collapse issue [26, 22].
Our experiments also show this EMA-Teacher framework has better results and more stable training behaviors than FixMatch when training Semi-ViT.
2.3 Semi-supervised Learning Formulation
In the EMA-Teacher framework, there are both labeled and unlabeled samples in a minibatch during training.
The loss on the labeled samples { ( x li , y li ) } N l i =1 is the standard cross-entropy loss, L l = 1 N l (cid:80) N l i =1 CE ( x li , y li ) .
For an unlabeled sample x u ∈ { x ui } N u i =1 , a weak and a strong augmentation are applied to it, generating x u,w and x u,s , respectively.
The weak augmented x u,w is forwarded through the teacher network, and outputs the probabilities over classes, p = f
( x u,w ; θ ′ ) .
Then the pseudo label is produced by ˆ y = arg max c p c with its associated confidence o = max p c .
The pseudo label with confidence higher than a confidence threshold τ is then used to supervise the learning of the student on the strong augmented sample x u,s ,
[Equation]
where [ · ] is the indicator function.
The overall loss is L =
L l + µ L u , where µ is the trade-off weight.
Note that only the pseudo labels with confidence higher than a threshold contribute to the final loss; 𝑥 !
𝑥 " 𝑥 # 𝑥 $ 𝑥 % 𝑥 & 𝑥 ’ 𝑥 ( 𝑥 # 𝑥
% 𝑥 ( 𝑥 !
𝑥 $ 𝑥 ’ 𝑥 " 𝑥 & 𝑥" !
𝑥" " 𝑥" # 𝑥" $ 𝑥" % 𝑥" & 𝑥" ’ 𝑥" ( ℬ : ℬ$ : ℬ% : 𝑥" !
= 𝜆𝑥 !
+ 1 −𝜆 𝑥 " ,𝜆~𝐵𝑒𝑡𝑎(𝛼,𝛼)
[Figure]
Pseudo Mixup
[Figure]
the others are instead not used.
The philosophy behind this filtering is that the pseudo labels with low confidences are noisier and could hijack the SSL training.
Probabilistic Pseudo Mixup 3.1 Mixup
Mixup [75] performs convex combinations of pairs of samples and their labels,
[Equation]
where the mixup ratio λ ∼ Beta ( α, α ) ∈ [0 , 1] , for α ∈ (0 , ∞ ) .
The samples are mixed-up usually in a single minibatch during training.
Given a minibatch B and its shuffled version ¯ B , the mixed-up minibatch is ˜ B = λ B + (1 − λ ) ¯ B , where λ could be either batch-wise or element-wise.
Due to the nature of weak inductive bias, ViT is more data hungry than CNN, thus effective data augmentation, e.g. , mixup, is critical for training fully-supervised ViT [18, 63, 44, 73].
This also applies to SemiViT since it inherits the nature of weak inductive bias from ViT. Although it is standard to use mixup in supervised learning, how to employ it under pseudo-labeling based SSL framework, e.g. ,
EMA-Teacher, is still unclear yet, and we are going to discuss it next.
3.2 Pseudo Mixup
Under the pseudo-labeling based SSL framework [40, 55, 53, 10], given a unlabeled sample and its pseudo label ( x u , ˆ y ) , only when its confidence o is not smaller than the confidence threshold τ , it will contribute to loss L u , as seen in (2).
According to their confidence scores, the unlabeled minibatch B u can be grouped into a clean subset ˆ B u = { ( x ui , ˆ y i )
|
o
i ≥ τ } and a noisy subset ˙ B u = B u − ˆ B u .
One straightforward solution is to apply mixup on the full unlabeled minibatch B u , with no differentiation between clean and noisy samples, denoted as pseudo mixup , as show in Figure 3 (a).
After the pseudo mixup, still only the samples in ˆ B u contribute to the loss, and samples in ˙ B u are abandoned.
In this way, the mixup operation is more than just a data augmentation.
In fact, a sample in ˙ B u will also contribute to the final loss if it is mixed-up with a sample in ˆ B u .
As a result, it could involve a substantial number of noisy samples into the loss calculation due to the randomness, which, however, is against the philosophy of pseudo-labeling.
Since only the clean subset ˆ B u contributes to the final loss, another choice is to use mixup only on ˆ B u , denoted as pseudo mixup+ , as shown in Figure 3 (b).
In this way, no sample in the noisy subset ˙ B u will affect the training.
3.3 Probabilistic Pseudo Mixup
Although the samples in ˙ B u are noisy, they still carry some useful information for the model to learn.
The pseudo mixup above can somehow leverage those information by blending the noisy and clean pseudo samples together.
However, the problem is the mixup ratio is randomly generated from a Beta distribution, which does not depend on the confidence of each sample.
This is not ideal.
For example, when two samples are mixed-up, the sample with higher confidence should have higher mixup ratio, such that it can weigh more in the final loss.
Motivated by this intuition, we propose probabilistic pseudo mixup (Figure 3 (c)), where the mixup ratio λ relfects the sample confidences,
[Equation]
[Table] [Table]
Also, the confidence score of x ui is updated after mixup operation as
[Equation]
because the confidence score should align with the majority of the image content.
And the final clean subset ˜ B u = { (˜ x ui , ˜ y ui ) | o ∗ i ≥ τ } will contribute to the final loss.
Using this probabilistic pseudo mixup can enhance regularization
[Table]
leverage information from all samples, even the noisy ones
[Table]
and not violate the philosophy of pseudo labeling at the same time.
It can effectively alleviate the issue of weak inductive bias of Semi-ViT and bring substantial gains, as will be shown in our experiments.
Experiments
We evaluate Semi-ViT mainly on ImageNet, which consists of ∼ 1.28M training and 50K validation images.
We sample 10%/1% labels from ImageNet training set for semi-supervised evaluation.
We study both scenarios: with and without self-supervised pre-training.
Without self-pretraining, we only evaluate on 10% labels, since learning from scratch on 1% labels is very difficult.
When self-pretrained, MAE [25] is mainly used, and we directly use their pretrained models.
All learning is optimized with AdamW [46], using cosine learning rate schedule, with a weight decay of 0.05.
The default momentum decay m of (1) is 0.9999.
In a minibatch, N u = 5 N l , and the loss trade-off µ = 5 .
The mixup is the combination of mixup [75] and Cutmix [74] as the implementation of [69].
More details can be found in the appendix.
4.1 Semi-ViT Results
When the model is self-pretrained by MAE [25], we first evaluate the fine-tuning performances of MAE on the labeled data only, as the common practice in self/un-supervised learning literature [26, 14, 22], with results shown Table 1.
This already leads to strong semi-supervised baselines, e.g. , 81.4 top-1 accuracy for ViT-Huge on 10% labels, indicating MAE is a strong self-supervised learning technique.
However, Semi-ViT has additional significant improvements over the strong baselines for all models, e.g. , 8.5-13.6 points for 1% labels and 2.9-6.0 points for 10% labels.
The fine-tuning results on 100% data are provided as upper-bounds for our Semi-ViT, and their gaps to
Semi-ViT are small, e.g. , 4.0/2.7/2.6 points for ViT-Base/Large/Huge on 10% labels.
An interesting observation is that the larger model is more effective for smaller number of labels, which is consistent with the observations in [15].
For example, the fine-tuning gaps between 1% and 100% labels are 26.3/18.9/15.4 points for ViT-Base/Large/Huge, which are decreasing.
The observation on Semi-ViT results is similar, e.g. , 12.7/8.7/6.9 points to their upper-bounds on 1% labels.
These results have shown that vision transformers can also perform very well in semi-supervised learning, as well as supervised learning and un/self-supervised learning.
[Table]
[Table]
4.2 Ablation Studies
We ablate different factors of Semi-ViT in this section.
FixMatch v.s. EMA-Teacher is compared in Table 2.
These experiments do not use the pseudo mixup techniques of Section 3 yet.
When the model is not self-pretrained, the training of FixMatch is unstable and often failed.
When the model is self-pretrained, FixMatch training becomes stable, and start to achieve reasonable results, e.g. , 74.8 for ViT-Base on 10% labels, which is already better than the prior art on ResNet-50, e.g. , 73.9 of MPL [53] and 74.0 of EMAN [10].
But it is only 1.
points higher than the fine-tuning baseline of Table 1, indicating FixMatch is not an effective SSL framework for ViT.
But EMA-Teacher achieves much better results, 3.3 points improvement over
FixMatch when self-pretrained.
Even without self-pretraining, EMA-Teacher can still achieve decent numbers, but FixMatch is failed.
Probabilistic Pseudo Mixup Different mixup variations on unlabeled data are compared in Table 3.
Note that the standard mixup with implementation of [69] is used to the labeled data as usual.
The EMA-Teacher does not use any mixup mechanism on the unlabeled data, serving as baselines here.
When pseudo mixup of Figure 3 (a) is applied on the unlabeled data, the performances usually have some substantial gains over the EMA-Teacher baselines, especially for the scenarios that the training is more difficult, e.g. , without self-pretraining or on 1% labels.
This shows the importance to use mixup on the unlabeled data for improved regularization.
However, as discussed Section 3.2, pseudo mixup could involve many noisy samples into training.
On the other hand, pseudo mixup+ of
Figure 3 (b) can increase over pseudo mixup constantly, by about 0.5 points, showing that removing those noisy samples does help.
In addition, probabilistic pseudo mixup of Figure 3 (c) can further improve over pseudo mixup+ by 1-2 points in all cases.
These results have implied that those noisy samples do carry some useful information for SSL training, but their weights should be suppressed especially when their confidences are low.
This data augmentation technique also effectively alleviate the training difficulty of semi-supervised vision transformers with weak inductive bias.
Effect of Confidence Threshold We ablate the effect of confidence threshold τ of (2) in Table 4.
We can find Semi-ViT is quite robust to the low confidence thresholds.
One possible reason is that Semi-ViT uses probabilistic pseudo mixup .
When τ is low, the low-confidence samples will not hijack the training since their contributions depend on their confidence scores.
These imply that the hyperparameter of τ can possibly be removed ( τ = 0 ) in Semi-ViT.
But EMA-Teacher has a drop of 2-3 points when the confidence threshold is removed.
The final choices of τ for different Semi-ViT models are shown in Table 13 in the appendix.
[Table]
[Table]
Effect of Momentum Decay Table 5 shows the effect of momentum decay m in the EMA teacher.
Note that when m = 0 , the frameworks is reduced to FixMatch.
We can find Semi-ViT is robust to m .
For 10% labels, Semi-ViT has very minor changes when m decreases to 0, but EMA-Teacher has a drop of 3.3 points.
For 1% labels, the training is more challenging
[Table]
and the choice of m becomes more important.
In this case, Semi-ViT has a drop of 2.3 points from m = 0 .
999 to m = 0 , but
EMA-Teacher could fail when m decreases to 0.
The robustness of Semi-ViT to momentum decay m is also attributed to the use of probabilistic pseudo mixup .
Effect of Self-pretraining The self-pretraining of MAE [25] has a substantial boost in performances, as seen in Table 3.
For ViT-Base, MAE helps to improve by 6.2 and 9.2 points for EMATeacher with and without probabilistic pseudo mixup , respectively.
In addition, it helps to train the models in more challenging scenarios, e.g. , 1% labels.
Without self-pretraining, the training fails to deliver good results on 1% labels.
Notice that, even without pre-training, our Semi-ViT (“ProbPseudo
Mixup” in Table 3) also achieves slightly better performance than the CNN counterparts: 70.9 of
Semi-ViT-Small v.s. 67.1 of FixMatch-ResNet50 or 69.2 [55] of EMAN-ResNet50 [10] when trained from scratch for 100 epochs.
Effect of Supervised Fine-tuning is ablated in Table 6 by varying the number of supervised fine-tuning epochs.
The rows of “Supervised-ViT” are the numbers of supervised fine-tuning , where the latter semi-supervised fine-tuning begins.
Semi-ViT is still robust to the length of supervised fine-tuning , where the accuracy decrease is 0.4 (1.3) for 10% (1%) labels when supervised fine-tuning is removed.
However, the performance of EMA-Teacher decreases 1.7 (4.2) points.
These show sufficient supervised fine-tuning does stabilize the training procedure, especially for less robust framework, e.g. , EMA-Teacher.
However, notice that supervised fine-tuning could sometimes hurt the performance if it is not sufficient ( e.g. , epochs=10) for EMA-Teacher.
Other Self-pretraining Techniques Beyond MAE, we also experiment on other self-pretraining technique, including MoCo-v3 [16] and DINO [12], in Table 7.
By comparing the fine-tuning results,
DINO is close to MoCo-v3 for ViT-Base but much better for ViT-Small, and both of them are better than MAE for ViT-Base, suggesting that DINO could be a better self-pretraining technique for smaller scales of ViT models.
On top of the strong fine-tuning baselines, the semi-supervised fine-tuning , using EMA-Teacher, still has nontrivial improvements for both DINO and MoCo-v3, e.g. , 5.8 (2.1) points on 1% (10%) labels for DINO-ViT-Base.
In addition, the probabilistic pseudo mixup can further improve over the EMA-Teacher, independent of the self-pretraining algorithms.
And the final
Semi-ViT-Base of DINO is 2.1 (0.5) points better than that of MAE on 1% (10%) labels.
Other Network Architectures Although in this paper we mainly focus on ViT architectures, the proposed probabilistic pseudo mixup is not limited to them.
We also try it for CNN architectures, e.g. ,
ResNet [27].
However, we find the direct use of the standard mixup does not improve fully-supervised
ResNet performances, so will the probabilistic pseudo mixup for its SSL setting.
Instead, we evaluate it on the recently proposed ConvNeXt [45], which uses mixup for improved results.
Since the goal is
[Table] [Table]
not to fully reproduce the results of [45], all models are trained only for 100 epochs, including the supervised upper-bounds.
The results in Table 8 demonstrate that probabilistic pseudo mixup is not limited to ViT
[Table]
but also to CNN architectures, e
[Table]
g. , with improvements of 3-4 points [Table] suggesting it can be well generalized
[Table]
4.3 Comparison with the State-of-the-Art
Semi-ViTs are compared with the state-of-the-art semi-supervised learning algorithms in Table 9.
When the model capacity is close, our Semi-ViT has shown much better results than the prior art, e.g. ,
MPL-RN-50 [53] v.s. Semi-ViT-Small, CowMix-RN152 [20] v.s. Semi-ViT-Base, S4L-RN50-4 × [8] v.s. Semi-ViT-Large and SimCLRv2+KD-RN152-3 × -SK [15] v.s. Semi-ViT-Huge.
The only transformer based SSL method is SemiFormer [68], but it requires to use CNN as the teacher model and blend convolution and transformer modules together for good performances.
However, our
Semi-ViT is pure ViT based, without any additional parameters and architecture changes, and the
Semi-ViT-Small model is already better than SemiFormer (77.1 v.s. 75.5).
These comparisons support that Semi-ViT does advance the state-of-the-art of semi-supervised learning.
Scalability is an advantage of ViT, and we compare the scalability of Semi-ViT with previous works in Figure 1 (a) and (b).
The comparison has shown that Semi-ViT can achieve better trade-off between model capacity and accuracy and can be scaled up more effectively than the prior art, SimCLRv2 [15].
For example, SimCLRv2 and PAWS [2] scale up the model usually in terms of network depth and width, and they seem to saturate when the model is of medium size, e.g. , around 300M parameters, but our Semi-ViT continues to improve steadily beyond that point.
Semi-ViT is also compared with the supervised state-of-the-art in Table 10.
Our Semi-ViT-Huge is comparable with Inception-v4 [59], but with 100 × annotation cost reduction, and comparable with ConvNeXt-L [45] (better than Swin-B [44]), but with 10 × annotation cost reduction.
These comparisons imply that Semi-ViT has great potential for labeling cost reduction.
4.4 Other Datasets
The generalization of Semi-ViT is evaluated on datasets including Food-101 [9], iNaturalist [30] and
GoogleLandmark [52].
Since these datasets are beyond ImageNet, we assume that the ImageNet dataset is available and the model is already supervised pretrained on ImageNet, and then the model is finetuned to different target datasets with a few labels.
The results are shown in Table 11.
On [Table] [Table] these dataset, our Semi-ViT can improve over the fine-tuning baselines by 13-21 (7-10) points on 1% (10%) labels.
Note that on Food-101, Semi-ViT on 1% (10%) labels is close to fine-tuning baseline on 10% (100%) labels, i
[Table]
e. , 82.1 v.s. 84.5 (91.3 v.s. 93.1), indicating that using Semi-ViT can help to save annotation costs by about 10 times on this dataset.
Related Work
Semi-supervised learning has a long history of research [78, 13].
The recent works can be roughly clustered into two groups, consistency-based [39, 62, 50, 70, 66] and pseudo-labeling based [40, 55, 53, 10].
Consistency-based methods usually add some noise to the input or model [Table] and then enforce their feature or probability outputs to be consistent.
For example, to construct two outputs for later consistency regularization, Π -model [39] adds noise to the model weights using dropout [57], Mean-teacher [62] builds a teacher model by EMA updated from the student model, and UDA [70] applies a weak and a strong data augmentation to the input.
On the other hand, the idea of pseudo-labeling or self-training can be traced back to [34, 49], which uses model predictions as hard pseudo labels to guide the learning on unlabeled data.
This idea becomes popular in SSL recently [40, 55, 53, 10, 71], and some theoretical explanations are available [76, 24].
In the offline pseudo labeling [40, 71], the model used to generate pseudo labels is usually frozen or updated once in a while during training, e.g. , at the end of every training epoch, but for online pseudo-labeling [55, 10] the teacher model is updated continuously along with the student.
Beyond classification, pseudo-labeling has also achieved promising progresses in more challenging tasks, e.g. , object detection [56, 43, 67].
Our Semi-ViT also falls into the category of online pseudo-labeling.
[Table] Mixup [75] is an effective data augmentation technique, which interpolates the input samples and their labels linearly and performs vicinal risk minimization.
It has been successfully used in image classification and some other domains, e.g. , generative adversarial networks [47], sentence classification [23], etc.
Other variants have also been developed, e.g. , Manifold Mixup [65] that mixes up in the feature space or CutMix [74] which cuts a patch from one image and pastes it into another one.
Mixup has also been successfully adopted in self-supervised learning [37, 41] and semi-supervised learning [7, 66, 6].
Although [7, 66, 6] also used mixup for SSL, they have differences with our probabilistic pseudo mixup : 1) they are consistency-based SSL framework, but ours is pseudo-labeling based; 2) their mixup ratio is random sampled, but ours depends on the pseudo label confidence; 3) they have only shown successes on small CNN architectures and small datasets, e.g. , CIFAR [38] and
SVHN [51], but our successes are built on various scales of transformer architectures and large-scale datasets, e.g. , ImageNet [54], INaturalist [30], GoogleLandmark [52], etc.
Conclusion
In this paper, we propose Semi-ViT for vision transformers based semi-supervised learning.
This is the first time that pure vision transformers can achieve promising results on semi-supervised learning and even surpass the previous best CNN based counterparts by a large margin.
In addition,
Semi-ViT inherits the scalable benefits from ViT, and the larger model leads to smaller gap to the fully supervised upper-bounds.
This has shown to be a promising direction for semi-supervised learning.
And the advantages of Semi-ViT can be well generalized to other datasets, suggesting potentially broader impacts.
We hope these promising results could encourage more efforts in semi-supervised vision transformers.
Limitations Our paper only considers the standard semi-supervised classification settings where the full dataset, e.g. , ImageNet, is downsampled to smaller scales, and not the advanced settings where full ImageNet is used as labeled and additional data, e.g. , ImageNet-21K, is used as unlabeled.
And we have only evaluated on classification task.
It is unclear whether the same conclusion can hold to that advanced classification setting and more challenging tasks, e.g. , detection or segmentation.
Potential Negative Social Impacts Semi-ViT has shown that strong models can be obtained with only a few labels, e.g. , 1%.
This increases the good AI models accessibility to anyone, which could potentially lead to inappropriate use of them.
A Implementation Details
Dataset Sampling We sample 1% (10%) images per class from the datasets we use for the semisupervised learning experiments of 1% (10%) labels.
For example, on ImageNet, the number of sampled images is 12,820 (128,118) in total, for 1% (10%) labels.
ViT Architectures We use exactly same architecture as the standard ViT [18].
For the position tokens, we use the learnable positional embedding when the model is self-pretrained, but the sinecosine version of positional embedding (non-learnable) when not self-pretrained since we find it leads to better results than the learnable one.
For all architectures, the classifier is built on top of the average pooling of the encoder output, except for ViT-Huge where we find the classifier on top of the output of the class token leads to higher performances.
Data Augmentation We use the common data augmentations of RandomResizedCrop ,
RandomHorizontalFlip ,
RandAugment(‘m9-mstd0.5-inc1’)
[17] and
RandomErasing [77] on the labeled data.
The same augmentations are also used on the unlabeled data as the strong augmentation in EMA-Teacher, whereas the weak augmentations are
RandomResizedCrop , RandomHorizontalFlip and ColorJitter(0.4) .
We do not extensively explore the space of data augmentation for weak and strong augmentations.
The center 224 × 224 crop is used at inference.
Self-supervised Pretraining We directly use the pretrained models from MAE [25], DINO [16] and MoCo-v3 [12].
The final Semi-ViT-Small is with DINO self-pretraining.
Supervised Fine-tuning Settings
The settings for the stage of supervised fine-tuning , with and without self-pretraining, are shown in Table 12.
These settings are mainly following the settings of finetuning and learning from scratch in [25], with some minor modifications.
We also use the linear learning rate scaling rule [21]: lr = base _ lr × batchsize/ 256 .
When supervised fine-tuning on 1% data, we find the performances are bad when the regularization is strong, hence we do not use mixup/cutmix, drop path and random erasing, in that case, except ViT-Small.
Semi-supervised Fine-tuning Settings
The settings for the stage of semi-supervised fine-tuning , with and without self-pretraining, are shown in Table 13.
When semi-supervised fine-tuning small models (ViT-Small/Base) on 1% data, we find the performances are bad when the regularization is strong, hence we do not use mixup/cutmix on labeled data, and drop path and random erasing on both labeled and unlabeled data, in that case.
The linear learning rate scaling rule is: lr = base _ lr × batchsize l / 256 , where batchsize l is the batch size of the labeled data.
Confidence Threshold
The confidence thresholds of Semi-ViT are shown in Table 13, even though sometimes those are not optimal according to the ablation study in Table 4.
Computing Resources We run all experiments on V100 GPUs of 32G memory.
For Semi-ViTSmall/Base/Large/Huge, we use 8/8/16/32 GPUs for training 100/100/100/50 epochs, which takes about 19/31/61/115 hours.
Random Seeds and Error Bar
Since some of the experiments are expensive to run, e.g., SemiViT-Huge.
We only test the randomness on Semi-ViT-Base models.
We sample three different 10%/1% subsets of ImageNet and repeat the experiments for three times with different random seeds.
When self-pretrained by MAE, the accuracy is 79 .
71 ± 0 .
95 ± 0 . 029 ) for 10% (1%) labels.
When not self-pretrained, the accuracy is 73 .
44 ± 0 .
065 for 10% labels.
The results have shown that
Semi-ViT is quite robust to different random seeds and different subsets of the samples.
To keep consistency, all experiments in the paper are run with the same ImageNet subset and the same random seed.
Settings on Other Datasets The model is ViT-Base on the datasets of Food-101 [9], iNaturalist [30] and GoogleLandmark [52].
The supervised fine-tuning settings are almost the same as those with self-pretraining in Table 12, with the differences: 1) we search the optimal base learning rate from { 1 e − 4 , 2 .
5 e − 4 , 5 e − 4 , 1 e − 3 } and the optimal layer-wise learning rate decay from { 0 . 65 , 0 . 75 } ;
[Table] [Table]
2) we enable mixup/cutmix, drop path and random erasing for 1% experiments (except iNaturalist), with the same values of those for 10% experiments.
The semi-supervised fine-tuning settings are almost the same as those with self-pretraining in Table 13, with the difference that we set the EMA momentum decay m = 0 .
999 instead of m = 0 .
9999 as in the ImageNet experiments, since these datasets are smaller and need faster EMA