Semi-supervised Vision Transformers at Scale
Abstract
We study semi-supervised learning (SSL) for vision transformers (ViT), an underexplored topic despite the wide adoption of the ViT architecture to different tasks.
To tackle this problem, we use a SSL pipeline, consisting of ﬁrst un/self-supervised pre-training , followed by supervised ﬁne-tuning , and ﬁnally semi-supervised ﬁnetuning .
At the semi-supervised ﬁne-tuning stage, we adopt an exponential moving average (EMA)-Teacher framework instead of the popular FixMatch, since the former is more stable and delivers higher accuracy for semi-supervised vision transformers.
In addition, we propose a probabilistic pseudo mixup mechanism to interpolate unlabeled samples and their pseudo labels for improved regularization, which is important for training ViTs with weak inductive bias.
Our proposed method, dubbed Semi-ViT , achieves comparable or better performance than the
CNN counterparts in the semi-supervised classiﬁcation setting.
Semi-ViT also enjoys the scalability beneﬁts of ViTs that can be readily scaled up to largesize models with increasing accuracy.
For example, Semi-ViT-Huge achieves an impressive 80% top-1 accuracy on ImageNet using only 1% labels, which is comparable with Inception-v4 using 100% ImageNet labels.
The code is available at https://github.com/amazon-science/semi-vit .
Introduction
[Figure]
a c c u r a cy
ImageNet 10% Labels
Semi-ViT
SimCLRv2PAWSFixMatchMPL (b)        1% labels 10% labels 100% labels
I m a g e N e t t o pa cc u r a c y
S e
m i - V i T - H
R e s N e tI n c e p
t
i o n - v
C o n v N e
X t - L
S w i n - B
E ff i c
i e n t
N e t - L
S e
m i - V i T - H (c)
In the past few years, Vision Transformers (ViT) [18], which adapt the transformer architecture [64] to the visual domain, have achieved remarkable progress in supervised learning [63, 44, 73], un/self-supervised learning [16, 12, 25], and many other computer vision tasks [11, 19, 1, 58] (with architecture modiﬁcations).
However, ViTs have yet to show the same advantage in semi-supervised learning (SSL), where only a small subset of the training data is labeled, a problem in the middle between supervised and un/self-supervised learning.
Although several recent methods in SSL have 36th Conference on Neural Information Processing Systems (NeurIPS 2022). signiﬁcantly advanced the ﬁeld [39, 62, 7, 55, 70, 10, 53], the transfer of these methods from
Convolutional Neural Networks (CNN) to ViT architectures has yet to show much promise.
For example, as discussed in [68], the direct application of FixMatch [55], one of the most popular SSL methods, to ViT leads to inferior performance (about 10 points worse) than when used with a CNN architecture.
The challenge could be potentially caused by the fact that ViTs are known to require more data for training and to have a weaker inductive bias than CNNs [18].
However, in this paper we show that semi-supervised ViTs can outperform the CNN counterparts when trained properly, suggesting promising potential to advance SSL beyond CNN architectures.
To achieve that, we use the following SSL pipeline: 1) un/self-supervised pre-training on all data (both labeled and unlabeled), followed by 2) supervised ﬁne-tuning only on labeled data, and ﬁnally 3) semi-supervised ﬁne-tuning on all data.
This pipeline is stable and helps reduce the sensitivity of hyperparameter tuning when training ViTs for SSL in our experiments.
At the ﬁnal stage of semi-supervised ﬁne-tuning , we adopt the EMA-Teacher framework [62, 10], an improved version of the popular FixMatch [55].
Unlike FixMatch, which often fails to converge when training semisupervised ViT, the EMA-Teacher shows more stable training behaviors and better performance.
In addition, we propose probabilistic pseudo mixup for pseudo-labeling based SSL methods, a method that interpolates the unlabeled samples coupled with pseudo labels for enhanced regularization.
In the standard mixup [75] the mixup ratio is randomly sampled from a Beta distribution.
In contrast, in the probabilistic pseudo mixup the ratio depends on the respective conﬁdence of two mixed-up samples, such that the sample with higher conﬁdence will weigh more in the ﬁnal interpolated sample.
This new data augmentation technique brings non-negligible gains since ViT has weak inductive bias, especially for scenarios where the training is more difﬁcult, e.g. , without un/self-supervised pre-training or on data regimes with very few labeled samples ( e.g. , 1% labels).
We call our method
Semi-ViT .
Notice that Semi-ViT is built on exactly the same design of ViTs ( i.e. , there are neither additional parameters nor architectural changes).
Semi-ViT achieves promising results on several fronts (Figure 1).
For the ﬁrst time, we show that pure ViTs can reach comparable or better accuracy than CNNs on SSL 1 . 2)
Semi-ViT can be readily scaled up under the SSL setting.
This is illustrated in Figure 1 (a) and (b) on ViT architectures at different scales, ranging from ViT-Small to ViT-Huge, and Semi-ViT outperforms the prior art such as SimCLRv2 [15].
3) Semi-ViT has shown the potential for a substantial reduction of labeling cost.
For example, as seen in Figure 1 (c), Semi-ViT-Huge with 1% (10%) ImageNet labels achieves comparable performance of a fully-supervised Inception-v4 [59] (ConvNeXt-L [45]).
This implies a 100 ⇥ ( 10 ⇥ ) reduction in human annotation cost.
4) Semi-ViT achieves the state-of-the-art SSL results on ImageNet, e.g. , 80.0% (84.3%) top-1 accuracy with only 1% (10%) labels.
In addition, the substantial boost in performance by Semi-ViT is not isolated on ImageNet: we ﬁnd an increase of 13%-21% (7%-10%) top-1 accuracy with 1% (10%) labels over the supervised ﬁne-tuning baselines, for other datasets including Food-101 [9], iNaturalist [30] and GoogleLandmark [52].
Semi-supervised Vision Transformers 2.1 Pipeline
Some pipelines for semi-supervised learning exist in the literature.
For example: 1) the model is directly trained from scratch using SSL techniques, e.g. , FixMatch [55]; 2) the model is un/selfsupervised pretrained ﬁrst and ﬁnetuned on labeled data later [26, 14, 22]; 3) the model is selfsupervised pretrained ﬁrst and then ﬁnetuned via semi-supervised learning on both labeled and unlabeled data [10].
In this paper, we instead adopt the following pipeline: ﬁrst, optional selfsupervised pre-training on all data without using any labels; next, standard supervised ﬁne-tuning on available labeled data; and ﬁnally, semi-supervised ﬁne-tuning on both labeled and unlabeled data.
This procedure is similar to [15], with the difference that they use knowledge distillation [29] in their ﬁnal stage.
We ﬁnd that this training pipeline trains semi-supervised vision transformers in a stable manner and achieves promising results, with possibly less hyperparameter tuning.
[Figure]
2.2 EMA-Teacher Framework
FixMatch [55] emerged as a popular SSL method in the past few years.
As discussed in [10], it can be interpreted as a student-teacher framework, where the student and teacher models are identical, as seen in Figure 2 (a).
However, FixMatch has unexpected behaviors, especially when the model incorporates batch normalization (BN) [35].
Although ViT uses Layer Normalization (LN) [4] instead of BN as normalization, we still found that the FixMatch with ViT underperforms the CNN counterparts and often does not converge.
This phenomenon was also observed in [68].
A potential reason to this is that the student and the teacher models are identical in FixMatch, which could easily lead to model collapse [26, 22].
This instability of the identical student-teacher framework has also been observed in other areas, e.g. semi-supervised speech recognition [42, 48, 28].
As suggested in [10], the EMA-Teacher (shown in Figure 2 (b)) is an improved version of the FixMatch, thus we adopt it for our Semi-ViT.
In the EMA-Teacher framework, the teacher parameters ✓ 0 are updated by the exponential moving average (EMA) from the student parameters ✓ ,
[Equation]
where the momentum decay m is a number close to 1, e.g. , 0.9999.
The student parameters are updated by standard learning optimization, e.g. , SGD or AdamW [46].
The other components are exactly the same as the FixMatch, as seen in Figure 2.
This temporal weight averaging can stabilize the training trajectories [3, 36] and avoids the model collapse issue [26, 22].
Our experiments also show this EMA-Teacher framework has better results and more stable training behaviors than FixMatch when training Semi-ViT.
2.3 Semi-supervised Learning Formulation
In the EMA-Teacher framework, there are both labeled and unlabeled samples in a minibatch during training.
The loss on the labeled samples { ( x li , y li ) } N l i =1 is the standard cross-entropy loss, L l = N l P N l i =1 CE ( x li , y li ) .
For an unlabeled sample x u 2 { x ui } N u i =1 , a weak and a strong augmentation are applied to it, generating x u,w and x u,s , respectively.
The weak augmented x u,w is forwarded through the teacher network, and outputs the probabilities over classes, p = f
( x u,w ; ✓ 0 ) .
Then the pseudo label is produced by ˆ y = arg max c p c with its associated conﬁdence o = max p c .
The pseudo label with conﬁdence higher than a conﬁdence threshold ⌧ is then used to supervise the learning of the student on the strong augmented sample x u,s ,
[Equation]
[Equation]
where [ · ] is the indicator function.
The overall loss is L =
L l + µ L u , where µ is the trade-off weight.
Note that only the pseudo labels with conﬁdence higher than a threshold contribute to the ﬁnal loss;
# ! $ ! ( ! ! ! $ ! ' ! " ! & !" ! !" # !" $ !"
( ℬ : ℬ$ : ℬ% : !" ! = $! ! + 1 −$ ! " ,$~*+,-(/,/) (a) Pseudo Mixup ! ! ! " !
# ! $ ! # ! $ ! # !" $ !"
( ℬ : ℬ$ : ℬ% : !" ! = $! ! + 1 −$ ! " ,$~*+,-(/,/) (b) Pseudo Mixup+ ! ! ! " !
# ! $ !
( ! ! ! $ ! ' ! " ! & !" ! !" # !" $ !" ( ℬ : ℬ$ : ℬ% : !" !
= $ ! ! !
+ 1− $ ! ! " ,$ ! = ) ! / + ) " )
[Figure]
the others are instead not used.
The philosophy behind this ﬁltering is that the pseudo labels with low conﬁdence are noisier and could hijack the SSL training.
Probabilistic Pseudo Mixup 3.1 Mixup
Mixup [75] performs convex combinations of pairs of samples and their labels, [Equation]
[Equation]
where the mixup ratio  ⇠ Beta ( ↵ , ↵ ) 2 [0 , 1] , for ↵ 2 (0 , 1 ) .
The samples are mixed-up usually in a single minibatch during training.
Given a minibatch B and its shufﬂed version ¯ B , the mixed-up minibatch is ˜ B =  B + (1   ) ¯ B , where  could be either batch-wise or element-wise.
Due to the nature of weak inductive bias, ViT is more data hungry than CNN, thus effective data augmentation, e.g. , mixup, is critical for training fully-supervised ViT [18, 63, 44, 73].
This also applies to Semi-ViT since it inherits the nature of weak inductive bias from ViT. Although it is standard to use mixup in supervised learning, how to employ it under pseudo-labeling based SSL framework, e.g. , the
EMA-Teacher, is still unclear yet, and we are going to discuss it next.
3.2 Pseudo Mixup
Under the pseudo-labeling based SSL framework [40, 55, 53, 10], given an unlabeled sample and its pseudo label ( x u , ˆ y ) , only when its conﬁdence o is not smaller than the conﬁdence threshold ⌧ , it will contribute to the loss L u , as seen in (2).
According to their conﬁdence scores, the unlabeled minibatch B u can be grouped into a clean subset ˆ B u = { ( x ui , ˆ y i ) | o i  ⌧ } and a noisy subset ˙
B u = B u  ˆ B u .
One straightforward solution is to apply mixup on the full unlabeled minibatch B u , with no differentiation between clean and noisy samples, denoted as pseudo mixup , as show in Figure 3 (a).
After the pseudo mixup, still only the samples in ˆ B u contribute to the loss, and the samples in ˙ B u are abandoned.
In this way, the mixup operation is more than just a data augmentation.
In fact, a sample in ˙ B u will also contribute to the ﬁnal loss if it is mixed-up with a sample in ˆ B u .
As a result, it could involve a substantial number of noisy samples into the loss calculation due to the randomness, which, however, is against the philosophy of pseudo-labeling.
Since only the clean subset ˆ B u contributes to the ﬁnal loss, another choice is to use mixup only on ˆ B u , denoted as pseudo mixup+ , as shown in Figure 3 (b).
In this way, no sample in the noisy subset ˙ B u will affect the training.
3.3 Probabilistic Pseudo Mixup
Although the samples in ˙ B u are noisy, they still carry some useful information for the model to learn.
The pseudo mixup above can somehow leverage those information by blending the noisy and clean pseudo samples together.
However, the problem is the mixup ratio is randomly generated from a Beta distribution, which does not depend on the conﬁdence of each sample.
This is not ideal.
For example, when two samples are mixed-up, the sample with higher conﬁdence should have higher mixup ratio, [Table] [Table] such that it can weigh more in the ﬁnal loss.
Motivated by this intuition
[Table]
we propose probabilistic pseudo mixup (Figure 3 (c))
[Table]
where the mixup ratio  reﬂects the sample conﬁdence,
[Equation]
Also [Table] the conﬁdence score of x ui is updated after the mixup operation as
[Equation]
because the conﬁdence score should align with the majority of the image content.
The ﬁnal clean subset ˜ B u = { (˜ x ui , ˜ y ui ) | o ⇤ i  ⌧ } will contribute to the ﬁnal loss.
Probabilistic pseudo mixup can enhance regularization, leverage information from all samples, even the noisy ones
[Table]
and not violate the philosophy of pseudo labeling at the same time.
It can effectively alleviate the issue of weak inductive bias of Semi-ViT and bring substantial gains, as will be shown in our experiments.
Experiments
We evaluate Semi-ViT mainly on ImageNet, which consists of ⇠ 1.28M training and 50K validation images.
We sample 10%/1% labels from the ImageNet training set for the semi-supervised evaluation.
We study both scenarios: with and without self-supervised pre-training.
Without self-pretraining, we only evaluate on 10% labels, since learning from scratch on 1% labels is very difﬁcult.
When self-pretrained, MAE [25] is mainly used, and we directly use their pretrained models.
All learning is optimized with AdamW [46], using cosine learning rate schedule, with a weight decay of 0.05.
The default momentum decay m of (1) is 0.9999.
In a minibatch, N u = 5 N l , and the loss trade-off µ = 5 .
The mixup is a combination of mixup [75] and Cutmix [74] as in the implementation of [69].
More details can be found in the appendix.
4.1 Semi-ViT Results
When the model is self-pretrained by MAE [25], we ﬁrst evaluate the ﬁne-tuning performance of MAE on the labeled data only, as the common practice in self/un-supervised learning literature [26, 14, 22], with results shown in Table 1.
This already leads to strong semi-supervised baselines, e.g. , 81.4 top-1 accuracy for ViT-Huge on 10% labels, indicating that MAE is a strong self-supervised learning technique.
However, Semi-ViT has additional signiﬁcant improvements over the strong baselines for all models, e.g. , 8.5-13.6 points for 1% labels and 2.9-6.0 points for 10% labels.
The ﬁne-tuning results on 100% data are provided as upper-bounds for our Semi-ViT, and their gaps to
Semi-ViT are small, e.g. , 4.0/2.7/2.6 points for ViT-Base/Large/Huge on 10% labels.
An interesting observation is that the larger model is more effective for smaller number of labels, which is consistent with the observations in [15].
For example, the ﬁne-tuning gaps between 1% and 100% labels are 26.3/18.9/15.4 points for ViT-Base/Large/Huge, which are decreasing.
The observation on Semi-ViT [Table] [Table] results is similar, e.g. , 12.7/8.7/6.9 points to their upper-bounds on 1% labels.
These results have shown that vision transformers can also perform very well in semi-supervised learning, as well as supervised learning and un/self-supervised learning.
4.2 Ablation Studies
We ablate different factors of Semi-ViT in this section.
FixMatch v.s. EMA-Teacher is compared in Table 2.
These experiments do not use the pseudo mixup techniques of Section 3 yet.
When the model is not self-pretrained, the training of the FixMatch is unstable and often failed.
When the model is self-pretrained, the FixMatch training becomes stable, and starts to achieve reasonable results, e.g. , 74.8 for ViT-Base on 10% labels, which is already better than the prior art on ResNet-50, e.g. , 73.9 of MPL [53] and 74.0 of EMAN [10].
But it is only 1.
points higher than the ﬁne-tuning baseline of Table 1, indicating the FixMatch is not an effective SSL framework for ViT.
But the EMA-Teacher achieves much better results, 3.3 points of improvement over the FixMatch when self-pretrained.
Even without self-pretraining, the EMA-Teacher can still achieve decent numbers, but the FixMatch is failed.
Probabilistic Pseudo Mixup Different mixup variations on unlabeled data are compared in Table 3.
Note that the standard mixup with implementation of [69] is used to the labeled data as usual.
The EMA-Teacher does not use any mixup mechanism on the unlabeled data, serving as baselines here.
When pseudo mixup of Figure 3 (a) is applied on the unlabeled data, the performance usually has some substantial gains over the EMA-Teacher baselines, especially for the scenarios where the training is more difﬁcult, e.g. , without self-pretraining or on 1% labels.
This shows the importance to use mixup on the unlabeled data for improved regularization.
However, as discussed Section 3.2, pseudo mixup could involve many noisy samples into training.
On the other hand, pseudo mixup+ of
Figure 3 (b) can increase over pseudo mixup constantly, by about 0.5 points, showing that removing those noisy samples does help.
In addition, probabilistic pseudo mixup of Figure 3 (c) can further improve over pseudo mixup+ by 1-2 points in all cases.
These results have implied that those noisy samples do carry some useful information for SSL training, but their weights should be suppressed especially when their conﬁdence scores are low.
This data augmentation technique also effectively alleviates the training difﬁculty of semi-supervised vision transformers with weak inductive bias.
Effect of Conﬁdence Threshold We ablate the effect of conﬁdence threshold ⌧ of (2) in Table 4.
We can ﬁnd Semi-ViT is quite robust to the low conﬁdence thresholds.
One possible reason is that Semi-ViT uses probabilistic pseudo mixup .
When ⌧ is low, the low-conﬁdence samples will not [Table] [Table] hijack the training since their contributions depend on their conﬁdence scores.
These imply that the hyperparameter of ⌧ can possibly be removed ( ⌧ = 0 ) in Semi-ViT. But the EMA-Teacher has a drop of 2-3 points when the conﬁdence threshold is removed.
The ﬁnal choices of ⌧ for different Semi-ViT models are shown in Table 13 in the appendix.
Effect of Momentum Decay Table 5 shows the effect of momentum decay m in the EMA teacher.
Note that when m = 0 , the frameworks is reduced to the FixMatch.
We can ﬁnd Semi-ViT is robust to m .
For 10% labels, Semi-ViT has very minor changes when m decreases to 0, but the EMA-Teacher has a drop of 3.3 points.
For 1% labels, the training is more challenging, and the choice of m becomes more important.
In this case, Semi-ViT has a drop of 2.3 points from m = 0 .
999 to m = 0 , but the
EMA-Teacher could fail when m decreases to 0.
The robustness of Semi-ViT to momentum decay m is also attributed to the use of probabilistic pseudo mixup .
Effect of Self-pretraining The self-pretraining of MAE [25] has a substantial boost in performance, as seen in Table 3.
For ViT-Base, MAE helps to improve by 6.2 and 9.2 points for the EMA-Teacher with and without probabilistic pseudo mixup , respectively.
In addition, it helps to train the models in more challenging scenarios, e.g. , 1% labels.
Without self-pretraining, the training fails to deliver good results on 1% labels.
Notice that, even without pre-training, our Semi-ViT (“ProbPseudo
Mixup” in Table 3) also achieves slightly better performance than the CNN counterparts: 70.9 of
Semi-ViT-Small v.s. 67.1 of FixMatch-ResNet50 or 69.2 [55] of EMAN-ResNet50 [10] when trained from scratch for 100 epochs.
Effect of Supervised Fine-tuning is ablated in Table 6 by varying the number of supervised ﬁne-tuning epochs.
The rows of “Supervised-ViT” are the numbers of supervised ﬁne-tuning , where the latter semi-supervised ﬁne-tuning begins.
Semi-ViT is still robust to the length of supervised ﬁne-tuning , where the accuracy decrease is 0.4 (1.3) for 10% (1%) labels when supervised ﬁne-tuning is removed.
However, the performance of the EMA-Teacher decreases 1.7 (4.2) points.
These show sufﬁcient supervised ﬁne-tuning does stabilize the training procedure, especially for less robust framework, e.g. , the EMA-Teacher.
However, notice that supervised ﬁne-tuning could sometimes hurt the performance if it is not sufﬁcient ( e.g. , epochs=10) for the EMA-Teacher.
Other Self-pretraining Techniques Beyond MAE, we also experiment on other self-pretraining techniques, including MoCo-v3 [16] and DINO [12], in Table 7.
By comparing the ﬁne-tuning results,
DINO is close to MoCo-v3 for ViT-Base but much better for ViT-Small, and both of them are better than MAE for ViT-Base, suggesting that DINO could be a better self-pretraining technique for smaller scales of ViT models.
On top of the strong ﬁne-tuning baselines, the semi-supervised ﬁne-tuning , using the EMA-Teacher, still has nontrivial improvements for both DINO and MoCo-v3, e.g. , 5. (2.1) points on 1% (10%) labels for DINO-ViT-Base.
In addition, the probabilistic pseudo mixup can further improve over the EMA-Teacher, independent of the self-pretraining algorithms.
And the ﬁnal
Semi-ViT-Base of DINO is 2.1 (0.5) points better than that of MAE on 1% (10%) labels.
[Table] [Table] Other Network Architectures Although in this paper we mainly focus on ViT architectures, the proposed probabilistic pseudo mixup is not limited to them.
We also try it for CNN architectures, e.g. ,
ResNet [27].
However, we ﬁnd the direct use of the standard mixup does not improve fully-supervised
ResNet performance, so will the probabilistic pseudo mixup for its SSL setting.
Instead, we evaluate it on the recently proposed ConvNeXt [45]
[Table]
which uses mixup for improved results.
Since the goal is not to fully reproduce the results of [45], all models are trained only for 100 epochs, including the supervised upper-bounds.
The results in Table 8 demonstrate that probabilistic pseudo mixup is not limited to ViT, but also to CNN architectures, e.g. , with improvements of 3-4 points, suggesting it can be well generalized.
4.3 Comparison with the State-of-the-Art
Semi-ViTs are compared with the state-of-the-art semi-supervised learning algorithms in Table 9.
When the model capacity is close, our Semi-ViT has shown much better results than the prior art, e.g. , MPL-RN-50 [53] v.s. Semi-ViT-Small, CowMix-RN152 [20] v.s. Semi-ViT-Base, S4L-RN504 ⇥ [8] v.s. Semi-ViT-Large and SimCLRv2+KD-RN152-3 ⇥ -SK [15] v.s. Semi-ViT-Huge.
The only transformer based SSL method is SemiFormer [68], but it requires to use CNN as the teacher model and blend convolution and transformer modules together for good performance.
However, our Semi-ViT is pure ViT based, without any additional parameters and architecture changes, and the Semi-ViT-Small model is already better than SemiFormer (77.1 v.s. 75.5).
These comparisons support that Semi-ViT does advance the state-of-the-art of semi-supervised learning.
Scalability is an advantage of ViT, and we compare the scalability of Semi-ViT with previous works in Figure 1 (a) and (b).
The comparison has shown that Semi-ViT can achieve better trade-off between model capacity and accuracy and can be scaled up more effectively than the prior art, SimCLRv2 [15].
For example, SimCLRv2 and PAWS [2] scale up the model usually in terms of network depth and width, and they seem to saturate when the model is of medium size, e.g. , around 300M parameters, but our Semi-ViT continues to improve steadily beyond that point.
Semi-ViT is also compared with the supervised state-of-the-art in Table 10.
Our Semi-ViT-Huge is comparable with Inception-v4 [59], but with 100 ⇥ annotation cost reduction, and comparable with ConvNeXt-L [45] (better than Swin-B [44]), but with 10 ⇥ annotation cost reduction.
These comparisons imply that Semi-ViT has great potential for labeling cost reduction. [Table] [Table]
4.4 Other Datasets
The generalization of Semi-ViT is evaluated on datasets including Food-101 [9], iNaturalist [30] and
GoogleLandmark [52].
Since these datasets are beyond ImageNet
[Table] we assume that the ImageNet dataset is available and the model is already supervised pretrained on ImageNet [Table] and then the model is ﬁnetuned to different target datasets with a few labels.
The results are shown in Table 11.
On these datasets
[Table]
our Semi-ViT can improve over the ﬁne-tuning baselines by 13-21 (7-10) points on 1% (10%) labels.
Note that on Food-101, Semi-ViT on 1% (10%) labels is close to ﬁne-tuning baseline on 10% (100%) labels, i
[Table]
e. , 82.1 v.s. 84.5 (91.3 v.s. 93.1), indicating that using Semi-ViT can help to save annotation costs by about 10 times on this dataset
[Table]
Related Work
Semi-supervised learning has a long history of research [78, 13].
The recent works can be roughly clustered into two groups, consistency-based [39, 62, 50, 70, 66] and pseudo-labeling based [40, 55, 53, 10].
Consistency-based methods usually add some noise to the input or the model, and then enforce their feature or probability outputs to be consistent.
For example, to construct two outputs for later consistency regularization, ⇧ -model [39] adds noise to the model weights using dropout [57], Mean-teacher [62] builds a teacher model by EMA updated from the student model, and UDA [70] applies a weak and a strong data augmentation to the input.
On the other hand, the idea of pseudo-labeling or self-training can be traced back to [34, 49], which uses model predictions as hard pseudo labels to guide the learning on unlabeled data.
This idea becomes popular in SSL recently [40, 55, 53, 10, 71], and some theoretical explanations are available [76, 24].
In the ofﬂine pseudo labeling [40, 71], the model used to generate pseudo labels is usually frozen or updated once [Table] in a while during training, e.g. , at the end of every training epoch, but for online pseudo-labeling [55, 10] the teacher model is updated continuously along with the student.
Beyond classiﬁcation, pseudo-labeling has also achieved promising progress in more challenging tasks, e.g. , object detection [56, 43, 67].
Our Semi-ViT also falls into the category of online pseudo-labeling.
Mixup [75] is an effective data augmentation technique, which interpolates the input samples and their labels linearly and performs vicinal risk minimization.
It has been successfully used in image classiﬁcation and some other domains, e.g. , generative adversarial networks [47], sentence classiﬁcation [23], etc.
Other variants have also been developed, e.g. , Manifold Mixup [65] that mixes up in the feature space or CutMix [74] which cuts a patch from one image and pastes it into another one.
Mixup has also been successfully adopted in self-supervised learning [37, 41] and semi-supervised learning [7, 66, 6].
Although [7, 66, 6] also use mixup for SSL, they have differences with our probabilistic pseudo mixup : 1) they are consistency-based SSL framework, but ours is pseudo-labeling based; 2) their mixup ratio is random sampled, but ours depends on the pseudo label conﬁdence; 3) they have only shown successes on small CNN architectures and small datasets, e.g. , CIFAR [38] and
SVHN [51], but our successes are built on various scales of transformer architectures and large-scale datasets, e.g. , ImageNet [54], INaturalist [30], GoogleLandmark [52], etc.
Conclusion
In this paper, we propose Semi-ViT for vision transformers based semi-supervised learning.
This is the ﬁrst time that pure vision transformers can achieve promising results on semi-supervised learning and even surpass the previous best CNN based counterparts by a large margin.
In addition,
Semi-ViT inherits the scalable beneﬁts from ViT, and the larger model leads to smaller gap to the fully supervised upper-bounds.
This has shown to be a promising direction for semi-supervised learning.
And the advantages of Semi-ViT can be well generalized to other datasets, suggesting potentially broader impacts.
We hope these promising results could encourage more efforts in semi-supervised vision transformers.
Limitations Our paper only considers the standard semi-supervised classiﬁcation settings where the full dataset, e.g. , ImageNet, is downsampled to smaller scales, and not the advanced settings where full ImageNet is used as labeled and additional data, e.g. , ImageNet-21K, is used as unlabeled.
And we have only evaluated on classiﬁcation task.
It is unclear whether the same conclusion can hold to that advanced classiﬁcation setting and more challenging tasks, e.g. , detection or segmentation.
Potential Negative Social Impacts Semi-ViT has shown that strong models can be obtained with only a few labels, e.g. , 1%.
This increases the good AI models accessibility to anyone, which could potentially lead to inappropriate use of them.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope? [Yes] See the abstract and introduction sections. (b) Did you describe the limitations of your work? [Yes] See the conclusion section.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See the conclusion section.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results? [N/A] (b) Did you include complete proofs of all theoretical results? [N/A] 3. If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See the supplemental material.
The code will be released upon acceptance.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See the supplemental material.
We have tried our best to provide all experimental details.
And the code will be released upon acceptance for reproduction purpose.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See the supplemental material.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See the supplemental material.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] See the references.
(b) Did you mention the license of the assets? [No] Those are common assets.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[No] Those are common assets.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]