Debiased Self-Training for Semi-Supervised Learning
Abstract
Deep neural networks achieve remarkable performances on a wide range of tasks with the aid of large-scale labeled datasets.
Yet these datasets are time-consuming and labor-exhaustive to obtain on realistic tasks.
To mitigate the requirement for labeled data, self-training is widely used in semi-supervised learning by iteratively assigning pseudo labels to unlabeled samples.
Despite its popularity, self-training is well-believed to be unreliable and often leads to training instability.
Our experimental studies further reveal that the bias in semi-supervised learning arises from both the problem itself and the inappropriate training with potentially incorrect pseudo labels, which accumulates the error in the iterative self-training process.
To reduce the above bias, we propose Debiased Self-Training (DST) .
First, the generation and utilization of pseudo labels are decoupled by two parameter-independent classifier heads to avoid direct error accumulation.
Second, we estimate the worst case of self-training bias, where the pseudo labeling function is accurate on labeled samples, yet makes as many mistakes as possible on unlabeled samples.
We then adversarially optimize the representations to improve the quality of pseudo labels by avoiding the worst case.
Extensive experiments justify that DST achieves an average improvement of 6 . 3% against state-of-the-art methods on standard semisupervised learning benchmark datasets and 18 .
9% against FixMatch on 13 diverse tasks.
Furthermore, DST can be seamlessly adapted to other self-training methods and help stabilize their training and balance performance across classes in both cases of training from scratch and finetuning from pre-trained models.
Introduction
Deep learning has achieved great success in many machine learning problems in the past decades, especially where large-scale labeled datasets are present.
In real-world applications, however, manually labeling sufficient data is time-consuming and labor-exhaustive.
To reduce the requirement for labeled data, semi-supervised learning (SSL) improves the data efficiency of deep models by learning from a few labeled samples and a large number of unlabeled samples [20, 30, 51, 7].
Among them, self-training is an effective approach to deal with the lack of labeled data.
Typical self-training methods [30, 47] assign pseudo labels to unlabeled samples with the model’s predictions and then iteratively train the model with these pseudo labeled samples as if they were labeled examples.
Although self-training has achieved great advances in benchmark datasets, they still exhibit large training instability and extreme performance imbalance across classes.
For instance, the accuracy of
FixMatch [47], one of the state-of-the-art self-training methods, fluctuates greatly when trained from scratch (see Figure 7).
Though its performance will gradually recover after a sudden sharp drop, this is still not expected, since pre-trained models are more often adopted [14, 7, 24] are improve data efficiency, and the performance of pre-trained models is difficult to recover after a drastic decline due
Equal to catastrophic forgetting [25].
Besides, although FixMatch improves the average accuracy, it also leads to the Matthew effect , i.e. , the accuracy of well-behaved categories is further increased while that of poorly-behaved ones is decreased to nearly zero (see Figure 4).
This is also not expected, since most machine learning models prefer performance balance across categories, even when the class imbalance exists in the training data [65].
The above findings are caused by the bias between the pseudo labeling function with the unknown target labeling function.
Training with biased and unreliable pseudo labels has the chance to accumulate errors and ultimately lead to performance fluctuations.
And for those poorly-behaved categories, the bias of the pseudo labels gets worse and will be further enhanced as self-training progresses, ultimately leading to the Matthew effect.
We delved into the bias issues arising from the self-training process and found that they can be briefly grouped into two kinds: (1) Data bias , the bias inherent in the SSL tasks; (2) Training bias , the bias increment brought by self-training with incorrect pseudo labels.
In this regard, we present Debiased
Self-Training (DST) , a novel approach to decrease the undesirable bias in self-training.
Specifically, to reduce the training bias , the classifier head is only trained with clean labeled samples and no longer trained with unreliable pseudo-labeled samples.
In other words, the generation and utilization of pseudo labels are decoupled to mitigate bias accumulation and boost the model’s tolerance to biased pseudo labels.
Further, to decrease the data bias which cannot be calculated directly, we turn to estimate the worst case of training bias that implicitly reflects the data bias.
Then we optimize the representations to decrease the worst-case bias and thereby improve the quality of pseudo labels.
The contributions of this work are summarized as follows: (1) We systematically identify the problem and analyze the causes of self-training bias in semi-supervised learning.
(2) We propose DST , a novel approach to mitigate the self-training bias and boost the stability and performance balance across classes, which can be used as a universal add-on for mainstream self-training methods.
We conduct extensive experiments and validate that DST achieves an average boost of 6 . 3% against state-of-the-art methods on standard datasets and 18 . 9% against FixMatch on 13 diverse tasks.
Related Work 2.1 Self-training for semi-supervised learning
Self-training [60, 43, 20, 30] is a widely-used approach to utilize unlabeled data.
Pseudo Label [30], one popular self-training method, iteratively generates pseudo labels and utilizes them with the same model.
However, this paradigm suffers from the problem of confirmation bias [1], where the learner struggles to correct its own mistakes when learning from inaccurate pseudo labels.
The bias issue is also mentioned in DebiasMatch [54] where they define the bias as the quantity imbalance for each category.
Note that the bias in our paper refers to the deviation between the pseudo labeling function and the ground truth labeling function, which is a more essential problem existing in most self-training methods.
Recent works mainly tackle this bias issue from the following two aspects.
Generate higher-quality pseudo labels.
MixMatch [4] averages predictions from multiple augmentations as pseudo labels.
ReMixMatch [3], UDA [57], and FixMatch [47] adopt confidence thresholds to generate pseudo labels on weakly augmented samples and utilize these pseudo-labels as annotations for strongly augmented samples.
Dash [59] and FlexMatch [62] dynamically adjust the thresholds in a curriculum learning manner.
Label Propagation methods [46, 23] assign pseudo labels with the density of the local neighborhood.
DASO [38] blends the confidence-based pseudo labels and density-based pseudo labels differently for each class.
Meta Pseudo Labels [41] proposes to generate pseudo labels with a meta learner.
Different from the above methods that manually design specific criteria to improve the quality of pseudo labels, we estimate the worst case of self-training bias and adversarially optimize the representations to improve the quality of pseudo labels automatically.
Improve tolerance to inaccurate pseudo labels.
To mitigate the confirmation bias, existing methods maintain a mismatch between the generation and utilization of pseudo labels.
Temporal Ensembling [29] and Mean Teacher [51] generate pseudo labels from the average of previous predictions or an exponential moving average of the model, respectively.
Noisy Student [58] assigns pseudo labels by a fixed teacher from the previous round.
Co-training [5], MMT [17], DivideMix [31] and Multi-head
Tri-training [44] introduce multiple models or classifier heads and learn in an online mutual-teaching manner.
In these methods, each classifier head is still trained with potentially incorrect pseudo labels generated by other heads.
In contrast, in our method, the classifier head that generates pseudo labels is never trained with pseudo labels , leading to better tolerance to inaccurate pseudo labels (Table 3).
2.2 Self-supervised learning for semi-supervised learning
Self-supervised methods [14, 21] are also used on unlabeled data to improve the model with few labeled samples, either in the pre-training stage [7, 2] or in the downstream tasks [53, 32].
However, the training of self-supervision usually relies on big data and heavy computation, which is not feasible in most applications.
Besides, although these methods avoid the use of unreliable pseudo labels, it is difficult for them to learn task-specific information from unlabeled data for better performance.
2.3 Adversarial training for semi-supervised learning
Some works introduce adversarial training [18] into semi-supervised learning.
A line of works [37, 45, 12, 15] exploit fake samples from the generator by labeling them with a new “generated” class and forcing the discriminator to output class labels.
Another line of works use adversarial training to construct adversarial samples [19], e.g., VAT [34] injects additive noise into input, VAdD [39] introduces adversarial Dropout [48] layers and RAT [50] expands the noise in VAT into a set of input transformations.
These methods aim to impose a local smoothness on the model and do not involve training with pseudo labels.
In contrast, in our method, the goal of the adversarial training is to estimate the worst case of pseudo labeling and then avoid such cases (Section 4.2).
Analysis of Bias in Self-Training
In this section, we provide some analysis of where the bias in self-training comes from.
Let P denote a distribution over input space X .
For classification with K classes, let P k denote the class-conditional distribution of x conditioned on ground truth
f ∗ ( x ) = k .
Assume that pseudolabeler f pl is obtained via training a classifier on n labeled samples (cid:98) P n .
Let M ( f pl ) ≜ { x : f pl ( x ) ̸ = f ∗ ( x ) } denote the mistaken pseudolabeled samples.
The bias in the self-training refers to the deviation between the learned decision hyperplanes and the true decision hyperplanes , which can be measured by the fraction of incorrectly pseudolabeled samples in any classes B ( f pl ) =
{ P k ( M ( f pl )) }
Kk =1 [55].
By analyzing self-training bias under different training conditions, we have several nontrivial findings.
[Figure] [Figure]
Accuracy of 7 randomly selected categories with different pre-trained models on CIFAR-100 .
Different pre-trained models show different category preferences.
[Figure]
The sampling of labeled data will largely influence the self-training bias.
As shown in Figure 1, when the data sampling is different, the accuracy of the same category may vary dramatically.
The reason is that the distances between different data points and the true decision hyperplanes are not the same, with some supporting data points closer and others far away.
When there are few labeled data, there may be a big difference in the distances between supporting data of each category and the true decision hyperplanes, hence the learned decision hyperplanes will be biased towards some categories.
The pre-trained representations also affect the self-training bias.
Figure 2 shows that different pre-trained representations lead to different category bias, even if the pre-trained dataset and the downstream labeled dataset are both identical.
One possible reason is that the representations learned by different pre-trained models focus on different aspects of the data [64].
Therefore, the same data could also have different distances to the decision hyperplanes in the representation level with different pre-trained models.
Training with pseudo labels aggressively in turn enlarges the self-training bias on some categories.
Figure 3 shows that after training with pseudo labels ( e.g. , using FixMatch), the performance gap for different categories greatly enlarges, with the accuracy of some categories increasing from 60% to 80% and that of some categories dropping from 15% to 0% .
The reason is that for well-behaved categories, the pseudo labels are almost accurate, hence using them for training could further reduce the bias.
Yet for many poorly-behaved categories, the pseudo labels are not reliable, and the common self-training mechanism that uses these incorrect pseudo labels to train the model will further increase the bias, and fail to correct it back in the follow-up training.
This results in the Matthew effect.
FixMatch decreases the bias on well-behaved categories while increasing that of poorly-behaved categories.
In contrast, DST effectively balances the performance between different categories.
Based on the above observations, we divide the bias caused by self-training into two categories.
Data bias : the bias inherent in semi-supervised learning tasks, such as the bias of sampling and pretrained representations on unlabeled data.
Formally, data bias is defined as B ( f pl ( (cid:98) P n , ψ 0 ))
− B ( f ∗ ) (blue area in Figure 4), where the pseudolabeler f pl ( (cid:98) P n , ψ 0 ) is obtained from a biased sampling (cid:98) P n with a biased parameter initialization ψ 0 .
Training bias : the bias increment brought by some unreasonable training strategies.
Formally, training bias is B ( f pl ( (cid:98) P n , ψ 0 , S ))
− B ( f pl ( (cid:98) P n , ψ 0 ))
(yellow area in Figure 4) where f pl ( (cid:98) P n , ψ 0 , S ) is a pseudolabeler obtained with self-training strategy S .
Next we will introduce how to reduce training bias and data bias in self-training (red area in Figure 4).
Debiased Self-Training
In semi-supervised learning (SSL), we have a labeled dataset L = { ( x li , y li } n l i =1 of n l labeled samples and an unlabeled dataset U = { ( x uj ) } n u j =1 of n u unlabeled samples, where the size of the labeled dataset is usually much smaller than that of the unlabeled dataset, i.e. , n l ≪ n u .
Denote ψ the feature generator, and h the task-specific head.
The standard cross-entropy loss on weakly augmented labeled examples is
[Equation]
where α is the weak augmentation function.
Since there are few labeled samples, the feature generator and the task-specific head will easily over-fit, and typical SSL methods use these pseudo labels on plenty of unlabeled data to decrease the generalization error.
Different SSL methods design different pseudo labeling function (cid:98) f [30, 59, 42].
Take FixMatch [47] for an instance.
FixMatch first generates predictions (cid:98) p
= ( h ◦ ψ ◦ α )( x ) on a weakly augmented version of given unlabeled images, and adopts a confidence threshold τ to filter out unreliable pseudo labels,
[Equation]
where (cid:98) f ψ,h refers to the pseudo labeling by model h ◦ ψ , hyperparameter τ specifies the threshold above which a pseudo label is retained and − 1 indicates that this pseudo label is ignored in training.
Then FixMatch utilizes selected pseudo labels to train on strongly augmented unlabeled images,
[Equation]
where (cid:98) f is a notation of general pseudo labeling function and A is the strong augmentation function.
As shown in Figure 5(a), the optimization objective for FixMatch is
[Equation]
where λ is the trade-off between the loss on labeled data and that on unlabeled data.
FixMatch filters out low-confidence samples during the pseudo labeling process, yet two issues remain: (1) The pseudo labels are generated and utilized by the same head, which leads to the training bias, i.e. , the errors of the model might be amplified as the self-training progresses.
(2) When trained with extreme few labeled samples, the problem of unreliable pseudo labeling caused by data bias cannot be ignored anymore even with the confidence threshold mechanism.
To tackle the above issues, we propose two important designs to decrease training bias and data bias in Section 4.1 and 4.2 respectively.
4.1 Generate and utilize pseudo labels independently
The training bias of FixMatch stems from the way of training on the pseudo labels generated by itself.
To alleviate this bias, some methods turn to generate pseudo labels from a better teacher model, such as the moving average of the original model [51] in Figure 5(b) or the model obtained from the previous round of training [58] in Figure 5(c), and then utilize these pseudo labels to train both the feature generator ψ and the task-specific head h .
However, there is still a tight relationship between the teacher model that generates pseudo labels and the student model that utilizes pseudo labels in the above methods, and the decision hyperplanes of the student model h ◦ ψ strongly depend on the biased pseudo labeling (cid:98) f .
As a result, training bias is still large in the self-training process.
[Figure]
! 𝐿 "# 𝐱 $ #𝒑 !
#𝒑 $ 𝐿 "# %y $ 𝐱 !
𝐿 "# 𝐱 $ #𝒑 ! ’
𝒑 $ 𝐿 !"
# #𝒑 $ 𝐱 !
[Figure]
$ #𝒑 ! ’
𝒑 $
[Figure]
To further decrease the training bias when utilizing the pseudo labels, we optimize the task-specific head h , only with the clean labels on L and without any unreliable pseudo labels from U .
To prevent the deep models from over-fitting to the few labeled samples, we still use pseudo labels, but only for learning a better representation.
As shown in Figure 5(d), we introduce a pseudo head h pseudo , which is connected to the feature generator ψ and only optimized with pseudo labels from U .
Then the training objective is
[Equation]
where the pseudo labels are generated by head h and utilized by a completely parameter independent pseudo head h pseudo .
Although h and h pseudo are fed with features from the same backbone network, their parameters are independent, thus training the pseudo head h pseudo with some wrong pseudo labels will not accumulate the bias of head h directly in the iterative self-training process.
Note that the pseudo head h pseudo is only responsible for gradient backpropagation to the feature generator ψ during training and will be discarded during inference, and thus will introduce no inference cost.
4.2 Reduce generation of erroneous pseudo labels
Section 4.1 presents a solution to reduce the training bias, yet the data bias still exists in the pseudo labeling (cid:98) f .
As shown in Figure 6(a), due to the data bias, labeled samples of each class have different distances to the decision hyperplanes in the representation space, which leads to a deviation between the learned hyperplanes and the real decision hyperplanes, especially when the size of labeled samples is very small.
As a result, pseudo labeling (cid:98) f is very likely to generate incorrect pseudo labels on unlabeled data points that are close to these biased decision hyperplanes.
And our objective now is to optimize the feature representations to reduce the data bias, and finally improve the quality of pseudo labels.
Since we have no labels for U , we cannot directly measure the data bias and thereby reduce it.
Yet training bias has some correlations with data bias.
Recall in Section 4.1, the task-specific head h is only optimized with clean labeled data, since optimization with incorrect pseudo labels will push the learned hyperplanes in a more biased direction and lead to the training bias.
Therefore, training bias can be considered as the accumulation of data bias with inappropriate utilization of pseudo labels, which is training algorithm dependent.
And the worst training bias that can be achieved by some self-training methods is a good measure of data bias.
Specifically, the worst training bias corresponds to the worst possible head h
learned by pseudo labeling, such that h ′ predicts correctly on all the labeled samples L while making as many mistakes as possible on unlabeled data U ,
[Equation] [Equation]
where the mistakes of h ′ on unlabeled data are estimated by its discrepancy with the current pseudo labeling function (cid:98) f .
Equation 6 aims to find the worst-case of task-specific head h that might be learned in the future when trained with pseudo labeling on the current feature generator ψ and the
[Figure]
current data sampling.
It is also the worst hyperplanes as shown in Figure 6(b), which deviates as much as possible from the currently learned hyperplanes while ensuring that all labeled samples are correctly distinguished.
Note that Equation 6 measures the degree of data bias, which depends on the feature representations generated by ψ , thus we can adversarially optimize feature generator ψ to indirectly decrease the data bias,
[Equation]
As shown in Figure 6(c), Equation 7 encourages the feature of unlabeled samples to be distinguished correctly even by the worst hyperplanes, i.e. , be generated far away from the current hyperplanes, thereby reducing the data bias in feature representations.
Overall loss.
The final objective of the Debiased Self-Training (DST) approach is to reduce both training bias and data bias.
The overall loss function simultaneously decouples the generation and utilization of pseudo-labels and avoids the worst-case hyperplanes.
This is achieved by unifying
Equations 5–7 into a minimax game:
[Equation]
Experiments
Following [47, 59], we evaluate Debiased Self-Training (DST) with random initialization on common
SSL datasets, including CIFAR-10 [28], CIFAR-100 [28], SVHN [35] and STL-10 [10].
Following [53], we also evaluate DST with both supervised pre-trained models and unsupervised pre-trained models on 11 downstream tasks, including (1) superordinate-level object classification: CIFAR-
[28], CIFAR-100 [28], Caltech-101 [16]; (2) fine-grained object classification: Food-101 [6], CUB200-2011 [52], Stanford Cars [27], FGVC Aircraft [33], OxfordIIIT Pets [40], Oxford Flowers [36]; (3) texture classification: DTD [9]; (4) scene classification: SUN397 [56].
The complete training dataset size ranges from 2 , 040 to 75 , 750 and the number of classes ranges from 10 to 397 .
Following [26], we report mean accuracy per-class on Caltech-101 , FGVC Aircraft , OxfordIIIT Pets , Oxford
Flowers , and top-1 accuracy for other datasets.
Following [47], we construct a labeled subset withlabels per category to verify the effectiveness of DST in extremely label-scarce settings.
To make a fair comparison, we keep the labeled subset for each dataset the same throughout our experiments.
For experiments with random initialization, we follow [47] and adopt Wide ResNet variants [61].
For experiments with pre-trained models, we adopt ResNet50 [22] with an input size of 224 ×and pre-trained on ImageNet [13].
We adopt MoCo v2 [8] as unsupervised pre-trained models.
We compare our method with many state-of-the-art SSL methods, including Pseudo Label [30], Π -Model [29], Mean Teacher [51], VAT [34], ALI [15], RAT [50], UDA [57], MixMatch [4], ReMixMatch [3],
FixMatch [47], Dash [59], Self-Tuning [53], FlexMatch [62] and DebiasMatch [54].
When training from scratch, we adopt the same hyperparameters as FixMatch [47], with learning rate of 0 .
03 , mini-batch size of 512 .
For other experiments, we use SGD with momentum 0 .
9 and learning rates in { 0 . 001 , 0 . 003 , 0 . 01 , 0 . 03 } .
The mini-batch size is set to 64 following [49].
For each image, we first apply random-resize-crop and then use RandAugment [11] for strong augmentation
A and random-horizontal-flip for weak augmentation α .
More details on hyperparameter selection can be found in Appendix A.2.
Each experiment is repeated three times with different random seeds.
We have released a benchmark containing both the code for our method and that for all the baselines at https://github.com/thuml/Debiased-Self-Training .
5.1 Main results
Table 1 shows that DST yields consistent improvement on all tasks.
On the challenging CIFAR100 and STL-10 tasks, DST boosts the accuracy of FixMatch and FlexMatch by 8.3% and 10.7% , respectively.
Figure 7 depicts the top-1 accuracy during the training procedure on CIFAR-100 .
We observe that the performance of FixMatch suffers from significant fluctuations during training.
In contrast, the accuracy of DST (FixMatch) increases steadily and surpasses the best accuracy of
FixMatch by 10.9% , relatively.
Note that the accuracy of FlexMatch also drops by over 6% in the final stages of training while DST (FlexMatch) suffers from a much smaller drop by reducing erroneous pseudo labels during the self-training process.
Besides, DST also improves the performance balance across categories (see Appendix B.2).
[Table]
[Figure]
[Table]
[Figure]
5.2 Transfer from a pre-trained model
Supervisied pre-training.
Table 2 reveals that typical self-training methods, e.g. FixMatch, lead to relatively mild improvements with supervised pre-trained models, which is consistent with previous findings [49, 53].
In contrast, incorporating DST into FixMatch significantly boosts the performance and surpasses FixMatch by 19.9% on all datasets.
With a pre-trained model, self-training has better training stability.
Yet once the performance degradation occurs, the process is also irreversible (Appendix B.1), partly due to the catastrophic forgetting of pre-trained representation.
Also, selftraining suffers from a more severe performance imbalance across classes (Appendix B.1).
DST effectively tackles these issues, indicating the importance of reducing bias.
[Table]
Unsupervised pre-training.
Table 2 shows that with unsupervised pre-trained models, more methods suffer from performance degradation after self-training on the unlabeled data.
The difficulty comes from that the unsupervised pre-training task has a larger task discrepancy with the downstream classification tasks than the supervised pre-training task.
Thus, the representations learned by unsupervised pre-trained models usually exhibit stronger data bias, and inappropriate usage of pseudo labels will lead to rapid accumulation errors and increase the training bias.
By eliminating training bias and reducing data bias, DST brings improvement on all datasets and relatively outperforms FixMatch by 23.5% on average, superior to FlexMatch and DebiasMatch in 9 and 10 tasks, respectively [Table]
5.3 Ablation studies
We examine the design of our method on CIFAR-100 in Table 3 and have the following findings.
Compared with Mutual Learning [63, 17], where two heads provide pseudo labels to each other, the independent mechanism in our method where one head is only responsible for generating pseudo labels and the other head only uses them for self-training can better reduce the training bias.
A nonlinear pseudo head is always better than a linear pseudo head.
We conjecture that nonlinear projection can reduce the degeneration of representation with biased pseudo labels.
(3) The worst-case estimation of pseudo labeling improves the performance by large margins.
[Table]
5.4 Analysis
To further investigate how DST improves pseudo labeling and self-training performance, we conduct some analysis on CIFAR-100 .
For simplicity, we only give the results with supervised pre-trained models.
More comparisons can be found in Appendix B.4.
DST improves both the quantity and quality of pseudo labels.
As shown in Figures 8(a) and 8(b),
FixMatch exploits unlabeled data aggressively , on average producing more than 70% pseudo labels during training.
But the cost is that the accuracy of pseudo labels continues to drop, eventually falling below 60% , which is consistent with our motivation in Section 3 that inappropriate utilization of pseudo labels will in turn enlarges the training bias.
On the contrary, the accuracy of pseudo labels in
DST suffers from a smaller drop.
Rather, it keeps rising afterward and exceeds 70% throughout the training.
Besides, DST generates more pseudo labels in the later stages of training.
[Figure]
DST generates better pseudo labels for poorly-behaved classes.
To measure the quantity of pseudo labels on poorly-behaved classes, we calculate the class imbalance ratio I on a class-balanced validation set, I = max c N ( c ) / min c ′ N ( c ′ ) , where N ( c ) denotes the number of predictions that fall into category c .
As shown in Figure 8(c), the class imbalance ratio of FixMatch rises rapidly and reaches infinity after 5000 iterations, indicating that the model completely ignores those poorlylearned classes.
To measure the quality of pseudo labels on poorly-behaved classes, we calculate the average accuracy of 10 or 20 worst-behaved classes in Figure 8(d).
The average accuracy on the worst 20 classes of FixMatch is only 1.0% .
By reducing training bias with the pseudo head and data bias with the worst-case estimation, the average accuracy balloons to 28.5% and 34.5% , respectively.
5.5 Convergence and computation cost of the min-max optimization
We optimize ψ and h ′ with stochastic gradient descent alternatively.
The optimization can be viewed as an alternative form of GAN [18].
Figure 9 shows that the worst-case error rate of h ′ and worstcase loss in Equation 7 first increase ( h ′ dominates), and then gradually decrease and converge ( ψ dominates).
When training 1000 k iterations on CIFAR-100 using 4 2080 Ti GPUs, FixMatch takes 104 hours while DST takes 111 hours, only a 7% increase in time.
Note that DST introduces no additional computation cost during inference
[Figure]
[Figure] [Figure]
[Table]
5.6 DST as a general add-on
We incorporate DST into several representative self-training methods, including FixMatch [47], Mean
Teacher [51], Noisy Student [58], DivideMix [31] and FlexMatch [62].
Implementation details of
DST versions of these methods can be found in Appendix A.3.
Table 10 compares the original and
DST versions of these methods on CIFAR-100 with both supervised pre-trained and unsupervised pre-trained models.
Results show that the proposed DST yields large improvement on all these selftraining methods, indicating that self-training bias widely exists in existing vanilla or sophisticated self-training methods and DST can serve as a universal add-on to reduce the bias.
Conclusion
To mitigate the requirement for labeled data, pseudo labels are widely used on the unlabeled data, yet they suffer from severe confirmation bias.
In this paper, we systematically delved into the bias issues and present Debiased Self-Training (DST), a novel approach to decrease bias in self-training.
Experimentally, DST achieves state-of-the-art performance on 13 semi-supervised learning tasks and can serve as a universal and beneficial add-on for existing self-training methods.
Acknowledgements
This work was supported by the National Key Research and Development Plan (2021YFB1715200),
National Natural Science Foundation of China (62022050 and 62021002), Beijing Nova Program (Z201100006820041), BNRist Innovation Fund (BNR2021RC01002), and Kuaishou Research Fund.