Debiased Self-Training for Semi-Supervised Learning
Anonymous Author(s)
AffiliationAddress email
Abstract
Deep neural networks achieve remarkable performances on a wide range of taskswith the aid of large-scale labeled datasets.
Yet these datasets are time-consumingand labor-exhaustive to obtain on realistic tasks.
To mitigate the requirement forlabeled data, self-training is widely used in semi-supervised learning by iterativelyassigning pseudo labels to unlabeled samples.
Despite its popularity, self-training iswell-believed to be unreliable and often leads to training instability.
Our experimental studies further reveal that the bias in semi-supervised learning arises from boththe problem itself and the inappropriate training with potentially incorrect pseudolabels, which accumulates the error in the iterative self-training process.
To reducethe above bias, we propose Debiased Self-Training (DST) .
First, the generation andutilization of pseudo labels are decoupled by two parameter-independent classifierheads to avoid direct error accumulation.
Second, we estimate the worst caseof self-training bias, where the pseudo labeling function is accurate on labeledsamples, yet makes as many mistakes as possible on unlabeled samples.
We thenadversarially optimize the representations to improve the quality of pseudo labelsby avoiding the worst case.
Extensive experiments justify that DST achieves anaverage improvement of 6 . 3% against state-of-the-art methods on standard semisupervised learning benchmark datasets and 18 .
9% against FixMatch on 13 diversetasks.
Furthermore, DST can be seamlessly adapted to other self-training methodsand help stabilize their training and balance performance across classes in bothcases of training from scratch and finetuning from pre-trained models.
Introduction
Deep learning has achieved great success in many machine learning problems in the past decades,especially where large-scale labeled datasets are present.
In real-world applications, however,manually labeling sufficient data is time-consuming and labor-exhaustive.
To reduce the requirementfor labeled data, semi-supervised learning (SSL) improves the data efficiency of deep models bylearning from a few labeled samples and a large number of unlabeled samples [19, 29, 48, 6].
Amongthem, self-training is an effective approach to deal with the lack of labeled data.
Typical self-trainingmethods [29, 44] assign pseudo labels to unlabeled samples with the model’s predictions and theniteratively train the model with these pseudo labeled samples as if they were labeled examples.
Although self-training has achieved great advances in benchmark datasets, they still exhibit largetraining instability and extreme performance imbalance across classes.
For instance, the accuracy of
FixMatch [44], one of the state-of-the-art self-training methods, fluctuates greatly when trained fromscratch (see Figure 7).
Though its performance will gradually recover after a sudden sharp drop, thisis still not expected, since in most real-world applications, pre-trained models are more often adopted[13, 6, 23], and the performance of pre-trained models is difficult to recover after a drastic declinedue to catastrophic forgetting [24].
Besides, although FixMatch improves the average accuracy, italso leads to the Matthew effect , i.e. , the accuracy of well-behaved categories is further increasedwhile that of poorly-behaved ones is decreased to nearly zero (see Figure 4).
This is also not expected, since most machine learning models prefer performance balance across categories, even when theclass imbalance exists in the training data [62].
The above findings are caused by the bias betweenthe pseudo labeling function with the unknown target labeling function.
Training with biased andunreliable pseudo labels has the chance to accumulate errors and ultimately lead to performancefluctuations.
And for those poorly-behaved categories, the bias of the pseudo labels is worse and willbe further enhanced as self-training progresses, ultimately leading to the Matthew effect.
To escape from the dilemma, we first delved into the bias issues arising from the self-training processand found that they can be briefly grouped into two kinds: (1) Data bias which is the bias inherentin the SSL tasks; (2) Training bias which refers to the bias increment brought by self-training withincorrect pseudo labels.
Further, we present Debiased Self-Training (DST) , a novel approach todecrease the above bias in self-training.
Specifically, to reduce the training bias , the classifier headis only trained with clean labeled samples and no longer trained with unreliable pseudo-labeledsamples.
In other words, the generation and utilization of pseudo labels are decoupled to mitigatebias accumulation and boost the model’s tolerance to biased pseudo labels.
Further, to decrease thedata bias which cannot be calculated directly, we turn to estimate the worst case of training bias thatimplicitly reflects the data bias.
Then we optimize the representations to decrease the worst-case biasand thereby improve the quality of pseudo labels.
The contributions of this work are summarized as follows: (1) We systematically analyze the problemand the causes of self-training bias in SSL.
(2) We propose DST , a novel approach to mitigate theself-training bias and boost the stability and performance balance across classes, as well as a universaladd-on for mainstream self-training methods.
(3) We conduct extensive experiments and validatethat DST achieves an average improvement of 6 . 3% against state-of-the-art methods on standard SSLbenchmarks datasets and 18 .
9% against FixMatch on 13 diverse tasks.
Related Work
2.1 Self-training for semi-supervised learning
Self-training [57, 40, 19, 29] is a widely-used approach to utilize unlabeled data.
Pseudo Label [29],one popular self-training method, iteratively generates pseudo labels and utilizes them with the samemodel.
However, this paradigm suffers from the problem of confirmation bias [1], where the learnerstruggles to correct its own mistakes when learning from inaccurate pseudo labels.
The bias issueis also mentioned in DebiasMatch [51] where they define the bias as the quantity imbalance foreach category.
Note that the bias in our paper refers to the deviation between the pseudo labelingfunction and the ground truth labeling function, which is a more essential problem existing in mostself-training methods.
Recent works mainly tackle this bias issue from the following two aspects.
Generate higher-quality pseudo labels.
MixMatch [3] averages predictions from multiple augmentations as pseudo labels.
ReMixMatch [2], UDA [54], and FixMatch [44] adopt confidencethresholds to generate pseudo labels on weakly augmented samples and utilize these pseudo-labelsas annotations for strongly augmented samples.
Dash [56] and FlexMatch [59] dynamically adjustthe thresholds in a curriculum learning manner.
Label Propagation methods [43, 22] assign pseudolabels with the density of the local neighborhood.
Meta Pseudo Labels [38] proposes to generatepseudo labels with a meta learner.
Different from the above methods that manually design specificcriteria to improve the quality of pseudo labels, we estimate the worst case of self-training bias andadversarially optimize the representations to improve the quality of pseudo labels automatically.
Improve tolerance to inaccurate pseudo labels.
To mitigate confirmation bias, existing methodsmaintain a mismatch between the generation and utilization of pseudo labels.
Temporal Ensembling[28] and Mean Teacher [48] generate pseudo labels from the average of previous predictions or anexponential moving average of the model, respectively.
Noisy Student [55] assigns pseudo labels bya fixed teacher from the previous round.
Co-training [4], MMT [16], DivideMix [30] and Multi-head Tri-training [41] introduce multiple models or classifier heads and learn in an online mutual-teachingmanner.
In these methods, each classifier head is still trained with potentially incorrect pseudo labelsgenerated by other heads.
In contrast, in our method, the classifier head that generates pseudo labelsis never trained with pseudo labels , leading to better tolerance to inaccurate pseudo labels (Table 3).
2.2 Self-supervised learning for semi-supervised learning
Self-supervised methods [13, 20] are also used on unlabeled data to improve the model with fewlabeled samples, either in the pre-training stage [6] or in the downstream tasks [50].
However, the training of self-supervision usually relies on big data and heavy computation, which is not feasible inmost applications.
Besides, although these methods avoid the use of unreliable pseudo labels, it isdifficult for them to learn task-specific information from unlabeled data for better performance.
2.3 Adversarial training for semi-supervised learning
Some works introduce adversarial training [17] into semi-supervised learning. A line of works[35, 42, 11, 14] exploit fake samples from the generator by labeling them with a new “generated”class and forcing the discriminator to output class labels. Another line of works use adversarialtraining to construct adversarial samples [18], e.g., VAT [32] injects additive noise into input, VAdD[36] introduces adversarial Dropout [45] layers and RAT [47] expands the noise in VAT into a set ofinput transformations.
These methods aim to impose a local smoothness on the model and do notinvolve training with pseudo labels.
In contrast, in our method, the goal of the adversarial training isto estimate the worst case of pseudo labeling and then avoid such cases (Section 4.2).
Analysis of Bias in Self-Training
In this section, we provide some analysis of where the bias in self-training comes from.
Let P denote adistribution over input space X .
For classification with K classes, let P k denote the class-conditionaldistribution of x conditioned on ground truth
f ∗ ( x ) = k .
Assume that pseudolabeler f pl is obtainedvia training a classifier on the n labeled data (cid:98) P n .
Let M ( f pl ) ≜ { x : f pl ( x ) ̸ = f ∗ ( x ) } denote themistaken pseudolabeled samples.
The bias in the self-training refers to the deviation between thelearned decision hyperplanes and the true decision hyperplanes , which can be measured by thefraction of incorrectly pseudolabeled samples in any classes B ( f pl ) =
{ P k ( M ( f pl )) }
Kk =1 [52].
Byanalyzing self-training bias under different training conditions, we have the following findings.
The sampling of labeled data will largely influence the self-training bias.
As shown in Figure 1, whenthe data sampling is different, the accuracy of the same category may be very high or very low.
Thereason is that the distances between different data points and the true decision hyperplanes are not thesame, with some supporting data points closer and others far away.
When there are few labeled data,there may be a big difference in the distances between supporting data of each category and the truedecision hyperplanes, thus the learned decision hyperplanes will be biased towards some categories. [Figure] [Figure]
Accuracy of 7 randomly selected categories with different pre-trained models on CIFAR-100 .
Different pre-trained models show different category preferences.
[Figure]
The pre-trained representations also affect the self-training bias.
Figure 2 shows that differentpre-trained representations lead to different category bias, even if the pre-trained dataset and thedownstream labeled dataset are both identical.
One possible reason is that the representations learnedby different pre-trained models focus on different aspects of the data [61].
Therefore, the samedata could also have different distances to the decision hyperplanes in the representation level withdifferent pre-trained models.
Training with pseudo labels aggressively in turn enlarges the self-training bias on some categories.
Figure 3 shows that after training with pseudo labels ( e.g. , using FixMatch), the performance gapfor different categories greatly enlarges, with the accuracy of some categories increasing from 60%to 80% and that of some categories dropping from 15% to 0% .
The reason is that for well-behavedcategories, the pseudo labels are almost accurate, thus using them for training could further reducethe bias.
Yet for many poorly-behaved categories, the pseudo labels are not reliable, and the commonself-training mechanism that uses these incorrect pseudo labels to train the model will further increasethe bias, and fail to correct it back in the follow-up training.
Based on the above observations, we divide the bias in self-training into two categories.
FixMatch decreases the bias on well-behaved categories while increasing that of poorly-behaved categories effectively the performance different categories
Data bias : the bias inherent in SSL tasks, suchas the bias of sampling and pre-trained representations on unlabeled data.
Formally, data biasis defined as B ( f pl ( (cid:98) P n , ψ 0 ))
− B ( f ∗ ) (blue areain Fig. 4), where the pseudolabeler f pl ( (cid:98) P n , ψ 0 )is obtained from a biased sampling (cid:98) P n with abiased parameter initialization ψ 0 .
Training bias : the bias increment brought bysome unreasonable training strategies.
Formally, training bias is B ( f pl ( (cid:98) P n , ψ 0 , S ))
−
B ( f pl ( (cid:98) P n , ψ 0 ))
(yellow area in Fig. 4) wheref pl ( (cid:98) P n , ψ 0 , S ) is a pseudolabeler obtained withself-training strategy S .
Next we will introduce how to reduce training bias and data bias in self-training (red area in Fig. 4).
Debiased Self-Training
In SSL, we have a labeled dataset L = { ( x li , y li } n l i =1 of n l labeled samples and an unlabeled dataset
U = { ( x uj ) } n u j =1 of n u unlabeled samples, where the size of the labeled dataset is usually muchsmaller than that of the unlabeled dataset, i.e. , n l ≪ n u .
Denote ψ the feature generator, and h thetask-specific head.
The standard cross-entropy loss on weakly augmented labeled examples is
[Equation]
where α is the weak augmentation function.
Since there are few labeled samples, the feature generatorand the task-specific head will easily over-fit, and typical SSL methods use these pseudo labels onplenty of unlabeled data to decrease the generalization error.
Different SSL methods design differentpseudo labeling function (cid:98) f
Take FixMatch [44] for an instance.
FixMatch first generatespredictions (cid:98) p
= ( h ◦ ψ ◦ α )( x ) on a weakly augmented version of given unlabeled images, andadopts a confidence threshold τ to filter out unreliable pseudo labels
[Equation]
where (cid:98) f ψ,h refers to the pseudo labeling by model h ◦ ψ , hyperparameter τ specifies the thresholdabove which a pseudo label is retained and − 1 indicates that this pseudo label is ignored in training.
Then FixMatch utilizes selected pseudo labels to train on strongly augmented unlabeled images,
[Equation]
where (cid:98) f is a notation of general pseudo labeling function and A is the strong augmentation function.
As shown in Figure 5(a), the optimization objective for FixMatch is
[Equation]
where λ is the trade-off between the loss on labeled data and that on unlabeled data.
FixMatchfilters out low-confidence samples during the pseudo labeling process, yet two issues remain: (1) Thepseudo labels are generated and utilized by the same head, which leads to the training bias, i.e. , theerrors of the model might be amplified as the self-training progresses.
(2) When trained with extremefew labeled samples, the problem of unreliable pseudo labeling caused by data bias cannot be ignoredanymore even with the confidence threshold mechanism.
To tackle the above issues, we propose twoimportant designs to decrease training bias and data bias in Section 4.1 and 4.2 respectively.
4.1 Generate and utilize pseudo labels independently
The training bias of FixMatch stems from the way of training on the pseudo labels generated byitself.
To alleviate this bias, some methods turn to generate pseudo labels from a better teacher model,such as the moving average of the original model [48] in Figure 5(b) or the model obtained from theprevious round of training [55] in Figure 5(c), and then utilize these pseudo labels to train both thefeature generator ψ and the task-specific head h .
However, there is still a tight relationship betweenthe teacher model that generates pseudo labels and the student model that utilizes pseudo labels inthe above methods, and the decision hyperplanes of the student model h ◦ ψ strongly depend on thebiased pseudo labeling (cid:98) f .
As a result, training bias is still large in the self-training process.
[Figure]
T
TIndependent Model
Copied Model Unlabeled Data 𝐱 ! 𝐿 "# 𝐱 $ #𝒑 ! #𝒑 $ 𝐿 "# %y $ 𝐱 ! 𝐿 "# 𝐱 $ #𝒑 ! ’
𝒑 $ 𝐿 !"
# #𝒑 $ 𝐱 !
𝐿 "# 𝐱 $ #𝒑 ! ’
𝒑 $ 𝐿 %& #𝒑 $ 𝐱 !
𝐿 "# 𝐱 $ #𝒑 !
#𝒑 $ 𝐿 "# %y $
To further decrease the training bias when utilizing the pseudo labels, we optimize the task-specifichead h , only with the clean labels on L and without any unreliable pseudo labels from U .
To preventthe deep models from over-fitting to the few labeled samples, we still use pseudo labels, but onlyfor learning a better representation.
As shown in Figure 5(d), we introduce a pseudo head h pseudo ,which is connected to the feature generator ψ and only optimized with pseudo labels from U .
Thenthe training objective is
[Equation]
(5) where the pseudo labels are generated by head h and utilized by a completely parameter independentpseudo head h pseudo .
Although h and h pseudo are fed with features from the same backbone network,their parameters are independent, thus training the pseudo head h pseudo with some wrong pseudolabels will not accumulate the bias of head h directly in the iterative self-training process.
Note thatthe pseudo head h pseudo is only responsible for gradient backpropagation to the feature generator ψduring training and will be discarded during inference, and thus will introduce no inference cost.
4.2 Reduce generation of erroneous pseudo labels
Section 4.1 presents a solution to reduce the training bias, yet the data bias still exists in the pseudolabeling (cid:98) f .
As shown in Figure 6(a), due to the data bias, labeled samples of each class have differentdistances to the decision hyperplanes in the representation space, which leads to a deviation betweenthe learned hyperplanes and the real decision hyperplanes, especially when the size of labeled samplesis very small.
As a result, pseudo labeling (cid:98) f is very likely to generate incorrect pseudo labels onunlabeled data points that are close to these biased decision hyperplanes.
And our objective now is tooptimize the feature representations to reduce the data bias, and finally improve the quality of pseudolabels.
[Figure]
Since we have no labels for U , we cannot directly measure and thereby reduce data bias.
Yet trainingbias has some correlations with data bias.
Recall in Section 4.1, the task-specific head h is onlyoptimized with clean labeled data, since optimization with incorrect pseudo labels will push thelearned hyperplanes in a more biased direction and lead to the training bias.
Therefore, training biascan be considered as the accumulation of data bias with inappropriate utilization of pseudo labels,which is training algorithm dependent.
And the worst training bias that can be achieved by someself-training methods is a good measure of data bias.
Specifically, the worst training bias correspondsto the worst possible head h ′ learned by pseudo labeling, such that h ′ predicts correctly on all thelabeled samples L while making as many mistakes as possible on unlabeled data U , [Equation]
[Equation]
where the mistakes of h ′ on unlabeled data are estimated by its discrepancy with the current pseudolabeling function (cid:98) f .
Equation 6 aims to find the worst-case of task-specific head h that might belearned in the future when trained with pseudo labeling on the current feature generator ψ and thecurrent data sampling.
It is also the worst hyperplanes as shown in Figure 6(b), which deviates asmuch as possible from the currently learned hyperplanes while ensuring that all labeled samples arecorrectly distinguished.
Note that Equation 6 measures the degree of data bias, which depends on thefeature representations generated by ψ , thus we can adversarially optimize feature generator ψ toindirectly decrease the data bias,
[Equation]
As shown in Figure 6(c), Equation 7 encourages the feature of unlabeled samples to be distinguishedcorrectly even by the worst hyperplanes, i.e., be generated far away from the current hyperplanes,thereby reducing the data bias in feature representations. Overall loss.
The final objective of DST is to reduce both training bias and data bias.
The overallloss function simultaneously decouples the generation and utilization of pseudo-labels and avoids theworst-case hyperplanes.
This is achieved by unifying Equations 5–7 into a minimax game:
[Equation]
Experiments
Following [44, 56], we evaluate the proposed DST with random initialization on common SSLdatasets, including CIFAR-10 [27], CIFAR-100 [27], SVHN [33] and STL-10 [9].
Following [50], wealso evaluate DST with both supervised pre-trained models and unsupervised pre-trained models on 11downstream tasks, including (1) superordinate-level object classification: CIFAR-10 [27], CIFAR-100[27], Caltech-101 [15]; (2) fine-grained object classification: Food-101 [5], CUB-200-2011 [49],
Stanford Cars [26], FGVC Aircraft [31], OxfordIIIT Pets [37], Oxford Flowers [34]; (3) textureclassification: DTD [8]; (4) scene classification: SUN397 [53].
The complete training dataset sizeranges from 2 , 040 to 75 , 750 and the number of classes ranges from 10 to 397 .
Following [25], wereport mean accuracy per-class on Caltech-101 , FGVC Aircraft , OxfordIIIT Pets , Oxford Flowers ,and top-1 accuracy for other datasets.
Following [44], we construct a labeled subset with 4 labelsper category to verify the effectiveness of DST in extremely label-scarce settings.
To make a faircomparison, we keep the labeled subset for each dataset the same throughout our experiments.
For experiments with random initialization, we follow [44] and adopt Wide ResNet variants [58].
For experiments with pre-trained models, we adopt ResNet50 [21] with an input size of 224 × 224and pre-trained on ImageNet [12].
We adopt MoCo v2 [7] as unsupervised pre-trained models.
Wecompare our method with many state-of-the-art SSL methods, including Pseudo Label [29], Π -Model[28], Mean Teacher [48], VAT [32], ALI [14], RAT [47], UDA [54], MixMatch [3], ReMixMatch [2],
FixMatch [44], Dash [56], Self-Tuning [50], FlexMatch [59] and DebiasMatch [51].
When training from scratch, we adopt the same hyperparameters as FixMatch [44], with learningrate of 0 . 03 , mini-batch size of 512 .
For other experiments, we use SGD with momentum 0 .
andlearning rates in { 0 . 001 , 0 . 003 , 0 . 01 , 0 . 03 } .
The mini-batch size is set to 64 following [46].
For eachimage, we first apply random-resize-crop and then use RandAugment [10] for strong augmentation A and random-horizontal-flip for weak augmentation α .
More details on hyperparameter selectioncan be found in Appendix A.2.
Each experiment is repeated three times with different random seeds.
5.1 Main results
Table 1 shows that DST yields consistent improvement on all tasks.
On the challenging CIFAR100 and STL-10 tasks, DST boosts the accuracy of FixMatch and FlexMatch by 8.3% and 10.7% ,respectively.
Figure 7 depicts the top-1 accuracy during the training procedure on CIFAR-100 .
Weobserve that the performance of FixMatch suffers from significant fluctuations during training.
Incontrast, the accuracy of DST (FixMatch) increases steadily and surpasses the best accuracy of
FixMatch by 10.9% , relatively.
Note that the accuracy of FlexMatch also drops by over 6% inthe final stages of training while DST (FlexMatch) suffers from a much smaller drop by reducingerroneous pseudo labels during the self-training process.
Besides, DST also improves the performancebalance across categories (see Appendix B.2).
[Table]
[Figure]
[Figure]
5.2 Transfer from a pre-trained model
Supervisied pre-training.
Table 2 reveals that typical self-training methods, e [Figure] g. FixMatch, lead torelatively mild improvements with supervised pre-trained models, which is consistent with previousfindings [46, 50].
In contrast, incorporating DST into FixMatch significantly boosts the performance [Table] lt [Table]
C
I F
A
RC
I F A
RS UN
[Table]
D T D
A i r c r a f t
C U B
F l o w e r s P e t s C a r s F
[Table]
A v e r a g e S up e r [Table] and surpasses FixMatch by 19.9% on all datasets.
With a pre-trained model, self-training has bettertraining stability.
Yet once the performance degradation occurs, the process is also irreversible(Appendix B.1), partly due to the catastrophic forgetting of pre-trained representation.
Also, selftraining suffers from a more severe performance imbalance across classes (Appendix B.2).
DSTeffectively tackles these issues, indicating the importance of reducing bias.
Unsupervised pre-training.
Table 2 shows that with unsupervised pre-trained models, more methodssuffer from performance degradation after self-training on the unlabeled data.
The difficulty comesfrom that the unsupervised pre-training task has a larger task discrepancy with the downstreamclassification tasks than the supervised pre-training task.
Thus, the representations learned byunsupervised pre-trained models usually exhibit stronger data bias, and inappropriate usage of pseudolabels will lead to rapid accumulation errors and increase the training bias.
By eliminating training biasand reducing data bias, DST brings improvement on all datasets and relatively outperforms FixMatchby 23.5% on average, superior to FlexMatch and DebiasMatch in 9 and 10 tasks, respectively.
5.3 Ablation studies
We examine the design of our method on CIFAR-100 in Table 3 and have the following findings.
Compared with Mutual Learning [60, 16], where two heads provide pseudo labels to each other, theindependent mechanism in our method where one head is only responsible for generating pseudolabels and the other head only uses them can better reduce the training bias.
(2) A nonlinear pseudohead is always better than a linear pseudo head.
We conjecture that nonlinear projection can reduce [Table] the degeneration of representation with biased pseudo labels.
(3) The worst-case estimation of pseudolabeling improves the performance by large margins.
5.4 Analysis
To further investigate how DST improves pseudo labeling and self-training performance, we conductsome analysis on CIFAR-100 .
For simplicity, we only give the results with supervised pre-trainedmodels.
More comparisons can be found in Appendix B.4.
[Figure]
DST improves both the quantity and quality of pseudo labels.
As shown in Figure 8(a) and 8(b),
FixMatch exploits unlabeled data aggressively , on average producing more than 70% pseudo labelsduring training.
But the cost is that the accuracy of pseudo labels continues to drop, eventually fallingbelow 60% , which is consistent with our motivation in Section 3 that inappropriate utilization ofpseudo labels will in turn enlarges the training bias.
On the contrary, the accuracy of pseudo labels in
DST suffers from a smaller drop.
Rather, it keeps rising afterward and exceeds 70% throughout thetraining.
Besides, DST generates more pseudo labels in the later stages of training.
DST generates better pseudo labels for poorly-behaved classes.
To measure the quantity ofpseudo labels on poorly-behaved classes, we calculate the class imbalance ratio I on a class-balancedvalidation set, I = max c N ( c ) / min c ′ N ( c ′ ) , where N ( c ) denotes the number of predictions that fallinto category c .
As shown in Figure 8(c), the class imbalance ratio of FixMatch rises rapidly andreaches infinity after 5000 iterations, indicating that the model completely ignores those poorlylearned classes.
To measure the quality of pseudo labels on poorly-behaved classes, we calculatethe average accuracy of 10 or 20 worst-behaved classes in Figure 8(d).
The average accuracy on theworst 20 classes of FixMatch is only 1.0% .
By reducing training bias with the pseudo head, data biaswith the worst-case estimation, the average accuracy balloons to 28.5% and 34.5% , respectively.
5.5 DST as a general add-on
[Table]
[Table] We incorporate DST into several representative self-training methods, including FixMatch[44], Mean Teacher [48], Noisy Student [55], DivideMix [30] and FlexMatch [59].
Implementation details of DST versions of these methodscan be found in Appendix A.3.
Table 4 compares the original and DST versions of thesemethods on CIFAR-100 with both supervisedpre-trained and unsupervised pre-trained models.
Results show that the proposed DST yields largeimprovement on all these self-training methods,indicating that self-training bias widely existsin existing self-training methods and DST canserve as a universal add-on to reduce bias.
Conclusion
To mitigate the requirement for labeled data, pseudo labels are widely used on the unlabeled data, yetthey suffer from severe confirmation bias.
In this paper, we systematically delved into the bias issuesand present DST , a novel approach to decrease bias in self-training.
Experimentally, DST achievesstate-of-the-art performance on 13 tasks and can serve as a universal add-on.