Reimplementation of FixMatch and Investigation on Noisy (Pseudo) Labels and Conﬁrmation Errors of FixMatch
Anonymous Author(s)
AfﬁliationAddress email
Reproducibility Summary
Scope of
Reproducibility
The main objective of this work is to conﬁrm the effectiveness of
FixMatch (Sohn et al.
[2020]), which combines pseudo labelling and consistency regularization in semi-supervised learning (SSL) tasks, by achieving similar results on CIFAR-10 and demonstrating the key success of FixMatch via ablation studies.
Furthermore, we also investigated the the existence of conﬁrmation errors in FixMatch by reconstructing the batch structure during the training process
Methodology.
All the experiments in this work were conducted on CIFAR
10 using the samenetwork architecture, Wide
ResNet28-
A single V100 is used for each experiment with an averagetraining time of 70 hours.
We re-implemented FixMatch mainly based on the paper using Pytorch andrefer to the ofﬁcial implementation (in Tensorﬂow) for details and replicated similar results shownin the second-last row of Table of column CIFAR-
10 in
Sohn et al.
Ablation studies werefocused on two key factors of FixMatch ratio of unlabeled data and conﬁdence threshold, as shownin Figure 3 (a
& (b in Sohn et al.
Results.
Compared with the average error rate reported in Table 2 in Sohn et al.
[2020], ourimplementation achieves similar error rates by 3 . 77 lower on
CIFAR- with labels
22 higheron
CIFAR- with labels
, and 0 .
1 higher on
CIFAR- with 4000 labels.
Thus it is supportedthat FixMatch outperforms semi superivesed learning benchmarks.
And the results of ablation studiesexhibit almost the same trends as Figure 3 (a) & (b) show in the paper, which demonstrated that theauthor’s choices with respect to those ablations were experimentally sound.
We also conﬁrmed theexistence of conﬁrmation errors in pseudo labels by checking the confusion matrix of the predictionof unlabeled data in different training stages.
What was easy.
It is generally easy to re-implement FixMatch given all the experimental settingsin the paper, with key parameters clearly stated in each experimental section and detailed lists ofhyperparameters in appendix.
Compared with CTAugment, RandAugment is relatively easy toimplement since it requires no parameters tuning during training and coefﬁcients representing theseverity of all distortions are given in appendix.
Besides, it converges faster than CTAugment.
What was difﬁcult.
The ofﬁcial implementation is complicated thus not easy to follow.
And thereare some details missing in the paper compared to the code: 1.
the ofﬁcial implementation actuallyuse leaky ReLU instead of ReLU for ResNet; 2.
Exponential moving average is only mentioned forexperiments on ImageNet but actually also used on CIFAR-10; 3.
the details on how to update theweights of the magnitude bins of CTAugment are not given in the paper, and our implementationachieves a slightly worse results than the average error rate reported (1.14 higher with 250 labels).
Communication with original authors.
All the confused parts mentioned in the previous sectionare clariﬁed by the original authors via email and in the issues of their Github repository.
Abstract
FixMatch is a semi-supervised learning method, which achieves comparable resultswith fully supervised learning by leveraging a limited number of labeled data(pseudo labelling technique) and taking a good use of the unlabeled data (consistency regularization ).
In this work, we reimplement FixMatch and achieve reasonably comparable performance with the ofﬁcial implementation, which supportsthat FixMatch outperforms semi-superivesed learning benchmarks and demonstrates that the author’s choices with respect to those ablations were experimentallysound.
Next, we investigate the existence of a major problem of FixMatch, conﬁrmation errors , by reconstructing the batch structure during the training process.
It reveals existing conﬁrmation errors, especially the ones caused by asymmetric noise in pseudo labels.
To deal with the problem, we apply equal-frequencyand conﬁdence entropy regularization to the labeled data and add them in theloss function.
Our experimental results on CIFAR-10 show that using either ofthe entropy regularization in the loss function can reduce the asymmetric noisein pseudo labels and improve the performance of FixMatch in the presence of(pseudo) labels containing (asymmetric) noise.
Our code is available at the url:https://github.com/Celiali/FixMatch .
Introduction
Ghahramani [2020] summarized the reasons for the success of deep learning in his talk given as thechief scientist in Uber.
Firstly, with the availability of large datasets, large models can work well.
Secondly, training such large models with stochastic descent works surprisingly well.
Moreover,staying close to identity (such as ReLU) makes it stable to be trained.
The automate differentiationand a large number of open source softwares make it scale well.
Therefore, we can see deep learningin many applications, such as computer vision, natural language processing, bioinformatics etc.
However, it is not always the case where a huge number of labeled data are available.
In someareas, it is difﬁcult, expensive or even impossible to have a large labeled dataset, such as medicalimages [Kuznetsova et al., 2018].
In this case, it can be difﬁcult to train a Deep Neural Network(DNN) from scratch with the limited labeled data.
Luckily, Tajbakhsh et al.
[2016] shows that
a
DNN trained based on a pre-trained DNN, ﬁne-tuning, can outperform the one trained from scratch.
Moreover, Semi-Supervised Learning (SSL) is also a common method to deal with the scarcity andoften high acquisition cost of labelled data [von Kügelgen et al., 2020].
SSL efﬁciently leverageslabeled data and the relation with unlabeled data to train a DNN.
Among SSL methods, there is aclass of "match"-based methods, such as FixMatch [Sohn et al., 2020], MixMatch [Berthelot et al.
,2019], ReMixMatch [Kurakin et al., 2020] and DivideMatch [Li et al., 2020].
These methods utilizethe consistency regularization, pseudo-labelling and ensembling methods to boost the performancewith the use of unlabeled data.
In fact, they are leveraging prior knowledge to regularize the trainingof DNNs.
In this project, we focus on reproducing and investigating one of such methods, FixMatch[Sohn et al., 2020].
Nevertheless, SSL is still facing many challenges in theory and in practice.
Ben-David et al.
show that“as long as one does not make any assumptions about the behavior of the labels, SSL cannot help muchover algorithms that ignore the unlabeled data.”
Moreover, SSL can actually degrade performance ifcertain assumptions are not met [Chapelle et al., 2010].
In this line of works, Schölkopf et al.
[2012]consider the problem from a causal modeling perspective and conclude that in fact SSL is impossiblewhen predicting a target variable from its causes (causal learning) but possible from anti-causallearning.
Recently, the relation of causality and semi-supervised learning is further explored in [von
Kügelgen et al., 2020], i.e., predicting a target variable from both causes and effects at the sametime.
Moreover, in the light of consistency regularization and pseudo-labelling, a signiﬁcant issue ofthe "Match"-based methods is conﬁrmation error .
It happens especially when noisy samples are inthe labeled set.
A DNN can keep having lower loss by ﬁtting the noise and be further maintainedafter training with the wrong pseudo labels of unlabeled data , which keeps the errors in the modeland limits its generalization and performance [Tarvainen and Valpola].
This problem becomes moreserious in the presence of asymmetric noise in the training labels, which roughly speaking tends tolabel a class of data as another speciﬁc class.
Therefore, in this work, we are not only reimplementing FixMatch, but also investigating whether thepseudo labels made by the DNN contain harmful noise leading to conﬁrmation errors.
First, wedesign a stable and reliable method to examine the existence of conﬁrmation errors and noisy pseudolabels by reconstructing the batch structure.
Secondly, we ﬁnd methods to deal with (asymmetric)noise in (pseudo) labels of the training dataset.
We reconstruct the batch structure and add anequal-frequency entropy regularization on labeled data to the loss function of FixMatch.
Moreover,we also use a conﬁdence entropy regularization on labeled data to avoid the over-conﬁdent prediction.
It turns out that both entropy regularization is helpful for dealing with the noisy (pseudo) labels (evenfor the asymmetric noise) and conﬁrmation errors.
Our experimental results show that
our implementation can achieve almost the same performance even better for low-label
regimes.2.
there exists asymmetric noise in the pseudo labels leading to conﬁrmation errors.
With suchpseudo labels, the model is biased which in turn leads to more asymmetric noise in pseudolabels.3.
FixMatch with equal-frequency entropy regularization and FixMatch with conﬁdence en-
tropy regularization can reduce (asymmetric) noise in the pseudo labels and perform better
than the baseline FixMatch in the presence of asymmetric noise in (pseudo) labels .
Related work
As introduced in Sec. 1, conﬁrmation error is a serious issue of "Match"-based SSL methods and ourstudy is mainly about the conﬁrmation error and FixMatch in the presence of noisy (pseudo) labels.
Therefore, here we mainly introduce the noisy labeling and some related works for dealing with thenoisy label and conﬁrmation error in SSL.
Noisy labeling and noise-robust loss.
Suppose a dataset D = { ( x i , y i ) } ni =1 where y i is given bynoisy labeling.
To model noisy labeling process, we have p ( y i | (cid:101) y i ) where (cid:101)
y i is the ground truth labelunder the assumption that the noise label is conditionally independent from the input data given theground-true label; formally, p ( y i = k
| x i , (cid:101) y
i =
j )
=
p ( y i =
k | (cid:101) y i = j ) =
η kj .
In general, such noiseis called class dependent, which is also named as the asymmetric noise[Zhang and Sabuncu, 2018].
In contrary, when η kj = η , it is called symmetric noise.
Under the symmetric noise assumption,
Ghosh et al.
[2015] studied the functional form of loss function and concluded that by using thesymmetric loss function, one can get a global optima such that the learned model is noise tolerant.
For example, the MAE loss function is a symmetric function while the cross entropy loss functionis not.
However, using MAE loss function has poor accuracy performance on classiﬁcation taskscompared with the cross entropy loss function [Zhang and Sabuncu, 2018].
One can convinceoneself with Eqn.
(5) in [Zhang and Sabuncu, 2018], i.e., the cross entropy loss function enablesthe optimization process weighting the sample importance while the MAE loss function considerssamples equally.
Furthermore, Zhang and Sabuncu [2018] combine MAE and cross entropy lossfunctions with L’Hôpital’s rule, i.e.,
[Equation]
where f ( x ) is the model, j indexes the class, and f j ( x ) is the softmax output of j .
Interestingly,when q = 1 , L q ( f ( x ) , j ) is a MAE loss function; while lim q → 0 L q ( f ( x ) , j ) is a cross entropy loss.
Therefore, one can manipulate trade off by selecting a good hyper-parameter q .
Furthermore, italso introduces a better loss function, the truncated L q ( f ( x ) , j ) , which is essentially a practicallyimproved version of L q ( f ( x ) , j ) .
However, in theory the proposed method is based on the symmetricnoise assumption [Zhang and Sabuncu, 2018], which can be quite easy to be violated.
This is atrade-off between using a stricter assumption and estimating noisy labelling mechanisms [Patriniet al., 2017]
(which is a challenge).
SSL for noisy labeling and a potential solution for asymmetric noise.
Li et al.
[2020] considerthe noisy label problem as a semi-supervised learning problem by ﬁnding the similarity of unlabeledsamples in semi-supervised learning and noisy labels.
Suppose that we can successfully separate thenoisy and clean samples, we can treat the noisy ones as unlabeled data in semi-supervised learning,and then leverage the success of semi-supervised learning to tackle the noisy labeling problem.
Firstly, by observing that the loss of clean samples tends to be lower than the noisy ones [Arazo et al.
,2019], Li et al.
ﬁt a Gaussian Mixture Model for the two components, the noisy group andthe clean one.
Then given a loss, it can be inferred whether the sample is a noisy one or a cleanone.
Consequently, following the mentioned idea, semi-supervised learning methods are appliedto such a separated dataset.
Moreover, Li et al.
[2020] consider the inﬂuence of asymmetric noisein the supervised learning phase.
Because the bias introduced by the asymmetric noise can leadto severe consequences (conﬁrmation errors).
[Li et al., 2020] added a negative entropy penaltyterm −H =
(cid:80) j f j ( x ) log f j ( x ) for an input x in the cross-entropy loss function at the beginningof training to avoid over-conﬁdent prediction, which works well emperically.
To further reducethe inﬂuence of the conﬁrmation error introduced by the symmetric noise, it uses the MixMatch[Berthelot et al., 2019] procedure to train two independent DNNs and attractively exchange datasetswith each other for ﬁltering errors made by the other one.
This is actually an ensemble method, whichreduces the random noise in the prediction, especially in the presence of symmetric labelling noise.
Model bias in SSL.
Kurakin et al.
[2020] propose a distribution alignment method utilizing aprinciple introduced by Bridle et al.
It formulates an ideal classiﬁer which maximizes themutual information of model inputs and model outputs.
Furthermore, it argues that the second termof mutual information encourages a model to output with low entropy and high conﬁdence, whileanother one encourages equal frequency across the entire training set as shown in
[Equation]
(2) where θ is the model parameters.
As what Kurakin et al.
[2020] said, when the marginal distributionof a training dataset labels is not uniformly distributed, it is not proper to regularize the frequency.
Inour work, to deal with such case, we augment the training dataset and make the labels of labeled datain each batches to be uniformly distributed.
Methods
3.1 FixMatch
As one of the SSL methods, FixMatch [Sohn et al., 2020] leverages labeled data and introduces priorknowledge about unlabeled data in the training process.
For labeled data, FixMatch simply uses thecross entropy loss function for a batch,
[Equation]
[Equation]
where B is the number of labeled data in a batch, x b is a labeled sample, y b is the label, and α ( · ) isweak augmentation.
However, due to limited number of labeled samples, the performance of such
DNN is not ideal.
Therefore, the question is how to make a good use of the sufﬁcient unlabeled datato improve the performance?
Ideally, the performance can be close to the DNN trained with the fullylabeled dataset.
FixMatch considers the consistency of model prediction on the unlabeled data with weak and strongaugmentation (the augmentation methods are introduced in Sec. 4).
It ﬁrst uses the model to predictpseudo labels for unlabeled data and then compute the loss of unlabeled data with the pseudo labelsand the consistency regularization.
The loss function for the unlabeled samples u b is
[Equation] [Equation]
where µB is the number of unlabeled data in a batch, ˆ y b := arg max y p ( y | α ( u b ); θ f ) is the pseudolabel of u b , θ f is the neural network parameters of function f , and A ( · ) is the strong augmentation.
Note that to make pseudo labels reliable to be used, FixMatch considers the pseudo labels in the lossfunction only if the prediction has a higher probability than τ .
Next, together with the cross entropyloss of labeled data, the loss function of FixMatch is l s + λ u l u .
3.2 Investigation of noisy (pseudo) labels and conﬁrmation errors of FixMatch
Nosiy pseudo labels and conﬁrmation errors in FixMatch.
A main issue of "Match"-based SSLmethods is conﬁrmation errors.
Since FixMatch is trained on batches with both labeled and unlabeleddata, it is very likely to make prediction errors at the beginning of the training.
When the modelmakes wrong predictions of labeled data, since we have their ground-truth labels, the model canbecome better with the loss for labeled data.
But when it comes to unlabeled samples, since we don’thave the ground-truth labels, the model uses the conﬁdent pseudo labels as the labels for training.
In this case, if the pseudo-labels are noisy, the model can ﬁt such errors and become biased.
Inthe next batch, it can generate more wrong pseudo-labels with higher conﬁdence.
Moreover, theconsistency regularization can keep reinforcing the model to ﬁt such wrongly labeled data.
Finally, itdemonstrates a biased model with a poor performance on generalization and robustness.
Therefore,noise in the pseudo labels can lead to conﬁrmation errors in FixMatch.
Both asymmetric noise and symmetric noise in pseudo labels can lead to conﬁrmation errors, but ingeneral asymmetric noise is more harmful and harder to deal with.
For example, to reduce the impactof symmetric noise and get an unbiased model, one can use ensembling methods like [Li et al., 2020]to train multiple DNNs at the same time; however, this can fail in the presence of asymmetric noise.
In this work, we focus on asymmetric noise and one can simply extend the method to deal with theinﬂuence of symmetric noise with ensemble methods.
Investigation with class-balanced batches.
To check whether there exist conﬁrmation errors, weneed to check that during the training process errors are reinforced by the model.
Moreover, tosee the asymmetric noise in the pseudo labels, we need to check that in the training phase whether
FixMatch predicts a certain class of unlabeled data into certain other classes.
Thus, these require usto investigate the performance of FixMatch at each batch and check the pseudo labelling performanceregarding asymmetric noise in the pseudo labels.
However, in [Sohn et al., 2020], a batch is notnecessary to contain all the classes of training dataset and it can contain different classes with differentnumbers.
Therefore, the performance of pseudo labelling regarding asymmetric noise inherits therandomness of batch composition, which makes the investigation conclusion unreliable.
To deal with this issue, we reconstructed the batch structure which requires each batch to contain anequal number of images for all the classes on both labeled and unlabeled data, called Balanced-Class(BC) batches.
With such batches, we can fairly check the performance of pseudo labelling in eachbatch how many errors are made when the model predicts each class and whether it tends to labela class as other certain classes causing asymmetric noise.
Note that without further introducingregularization, BC batches on their own cannot improve the performance of FixMatch, which hasindistinguishable results without BC as shown in Sec.
Furthermore, we leverage the reconstructed batch structure to regularize the training process forreducing the noise in pseudo labels and improving the performance.
With the reconstructed batches,we know that the class of labeled data 1 is uniformly distributed, thus we can regularize the outputof labeled data with the negative entropy loss of the prediction frequency.
In this way we force theoutput of labeled data to be uniformly distributed.
Potentially this can regularize the asymmetric noisein the labeled data because the output class distribution is not likely to be uniformly distributed in thepresence of asymmetric noise.
Consequently, it can reduce the asymmetric noise in pseudo labelsbecause the prediction on both labeled and unlabeled data uses the same network which is unlikely tohave different prediction behavior.
Therefore, we add an equal-frequency entropy regularization tothe loss function, which is
[Equation]
[Equation]
where c is the number of classes and λ ef is a hyperparameter.
We also consider the conﬁdenceentropy loss regularization which can avoid over-conﬁdent prediction,
[Equation]
Note that since the loss function (6) aims for avoiding over-conﬁdent predictions, it seems to be ﬁne to regularize the unlabeled data as well.
However, we cannot do that for the same reason as the loss function (5) which has been discussed in the footnote.
Because −H ( · ) is a convex function, we have the Jensen’s inequality
[Equation]
In other words, conﬁdence entropy regularization can implicitly regularize the equal frequency of thedata labels.
Therefore, with the same reason, we should only apply it to the labeled data of whichlabel distribution is under our control with augmentation.
Data Preprocessing and Augmentation
FixMatch requires a weak augmentation α ( · ) and a strong augmentation A ( · ) .
For the weak augmentation, we randomly ﬂip an image with probability 0 .
5 as [Sohn et al., 2020] and translate an imageup to 12 .
5% with probability 0 .
For the strong augmentation, FixMatch uses either RandAugment(RA)
[Cubuk et al., 2020] or CTAugment [Kurakin et al., 2020] for their experiments.
However, weuse RA for our experiments with the maximum magnitude 10 (same as the ofﬁcial experiment setup)and 2 randomly selected operations per image.
Due to the limitation of computational resources, we examine the reproducibility of [Sohn et al.
,2020] on the dataset CIFAR-10 [Krizhevsky et al., 2009].
In CIFAR-10, there are 50000 trainingdata and 10000 testing data.
We take 5000 training data as the validation dataset.
Then we use theremaining training dataset to make labeled and unlabeled datasets and augment both datasets into thesame target number as in [Sohn et al., 2020].
After augmentation, we have 2 13 labeled images and2 13 × 7 unlabeled images for the CIFAR-10 training dataset.
Experiment
In the reproducibility experiments, we re-implement FixMatch from scratch using PyTorch andreproduce the essential experiments in the original paper with the similar results.
We use thehyperparameters ( λ u = 1 , η = 0 .
03 , β = 0 .
9 , τ = 0 .
95 , µ = 7 , B = 64 , K = 2 20 ) given by[Sohn et al., 2020] and focus on reproducing the performance on CIFAR-10 (Sec. 4.1 of [Sohn et al.
,2020]) and barely supervised learning experiments (Sec. 4.4 of [Sohn et al., 2020]).
Besides the earlyintroduced hyper-parameters, we use SGD with β = 0 .
9 for training the model, and the learningrate is decay with η cos( 7 πk 16 K ) , where K is the total time step and k is the current time step.
Eachexperiment takes around 68 hours on a single V100.
And all the error rates is generated from EMA(exponential moving average) of models in the SGD training trajectory.
Then, we investigate conﬁrmation errors of "Match"-based SSL methods to see whether there existssuch error and asymmetric noise of pseudo labels in FixMatch with the ofﬁcial experiment setup, i.e.unbalanced batches, in [Sohn et al., 2020].
Next, we examine the existence of conﬁrmation errorsand asymmetric noise for FixMatch again in a more reliable way using re-constructed batches asintroduced in Sec. 3.
Furthermore, we respectively add the equal-frequency entropy regularizationand conﬁdence entropy regularization on the labeled training data in the loss function and comparewith the baseline FixMatch without entropy regularization on the BC batches.
Finally, we addasymmetric noise to the labeled data in the training dataset and compare the performance of baseline
FixMatch and FixMatch with different entropy regularization.
5.1 Reproducibility
CIFAR-10.
We reproduced the experiments on CIFAR-10 with 40, 250, 4000 labeled data and5000 validation samples as the ofﬁcial implementation of FixMatch 3 .
But due to the limitation ofcomputational resources, we didn’t reproduce 5 "folds".
Thus, our result based on 1 fold doesn’t havethe standard deviation.
Our model uses the Wide ResNet-28-2 [Zagoruyko and Komodakis, 2016]with leaky ReLU activation function.
Our results are shown in Table 1 which is comparable to theperformance in [Sohn et al., 2020].
[Table]
Barely supervised learning.
We also reproduce the one example per class experiment.
[Sohnet al., 2020] hypothesize that the repressiveness of the chosen labeled data inﬂuences the resultssigniﬁcantly.
Since there are only one/few samples per class, this hypothesis is reasonable intuitively.
Then, Sohn et al.
[2020] categorized the training dataset into eight levels of “prototypicality”, i.e.,representative of the underlying class and then ordered the training samples by their “prototypicality”.
With the same hyperparameters, the model is trained with 10 provided most representative labeleddata under Random Augment.
The accuracy is 84 .
41% compared with the ofﬁcial implementation: amedian of 78% accuracy and a maximum of 84% accuracy.
5.2 Ablation studies
The ablation studies are based on FixMatch with 250 labels using CTAugment.
Study for Conﬁdence threshold.
We performance the ablation studies for conﬁdence threshold.
Due to the limited computation resource, we hypothesize that experiments with lower conﬁdencethreshold will achieve worse performance and explore more values around the optimal value ofconﬁdence threshold, 0.95 chosen by the authors.
Thus our examined threshold value is between0.7 to 0.98.
As shown in Figure1 (c), the error rate is between 6 . 54% and 6 .
and the highestperformance is under the threshold 0.98.
Ratio of unlabeled data.
We perform FixMatch under different ratios of unlabeled data.
Figure1(d) shows the error rate which is decreasing when the ratio of unlabeled data is higher.
A signiﬁcantincrease of the accuracy happens using a large number of unlabeled data.
The results show theconsistency with the ﬁnding in the original paper.
5.3 Investigation on conﬁrmation errors and asymmetric noisy (pseudo) labels
In this section, we show the investigation on conﬁrmation errors and asymmetric noise in labels andpseudo labels and whether the entropy regularization in loss functions (5) and (6) can deal with themand improve the performance of FixMatch.
The training dataset contains 150 labeled data beforeaugmentation and each BC batch in the training phase contains images with uniformly distributedclasses.
Existence of asymmetric noise and conﬁrmation errors in pseudo labels.
We examine theexistence of asymmetric noise in pseudo labels by checking the confusion matrix of the predictionof unlabeled data in different batches.
Top ﬁgures of Figure 2 show the confusion matrices in theexperiments without using BC batches.
We ﬁnd that asymmetric noise appears in a random manner,which is as our expectation as analyzed in Sec. 3.
The stochastic behavior is inherited from the a valid randomness of batch composition.
Next, we evaluate the asymmetric noise with BC batches, whichis a more reliable way as mentioned in Sec. 3.
We found that there exists consistent asymmetricnoise, which leads to the conﬁrmation errors, i.e., the model always tends to wrongly predict certainimages into certain classes as shown in bottom ﬁgures of Figure 2.
Moreover, the accuracy of ourimplementation is 93 .
6% without BC batches and 93 .
8% with BC batches, which shows that using
BC batches has rarely inﬂuence on the model performance compared with the one without BCbatches.
Equal-frequency and conﬁdence entropy regularization on the labeled data.
Due to limitationof the computational resources, we didn’t explicitly run grid search for the hyperparameters in the
Equal-Frequency (EF) loss function (5) and Conﬁdence-Entropy (CE) loss function (6).
Instead,we found that for the baseline method the training loss is around 0.2.
We then compute the equal- frequency entropy loss for the ideal scenario, equal frequency for all classes, which is 0 .
1 × ln 0 .
1 ≈ 2 .
We decide to try the hyperparameter λ ce , λ ef ≤ 0 .
1 to avoid making the entropy regularization lossdominate the loss value.
Then, we do a hyper-parameter search for the loss function (5) and (6).
Forall experiments in this experiment, we used cosine function decay for the parameters λ ce and λ eq ,which starts with value 1 and ends with value 0 in the training phase.
We ﬁnd that using the lossfunction (6) can achieve a better accuracy performance 94 .
Moreover, as an advantage, using theconﬁdence entropy regularization can reduce the asymmetric noise as shown in the bottom confusionmatrices of Figure 3.
As for the equal-frequency entropy regularization, it has a better accuracy,93 . 85% .
Moreover, the equal-frequency entropy regularization can penalize the asymmetric noise,which may transform it into symmetric noise as shown in the middle confusion matrices of Figure 3.
Note that there are plenty of ways to deal with symmetric noise, which is much easier to handle.
[Table]
Equal-frequency and conﬁdence entropy regularization on the labeled data containing asymmetric noise.
In this experiment, we use RA data augmentation and manually add asymmetricnoise to the labeled data in the training dataset to compare how FixMatch with different loss functionsperforms in the presence of asymmetric noise in the labeled data.
We respectively select 3 imagesfrom class 0 and class 1 in the validation dataset.
Then, for the labeled data in the training dataset,we keep the labels unchanged and replace 3 images in class 2 with the 3 images in class 0 .
Similarlywe replace 3 images in class 3 with the 3 images in class 1 .
In this way, the only difference withthe previous experiments in this section is that our ﬁnal validation dataset has 4994 images and thelabeled data in the training dataset contain asymmetric noise.
Table 3 shows error rates on 6 runs withdifferent random seeds. In the presence of asymmetric noise in labeled training data, all proposedmethods perform better than the baseline method, in which FixMatch with BC batches decreases theaverage error rate from 8.6 to 7.37, and the combination of conﬁdence-entropy regularization and BCbatches further lowers the error rate to 6.98.
Challenges
It is not clear how many steps are there in each epoch.
First the paper only states the total steps
K = 2 20 and the composition of one batch ( B labeled samples and µB unlabeled samples).
And theofﬁcial code indicates there are 2 16 labeled images observed by the model per epoch and a total of [Table] [Table] 2 26 images observed which suggests that there are 2 12 updates per epoch and 2 19 updates in total.
And this is not consistent with the total update steps K stated in the paper.
When performing weakaugmentations to the input data, the probability for randomly translating images is not speciﬁed.
Andit also remains unclear the ‘5 different folds’ mentioned in the paper, we are guessing it is a kind ofcross validation while there is not too much evidence supporting this neither in the paper nor in theofﬁcial code.
The paper doesn’t contain sufﬁcient details to reproduce all the experiments.
Thus, it is necessary tolook for details about reproducing the experiments in the ofﬁcial code.
We have not optimized ortuned the hyperparameters, and all the hyperparameters are the same as those mentioned in the paper.
Compared to the average error rates in the original paper, the reproduced results have a reasonablegood performance on a larger number of labeled data (4000/250 labels) and better but also reasonableperformance on fewer labeled data (40/10 labels) since the variance of error rates over 5 differentfolds for CIFAR-10 with 40 labels is 3 .
Moreover, to compare with the results of ablation studiesin the original paper, we also implement CTAugment, which supports a learnable magnitude.
Whilewe failed to conﬁrm the result that CTAugment behaves better than RandAugment on CIFAR-10.
Wehypothetically guess this is because it could affect the consistency regularization because of differentlevels of distortions controlled by magnitude.
Conclusion
In this work, we study and reimplement FixMatch from scratch.
We reproduced essential experiments,included the model performance on CIFAR 10, barely supervised learning, and ablation studies.
Experimental results show that our implementation achieves similar performance as the original
FixMatch results, which supports that FixMatch outperforms semi-superivesed learning benchmarksand that the author’s choices with respect to those ablations were experimentally sound.
We alsoconﬁrmed the existence of conﬁrmation errors in pseudo labels by checking the prediction confusionmatrix of unlabeled data in different training stages.
We adapted the training batch structure to becomposed of equal number of images in each class, which enable us to stably and reliably check thethe asymmetric noise in the training phase.
Based on the reconstructed batch structure, we used theequal-frequency and conﬁdence entropy regularization in the loss function, and theoretically showtheir relation.
The experiments indicate that these entropy regularization can reduce the asymmetricnoise in pseudo labels and improves the performance of FixMatch in the presence of training labelswith asymmetric noise.
Ethical consideration
The bias in the collected dataset is a serious problem when applying machine learning methods tothe real-world scenarios.
For example, applying machine learning methods to making automateddecision-making systems for criminal prediction, university admission or recruitment.
In these cases,we may very likely collect a dataset containing certain bias due to the historical reason or selectionbias in the data collection process.
If a model cannot deal with such bias in the dataset, it may inheritin the model by focusing on the unrelated or wrong relations in the dataset.
Consequently, the modelcan make biased decision which can disadvantage a certain group of people and may even diminishthis group in the society.
Unfortunately, FixMatch cannot only be inﬂuenced by the noise in the label of a training dataset, butalso it can make conﬁrmation errors causing a biased model even when the dataset itself is unbiased.
To deal with such issue, this work focuses on the asymmetric noise in the data labels and pseudo labels, which can lead to severe conﬁrmation error and the biased model.
And then, we applieddifferent methods to reduce such noise in pseudo labels and reduce its impact on the model.