Don’t fear the unlabelled: Safe semi-supervised learning via simple debiasing
Anonymous Author(s)
AffiliationAddress email
Abstract
Semi-supervised learning (SSL) provides an effective means of leveraging unlabelled data to improve a model’s performance.
Even though the domain hasreceived a considerable amount of attention in the past years, most methods presentthe common drawback of lacking theoretical guarantees.
Our starting point is tonotice that the estimate of the risk that most discriminative SSL methods minimiseis biased, even asymptotically.
This bias impedes the use of standard statisticallearning theory and can hurt empirical performance.
We propose a simple way ofremoving the bias.
Our debiasing approach is straightforward to implement andapplicable to most deep SSL methods.
We provide simple theoretical guarantees onthe trustworthiness of these modified methods, without having to rely on the strongassumptions on the data distribution that SSL theory usually requires.
In particular,we provide generalisation error bounds for the proposed methods.
We evaluatedebiased versions of different existing SSL methods, such as the Pseudo-labelmethod and Fixmatch, and show that debiasing can compete with classic deep SSLtechniques in various settings by providing better calibrated models.
Additionally,we provide a theoretical explanation of the intuition of the popular SSL methods.
Introduction
The promise of semi-supervised learning (SSL) is to be able to learn powerful predictive modelsusing partially labelled data.
In turn, this would allow machine learning to be less dependent onthe often costly and sometimes dangerously biased task of labelling data.
Early SSL approaches—e.g. Scudder’s (1965) untaught pattern recognition machine—simply replaced unknown labels bypredictions made by some estimate of the predictive model and used the obtained pseudo-labels torefine their initial estimate.
Other more complex branches of SSL have been explored since, notablyusing generative models (from McLachlan, 1977, to Kingma et al., 2014) or graphs (notably following
Zhu et al., 2003).
Deep neural networks, which are state-of-the art supervised predictors, have beentrained successfully using SSL.
Somewhat surprisingly, the main ingredient of their success is still thenotion of pseudo-labels (or one of its variants), combined with a systematic use of data augmentation(e.g.
Xie et al., 2019; Sohn et al., 2020; Rizve et al., 2021).
An obvious SSL baseline is to simply to throw away the unlabelled data.
We will call such a baselinethe complete case , following the missing data literature (e.g. Tsiatis, 2006).
As reported in van
Engelen & Hoos (2020), the main risk of SSL is the potential degradation caused by the introductionof unlabelled data.
Indeed, semi-supervised learning outperforms the complete case baseline onlyin specific cases (Singh et al., 2008; Schölkopf et al., 2012; Li & Zhou, 2014).
This degradation riskfor generative models has been analysed in Chapelle et al.
(2006, Chapter 4).
To overcome this issue,previous works introduced the notion safe semi-supervised learning for techniques which never reducepredictive performance by introducing unlabelled data (Li & Zhou, 2014; Guo et al., 2020).
Our loose definition of safeness is as follows: a SSL algorithm is safe if it has theoretical guarantees that aresimilar or stronger to the complete case baseline .
The “theoretical” part of the definition is motivatedby the fact that any empirical assessment of generalisation performances of an SSL algorithm is jeopardised by the scarcity of labels.
Unfortunately, popular deep SSL techniques generally does not benefitof theoretical guarantees without strong and essentially untestable assumptions on the data distribution(Mey & Loog, 2019) such the smoothness assumption (small perturbations on the features x do notcause large modification in the labels, p ( y | pert ( x ))
≈ p ( y | x ) ) or the cluster assumption (data pointsare distributed on discrete clusters and points in the same cluster are likely to share the same label).
[Figure]
Most semi-supervised methods rely on these distributional assumptions to ensure performancein entropy minimisation, pseudo-labelling andconsistency-based methods.
However, no proofis given that guarantees the effectiveness of stateof-the-art methods (Tarvainen & Valpola, 2017;
Miyato et al., 2018; Sohn et al., 2020; Phamet al., 2021).
To illustrate that SSL requires specific assumptions, we show in a toy example thatpseudo-labelling fails at learning.
To do so, wedraw samples from two uniform distributionswith a small overlap.
Both supervised and semisupervised neural networks are trained using thesame labelled dataset.
While the supervised algorithm learns perfectly the true distribution ofp (1 | x ) , the semi-supervised learning methods(both entropy minimisation and pseudo-label)underestimate p (1 | x ) for x ∈ [1 , 3] (see Figure1).
We also test our proposed method (DeSSL)on this dataset and show that the unbiased version of each SSL technique learns the true distribution accurately.
See Appendix A for theresults with Entropy Minimisation.
1.1 Contributions
Rather than relying on the strong geometric assumptions usually used in SSL theory, we simply usethe missing completely at random (MCAR) assumption, a standard assumption from the missing dataliterature (see e.g. Little & Rubin, 2019).
With this only assumption on the data distribution, wepropose a new safe SSL method derived from simply debiasing common SSL risk estimates.
Ourmain contributions are:
• We introduce debiased SSL (DeSSL), a safe method that can be applied to most deep SSLalgorithms without assumptions on the data distribution;
• We propose a theoretical explanation of the intuition of popular SSL methods.
We providetheoretical guarantees on the safeness of using DeSSL both on consistency and calibration of the method.
We also provide a generalisation error bound;
• We show how simple it is to apply DeSSL to the most popular methods such as Pseudo-label
and Fixmatch, and show empirically that DeSSL leads to models that are never worse than
their classical counterparts, generally better calibrated and sometimes much more accurate.
Semi-supervised learning2.1 Learning with labelled data
The ultimate objective of most of the learning frameworks is to minimise a risk R , defined asthe expectation of a particular loss function L over a data distribution p ( x, y ) , on a set of modelsf θ ( x ) , parametrised by θ ∈ Θ .
Thus, the learning task is finding θ ∗ that minimises the risk:
[Equation] θ
[Equation] , Y )] .
The distribution p ( x, y ) being unknown, we generally minimise an approximation of the risk, the empirical risk ˆ R ( θ ) computed on a sample of n i.i.d points drawnfrom p ( x, y ) .
ˆ R ( θ ) is an unbiased and consistent estimate of R ( θ ) under mild assumptions.
Itsunbiased nature is one of the basic properties that is used for the development of traditional learningtheory and asymptotic statistics (van der Vaart, 2000; Shalev-Shwartz & Ben-David, 2014).
2.2 Learning with both labelled and unlabelled data
Semi-supervised learning leverages both labelled and unlabelled data to improve the model’s performance and generalisation.
Further information on the distribution p ( x ) provides a better understanding of the distributions p ( x, y ) and also p ( y | x ) .
Indeed, p ( x ) may contain information on p ( y | x )(Schölkopf et al., 2012, Goodfellow et al., 2016, Chapter 7.6, van Engelen & Hoos, 2020).
In the following, we have access to n samples drawn from the distribution p ( x, y ) where some of thelabels are missing.
We introduce a new random variable r ∈ { 0 , 1 } that governs whether or not a datapoint is labelled ( r = 0 missing, r = 1 observed).
The MCAR assumption states that the missingnessof a label y is independent of its features and the value of the label: p ( x, y, r ) =
p ( x, y ) p ( r ) .
This isthe case when nor features nor label carry information about the potential missingness of the labels.
This description of semi-supervised learning as a missing data problem has already been done inmultiple works –e.g. Seeger, 2000; Ahfock & McLachlan, 2019.
Moreover, the MCAR assumptionis implicitly made in most of the SSL works to design the experiments, indeed, missing labels aredrawn completely as random in datasets such as MNIST, CIFAR or SVHN (Tarvainen & Valpola,2017; Miyato et al., 2018; Xie et al., 2019; Sohn et al., 2020).
Complete case: throwing the unlabelled data away
In missing data theory, the complete case is the learning scheme that only uses fully observedinstances, namely labelled data.
The natural estimator of the risk is then simply the empirical riskcomputed on the labelled data.
Fortunately, in the MCAR setting, the complete case risk estimatekeeps the same good properties of the traditional supervised one: it is unbiased and convergespointwisely to R ( θ ) .
Therefore, traditional learning theory holds for the complete case under MCAR.
While these observations are hardly new (see e.g. Liu & Goldberg, 2020), they can be seen asparticular cases of the theory that we develop below.
The risk to minimise is
[Equation]
2.2.2 Incorporating unlabelled data
A major drawback of the complete case framework is that a lot of data ends up not being exploited.
Aclass of SSL approaches, mainly inductive methods with respect to the taxonomy of van Engelen &
Hoos (2020), generally aim to minimise a modified estimator of the risk by including unlabelled data.
Therefore, the optimisation problem generally becomes finding ˆ θ that minimises the SSL risk,
[Equation]
where H is a term that does not depend on the labels and λ is a scalar weight which balances thelabelled and unlabelled terms.
In the literature, H can generally be seen as a surrogate of L .
Indeed,it looks like the intuitive choices of H are equal or equivalent to a form of expectation of L on adistribution given by the model.
2.2.3 Some examples of surrogates
A recent overview of the recent SSL techniques has been proposed by van Engelen & Hoos (2020).
In this work, we focus on methods suited for a discriminative probabilistic model p θ ( y | x ) thatapproximates the conditional p ( y | x ) .
We categorised methods into two distinct sections, the entropyand the consistency-based.
Entropy-based methods Entropy-based methods aim to minimise a term of entropy of the predictions computed on unlabelled data.
Thus, they encourage the model to be confident on unlabelleddata, implicitly using the cluster assumption.
Entropy-based methods can all be described as anexpectation of L under a distribution π x computed at the datapoint x :
[Equation]
For instance, Grandvalet & Bengio (2004) simply use the Shannon entropy as H ( θ ; x ) which can berewritten as equation (3) with π x (˜ x, ˜ y )
=
δ x (˜ x ) p θ (˜ y | ˜ x ) .
Also, pseudo-label methods, which consistin picking the class with the maximum predicted probability as a pseudo-label for the unlabelled data(Scudder, 1965), can also be described as Equation 3.
See Appendix B for complete description ofthe entropy-based literature (Berthelot et al., 2019; 2020; Xie et al., 2019;
Sohn et al., 2020; Rizveet al., 2021; Zhang et al., 2021a) and further details.
Consistency-based methods Another range of SSL methods minimise a consistency objectivethat encourages invariant prediction for perturbations either on the data either on the model in orderto enforce stability on model predictions.
These methods rely on the smoothness assumption.
Inthis category, we cite Π -model from (Sajjadi et al., 2016), temporal ensembling from (Laine & Aila,2017), Mean-teacher proposed by (Tarvainen & Valpola, 2017), virtual adversarial training (VAT)from (Miyato et al., 2018) and interpolation consistent training (ICT) from (Verma et al., 2019).
Weremark that these objectives H are equivalent to an expectation of L (see Appendix B).
The generalform of the unsupervised objective can be written as
[Equation]
where the Div is a non-negative function that measures the divergence between two distributions, ˆ θ isa fixed copy of the current parameter θ (the gradient is not propagated throught ˆ θ ) and 0 ≤ C 1 ≤ C 2 .
Previous works also remarked that H is an expectation of L for entropy-minisation and pseudo-label(Zhu et al., 2022; Aminian et al., 2022).
We describe a more general framework covering furthermethods and provide with our theory an intuition on the choice of H .
2.3 Theoretical guarantees
The main risk of SSL is the potential degradation caused by the introduction of unlabelled data whendistributional assumptions are not satisfied (Singh et al., 2008;
Schölkopf et al., 2012; Li & Zhou,2014), specifically in settings where the MCAR assumption does not hold anymore (Oliver et al.
Guo et al., 2020).
Additionally, in (Zhu et al., 2022), the authors show disparate impacts ofpseudo-labelling on the different sub-classes of the population.
To mitigate these problems, previousworks introduced the notion safe semi-supervised learning for techniques which never reduce learningperformance by introducing unlabelled data (Li & Zhou, 2014; Kawakita & Takeuchi, 2014; Li et al.
,2016; Gan et al., 2017; Trapp et al., 2017; Guo et al., 2020).
As remark by Oliver et al.
SSL performances are enabled by leveraging large validation sets which is not suited for real-worldapplications.
Then, theoretical guarantees are required to use safely SSL algorithms.
For this reason,in our work, we consider as safe a SSL algorithm that has theoretical guarantees that are similaror stronger than those of the complete case baseline.
Even though the methods presented aboveproduce good performances in a variety of SSL benchmarks, they generally do not benefit fromtheoretical guarantees, even elementary.
More over,Schölkopf et al.
(2012) identify settings on thecausal relation between the features x and the target y where SSL may systematically fail, even ifclassic SSL assumptions hold.
Our example of Figure 1 also shows that classic SSL may fail togeneralise in a very benign setting with a large number of labelled data.
Presented methods minimise a biased version of the risk under the MCAR assumption and thereforeclassical learning theory cannot be applied anymore, as we argue more precisely in Appendix C.
Learning over a biased estimate of the risk is not necessarily unsafe but it is difficult to providetheoretical guarantees on such methods even if some works try to do so with strong assumptionson the data distribution (Mey & Loog 2019, Section 4 and 5).
Additionally, we remark that thechoice of H can be confusing as seen in the literature.
For instance, Grandvalet & Bengio (2004) and
Corduneanu & Jaakkola (2003) perform respectively entropy and mutual information minimisationwhereas Pereyra et al.
(2017) and Krause et al.
(2010) perform maximisation of the same quantities.
2.4 Related works
Previous works already proposed safe SSL methods with theoretical guarantees.
Unfortunately,so far these methods come with either strong assumptions or important computational burden.
Li& Zhou (2014) introduced a safe semi-supervised SVM and showed that the accuracy of theirmethod is never worse than SVMs trained with only labelled data with the assumption that the truemodel is accessible.
However, if the distributional assumptions are not satisfied, no improvement ordegeneration is expected.
Sakai et al.
(2017) proposed an unbiased estimate of the risk for binaryclassification by including unlabelled data.
The key idea is to use unlabelled data to better evaluateon the one hand the risk of positive class samples and on the other the risk of negative samples.
They provided theoretical guarantees on its variance and a generalisation error bound.
The methodis designed only for binary classification and has not been tested in a deep learning setting.
It hasbeen extended to ordinal regression in follow-up work (Tsuchiya et al., 2021).
In the context ofkernel machines, Liu & Goldberg (2020) used an unbiased estimate of risk, like ours, for a specificchoice of H .
Guo et al.
(2020) proposed DS 3 L , a safe method that needs to approximately solvea bi-level optimisation problem.
In particular, the method is designed for a different setting, notunder the MCAR assumptiom, where there is a class mismatch between labelled and unlabelled data.
The resolution of the optimisation problem provides a solution not worse than the complete case butcomes with approximations.
They provide a generalisation error bound.
Also, the method does notoutperform classic SSL methods in the MCAR setting as it is designed for non-MCAR situations.
Sokolovska et al.
(2008) proposed a safe method with strong assumptions such that the feature spaceis finite and the marginal probability distribution of x is fully known.
Fox-Roberts & Rosten (2014)proposed an unbiased estimator in the generative setting applicable to a large range of models andthey prove that this estimator has a lower variance than the one of complete case.
DeSSL: Unbiased semi-supervised learning
In order to overcome the issues introduced by the second term in the approximation of the risk for thesemi-supervised learning approach, we propose DeSSL, an unbiased version of the SSL estimator using labelled data to annul the bias.
The idea here is to retrieve the properties of classical learning theory.
Fortunately, we will see that the proposed method can eventually have better properties than the complete case, in particular with regards to the variance of the estimate.
The proposed DeSSL objective is
[Equation]
Under the MCAR assumption, this estimator is unbiased for any value of the parameter λ .
For aproof of this result see Appendix D.
Intuitively, for entropy-based methods H should be applied only on unlabelled data to enforce theconfidence of the model only on unlabelled datapoints.
Whereas, for consistency-based method,
H can be applied to any subset of data points.
Our theory and proposed method remain the samewhether H is applied on all the available data or not (see Appendix I).
3.1 Does the DeSSL risk estimator make sense?
The most intuitive interpretation is that by debiasing the risk estimator, we get back to the basics oflearning theory.
This way of debiasing is closely related to the method of control variates (Owen,2013, Chapter 8) which is a common variance reduction technique.
The idea is to add an additionalterm to a Monte-Carlo estimator with a null expectation in order to reduce the variance of theestimator without modifying the expectation.
Here, DeSSL can also be interpreted as a control variateon the risk’s gradient itself and should improve the optimisation scheme.
This idea is close to theoptimisation schemes introduced by Johnson & Zhang (2013) and Defazio et al.
(2014) which reducethe variance of the gradients’ estimate to improve optimisation performance.
Another interesting way to interpret DeSSL is as a constrained optimisation problem.
Indeed, minimising ˆ R DeSSL is equivalent to minimising the Lagrangian of the following optimisation problem:
[Equation]
The idea of this optimisation problem is to minimise the complete case risk estimator by assessingthat some properties represented by H are on average equal for the labelled data and the unlabelleddata.
For example, if we consider entropy-minimisation, this program encourages the model to havethe same confidence on the unlabelled examples as on the labelled ones.
The debiasing term of our objective will penalise the confidence of the model on the labelled data.
Pereyra et al.
(2017) actually show that penalising the entropy in a supervised context acts as a strongregulator for supervised model and improves on the state-of-the-art on common benchmarks.
Thiscomforts us in the idea of debiasing using labelled data in the case of entropy-minimisation.
Similarly,the debiasing term in pseudo-label turns the problem into plausability inference as described by
Barndorff-Nielsen (1976).
Our objective also resembles doubly-robust risk estimates used for SSL in the context of kernelmachines by Liu & Goldberg (2020) and for deep learning in a recent preprint (Hu et al., 2022).
In both cases, their focus is quite different, as they consider weaker conditions than MCAR, butvery specific choices of H .
3.2 Is ˆ R DeSSL ( θ ) an accurate risk estimate?
Because of the connections between our debiased estimate and variance reduction techniques, wehave a natural interest in the variance of the estimate.
Having a lower-variance estimate of the riskwould mean estimating it more accurately, leading to better models.
Similarly to traditional controlvariates (Owen, 2013), the variance can in fact be computed, and optimised in λ :
Theorem 3.1.
The function λ (cid:55)→ V
( ˆ R DeSSL ( θ )) reaches its minimum for:
[Equation]
and at λ opt :
[Equation]
where ρ L,H =
Corr( L ( θ ; x, y ) , H ( θ ; x )) .
A proof of this theorem is available as Appendix E. This theorem provides a formal justificationto the heuristic idea that H should be a surrogate of L .
Indeed, DeSSL is a more accurate riskestimate when H is strongly positively correlated with L , which is likely to be the case when His equal or equivalent to an expectation of L .
Then, choosing λ positive is a coherent choice.
Wealso demonstrate in Appendix E that L and H are positively correlated when L is the negativelikelihood and H is the entropy.
Other SSL methods have variance reduction guarantees and alreadyhas shown great promises in SSL, see Fox-Roberts & Rosten (2014) and Sakai et al.
Ina purely supervised context, Chen et al. (2020) show that the effectiveness of data augmentationtechniques lays partially on the variance reduction of the risk estimate.
A natural application of thistheorem would be to tune λ automatically by estimating λ opt .
In our case however, the estimation of Cov( L ( θ ; x, y ) , H ( θ ; x )) with few labels led to extremely unstable unsatisfactory results.
3.3 Calibration
The calibration of a model is its capacity of predicting probability estimates that are representativeof the true distribution.
This property is determinant in real-world application when we needreliable predictions.
A scoring rule S is a function assigning a score to the predictive distribution p θ ( y | x ) relative to the event y | x ∼ p ( y | x ) , S ( p θ , ( x, y )) , where p ( x, y ) is the true distribution (seee.g.
Gneiting & Raftery, 2007).
A scoring rule measures both the accuracy and the quality ofpredictive uncertainty, meaning that better calibration is rewarded.
The expected scoring rule isdefined as S ( p θ , p ) =
E p [ S ( p θ , ( x, y ))] .
A proper scoring rule is defined as a scoring rule suchthat S ( p θ , p ) ≤ S ( p, p ) (Gneiting & Raftery, 2007).
The motivation behind having proper scoringrules comes from the following: suppose that the true data distribution p is accessible by ourset of models.
Then, the scoring rule encourages to predict p θ = p .
The opposite of a properscoring rule can then be used to train a model to encourage the calibration of predictive uncertainty:
L ( θ ; x, y )
=
−S ( p θ , ( x, y )) .
Most common losses used to train models are proper scorings rule suchas log-likelihood.
Theorem 3.2.
If S ( p θ , ( x, y ))
= − L ( θ ; x, y ) is a proper scoring rule, then S ′ ( p θ , ( x, y, r ))
=− ( rnn l L ( θ ; x, y )
+ λn ( 1 − r n u − rn l ) H ( θ ; x )) is also a proper scoring rule.
The proof is available in Appendix F, and follows directly from unbiasedness and the MCARassumption.
The main interpretation of this theorem is that we can expect DeSSL to be as wellcalibrated as the complete case.
3.4 Consistency
We say that ˆ θ is consistent if d (ˆ θ, θ ∗ ) p −→ 0 when n −→ ∞ , where d is a distance on Θ .
The asymptoticproperties of ˆ θ depend on the behaviours of the functions L and H .
We will thus require the followingstandard assumptions.
Assumption 3.3.
The minimum θ ∗ of R is well-separated: inf θ : d ( θ ∗ ,θ ) ≥ ϵ R ( θ ) >
R ( θ ∗ ) .
Assumption 3.4.
The uniform weak law of large number holds for both L and H .
Theorem 3.5.
Under the MCAR assumption, Assumption 3.3 and Assumption 3.4, ˆ θ =arg min ˆ R DeSSL is consistent.
For a proof of this theorem see Appendix F. This theorem is a simple application of van der Vaart’s(2000) Theorem 5.7 proving the consistency of a M-estimator.
Also, this results holds for the completecase, with λ = 0 which prove that the complete case is a solid baseline under the MCAR assumption.
Coupling of n l and n u under the MCAR assumption Under the MCAR assumption, n l and n uare random variables.
We have that r ∼ B ( π ) (i.e. any x has the probability π of being labelled).
Then, with n growing to infinity, we have n l n = n l n l + n u −→ π .
Therefore, both n l and n u grow toinfinity and n l n u −→ π − 1 π .
This implies n u
= O ( n l ) and then when n goes to infinity, both n u and n lgo to infinity too and even if n u >> n l .
3.5 Rademacher complexity and generalisation bounds
In this section, we prove an upper bound for the generalisation error of DeSSL.
The unbiasedness ofˆ R DeSSL can directly be used to derive generalisation bounds based on the Rademacher complexity(Bartlett & Mendelson, 2002), defined in our case as [Equation]
[Equation]
9) where ε i are i.i.d.
Rademacher variables independent of the data.
In the particular case of λ = 0 ,we recover the standard Rademacher complexity of the complete case.
We can then now bound thegeneralisation error of a model trained using our new loss function.
Theorem 3.6.
We assume that labels are MCAR and that both L and H are bounded.
Then, thereexists a constant κ > 0 , that depends on λ , L , H , and the ratio of observed labels, such that, withprobability at least 1 − δ , for all θ ∈ Θ ,
[Equation]
The proof follows Shalev-Shwartz & Ben-David (2014, Chapter 26), and is available in Appendix H.
Experiments
[Figure] We evaluate the performance of DeSSL against different classic methods.
The goal here is to compare
DeSSL methods and their orginal counterparts.
In particular, we perform experiments with simple
SSL methods such as pseudo-label (PseudoLabel) and entropy minimisation (EntMIN) with varyingλ on MNIST (LeCun & Cortes, 2010) and CIFAR-10 and CIFAR-100 (Krizhevsky, 2009) andcompare them to the debiased method, respectively DeEntMin and DePseudoLabel.
We also compare
PseudoLabel and DePseudoLabel on five small datasets of MedMNIST (Yang et al., 2021a;b) with afixed λ .
The results of these experiments are reported below.
In our figures, the error bars representthe size of the 95% confidence interval (CI).
Finally, we modified the implementation of Fixmatch(Sohn et al., 2020) and compare it with its debiased version on CIFAR-10.
We also compare DeEntMin and DePseudoLabel to the biased version on a large range of tabulardatasets commonly used in SSL benchmarks (Chapelle et al., 2006; Guo et al., 2010).
We do notobserve differences between the performance, see Appendix N. Finally, we show how simple it is todebias an existing implementation, by demonstrating it on the consistency-based models benchmarkedby (Oliver et al., 2018), namely VAT, Π -model and MeanTeacher on CIFAR-10 and SVHN (Netzeret al., 2011).
We observe similar performances between the debiased and biased version for the different methods, both in terms of cross-entropy and accuracy.
Moreover, these results have been obtainedusing the hyperparameters finetuned for the biased versions.
Therefore, it is likely that optimising thehyperparameters for DeSSL will yield even better with the right hyperparameters, see Appendix M.
4.1 MNIST
[Figure]
MNIST is an advantageous dataset for SSL since classes arewell-separeted.
We compare PseudoLabel and DePseudoLabelfor a LeNet-like architecture using n l = 1000 labelled dataon 10 different splits of the training dataset into a labelled andunlabelled set.
Models are then evaluated using the standard10 , 000 test samples.
We used 10% of n l as the validation set.
We test the influence of the hyperparameter λ and report theaccuracy, the cross-entropy and the expected calibration error(ECE, Guo et al., 2017) at the epoch of best validation accuracy,see Figure 2 and
Appendix J.
In this exemple SSL and DeSSLhave the almost the same accuracy for all λ , however, DeSSLseems to be alway better calibrated.
In order to break the clusterassumption, we reproduced the same experiment on a modified MNIST.
Indeed, we had label noise by replacing the true label for20% of the dataset by a randomly sampled label, see Appendix
J. In this setting, DeSSL performs better for large λ in term ofaccuracy and also provides a better calibration.
4.2 MedMNIST
We compare PseudoLabel and DePseudoLabel on different datasets of MedMNIST, a large-scale
MNIST-like collection of biomedical images.
We selected the five smallest 2D datasets of thecollection, for these dataset it is likely that the cluster assumption no longer holds.
We trained a5-layer CNN with a fixed λ = 1 and n l at 10% of the training data.
We report in Table 1 the meanaccuracy and cross-entropy on 5 different split of the labelled and unlabelled data and the numberof labelled data used.
We report the AUC in Appendix J. DePseudoLabel compete with PseudoLabelin terms of accuracy and even success when PseudoLabel’s accuracy is less than the completecase.
Moreover, DePseudoLabel is always better in term of cross-entropy, so calibration, whereas PseudoLabel is always worse than the complete case.
[Table]
[Figure]
We compare PseudoLabel and DePseudoLabel on CIFAR-10and CIFAR-100.
We trained a CNN-13 from Tarvainen &
Valpola (2017) on 5 different splits.
For this experiement, weuse n l = 4000 and use the rest of the dataset as unlabelled.
Models are then evaluated using the standard 10 , 000 testsamples.
For a more realistic validation set, we used 10% of n las the validation set.
We test the influence of the hyperparameterλ and report the accuracy and the cross-entropy at the epochof best validation accuracy, see Figure 3.
We report the ECE in Appendix K. The performance of both methods on CIFAR-100with nl = 10000 are reported in Appendix K. We observe DeSSL provides both a better cross-entropy and ECE with thesame accuracy for small λ .
For larger λ , DeSSL performs betterin all the reported metrics.
We performed a paired Student’st-test to ensure that our results are significant and reported thep-values in Appendix K.
The p-values indicates that for λ closeto 10, DeSSL is often significantly better in all the metrics.
Moreover, DeSSL for large λ provides a better cross-entropyand ECE than the complete case whereas SSL never does.
4.4 Fixmatch (Sohn et al., 2020)
[Table]
We debiased a version of Fixmatch, see Appendix L forfurther details.
For this experiment, we use nl = 4000 on 5different folds.
First, we report that a strong baseline usingdata augmentation reach 87 .
27% accuracy.
Then, we observe that on the debiasing method improve both accuracyand cross-entropy of this modified version of Fixmatch.
Inspired by Zhu et al.
(2022), we show that our methodimproved performance on “poor” classes more equally than the biased version.
Indeed, DeFixmatchimproves Fixmatch by 1 . 57% overall but by 4 .
91% on the worst class.
We report in Appendix Lthe accuracy per class of the different methods and the benefit ratio as defined by Zhu et al.
Conclusion
Motivated by the remarks of van Engelen & Hoos (2020) and Oliver et al.
(2018) on the missingnessof theoretical guarantees in SSL, we proposed a simple modification of SSL frameworks.
We considerframeworks based on the inclusion of unlabelled data in the computation of the risk estimator anddebias them using labelled data.
We show theoretically that this debiasing comes with severaltheoretical guarantees.
We demonstrate these theoretical results experimentally on several common SSL datasets and some more challenging ones such as MNIST with label noise.
DeSSL showscompetitive performance in term of accuracy compared to its biased version but improves significantlythe calibration.
There are several future directions open to us.
We showed that λ opt exists (Theorem3.1) and therefore our formula provides guidelines for the optimisation of λ .
Finally, an interestingimprovement would be to go beyond the MCAR assumption by considering settings with a distributionmismatch between labelled and unlabelled data (Guo et al., 2020; Cao et al., 2021;
Hu et al., 2022).
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See section
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
Theoretical
work
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendices
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[Yes] code and
instructions to run Fixmatch.
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes] See Appendices
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] Confidence intervals on all figures.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] An estimation in Appendix
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Appendix
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
Code
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[-As a consequence of the theorem, under the MCAR assumption, ˆ R CC ( θ ) is also unbiased as a specialcase of ˆ R DeSSL ( θ ) for λ = 
-]
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]