O N N ON -R ANDOM M ISSING L ABELS IN S EMI -S UPERVISED L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Semi-Supervised Learning (SSL) is fundamentally a missing label problem, in which the label Missing Not At Random (MNAR) problem is more realistic and challenging, compared to the widely-adopted
yet na¨ıve Missing Completely At
Random assumption where both labeled and unlabeled data share the same class distribution.
Different from existing SSL solutions that overlook the role of “class” in causing the non-randomness, e.g. , users are more likely to label popular classes, we explicitly incorporate “class” into SSL.
Our method is three-fold: 1) We propose Class-Aware Propensity (CAP) score that exploits the unlabeled data to train an improved classifier using the biased labeled data.
To encourage rare class training, whose model is low-recall but high-precision that discards too many pseudo-labeled data, we propose Class-Aware Imputation (CAI) that dynamically decreases (or increases) the pseudo-label assignment threshold for rare (or frequent) classes.
3) Overall, we integrate CAP and CAI into a Class-Aware
Doubly Robust (CADR) estimator for training an unbiased SSL model.
Under various MNAR settings and ablations, our method not only significantly outperforms existing baselines, but also surpasses other label bias removal SSL methods.
I NTRODUCTION
Semi-supervised learning (SSL) aims to alleviate the strong demand for large-scale labeled data by leveraging unlabeled data (Zhu, 2008; Yang et al., 2021).
Prevailing SSL methods first train a model using the labeled data, then uses the model to impute the missing labels with the predicted pseudolabels for the unlabeled data (Van Buuren, 2018), and finally combine the true- and pseudo-labels to further improve the model (Sohn et al., 2020; Berthelot et al., 2019a).
Ideally, the “improve” is theoretically guaranteed if the missing label imputation is perfect (Grandvalet & Bengio, 2005); otherwise, imperfect imputation causes the well-known confirmation bias (Arazo et al., 2019; Sohn et al., 2020).
In particular, the bias is even more severe in practice because the underlying assumption that the labeled and unlabeled data are drawn from the same distribution does not hold.
We term this scenario as label Missing Not At Random (MNAR) (Hern ´ an & Robins, 2020), as compared to the na¨ıve assumption called label Missing Completely At Random (MCAR).
MNAR is inevitable in real-world SSL due to the limited label annotation budget (Rosset et al., 2005)—uniform label annotations that keep MCAR are expensive.
For example, we usually deploy low-cost data collecting methods like crawling social media images from the Web (Mahajan et al., 2018).
However, the high-quality labels appear to be severely imbalanced over classes due to the imbalanced human preferences for the “class” (Misra et al., 2016; Coll´eony et al., 2017).
For example, people are more willing to tag a “Chihuahua” rather than a “leatherback” (a huge black turtle) since the former is more lovely and easier to recognize.
Thus, the imputation model trained by such biased supervision makes the model favoring “Chihuahua” rather than “leatherback”.
The “Chihuahua”-favored pseudo-labels, in turn, negatively reinforce the model confidence in the false “Chihuahua” belief and thus exacerbate the confirmation bias.
We design an experiment on CIFAR-10 to further illustrate how existing state-of-the-art SSL methods, e.g. , FixMatch (Sohn et al., 2020), fail in the MNAR scenario.
As shown in Figure 1(a), the overall training data is uniformly distributed over classes, but the labeled data is long-tailed distributed, which simulates the imbalanced class popularity.
Trained on such MNAR training data,
FixMatch even magnifies the bias towards the popular classes and ignores the rare classes (Fig-
FixMatchOurs            1 2 3 4 5 6 7 8
UnlabeledLabeled # un l a b e l e d # l a b e l e d
A cc u r a c y ( % )    1 2 3 4 5 6 7 8 90.50.
C o n f i d e n c e
[Figure] [Figure]
(c)
[Figure]
ure 1(b)).
We point out that the devil is in the imputation.
As shown in Figure 1(c), the confidence scores of FixMatch for popular classes are much higher than those for rare classes ( i.e. , averagely > 0.95 vs. < 0.05).
Since FixMatch adopts a fixed threshold for the imputation samples selection, i.e. , only the samples with confidence larger than 0.95 are imputed, FixMatch tends to impute the labels of more samples from the popular classes , which still keeps the updated labels with the newly added pseudo-labels long-tailed.
As a result, FixMatch is easily trapped in MNAR.
In fact, MNAR has attracted broad attention in statistics and Inverse Propensity Weighting (IPW) is one of the commonly used tools to tackle this challenge (Seaman & White, 2013; Jones et al., 2006).
IPW introduces a weight for each training sample via its propensity score, which reflects how likely the label is observed ( e.g. , its popularity).
In this way, IPW makes up a pseudo-balanced dataset by duplicating each labeled data inversely proportional to its propensity—less popular samples should draw the same attention as the popular ones—a more balanced imputation.
To combine the IPW true labels and imputed missing labels, a Doubly Robust (DR) estimator is used to guarantee a robust
SSL model if either IPW or imputation is unbiased (Seaman & Vansteelandt, 2018; Vansteelandt et al., 2010).
However, the above framework overlooks the role of “class” in causing the MNAR problem of SSL (Section 3), and we therefore propose a unified Class-Aware Doubly Robust (CADR) framework to address this challenge (Section 4).
CADR removes the bias from both the supervised model training and the unlabeled data imputation ends.
For the former end, we propose Class-Aware
Propensity (CAP) that exploits the unlabeled data to train an improved classifier using the biased labeled data (Section 4.1).
We use an efficient moving averaging strategy to estimate the propensity.
For the latter end, we design Class-Aware Imputation (CAI) that dynamically adjusts the pseudolabel assignment threshold with class-specific confidence scores to mitigate the bias in imputation (Section 4.2).
Intuitively, CAI lowers the threshold requirement of rare class samples to balance with frequent class samples.
Experiments on several image classification benchmarks demonstrate that our CADR framework achieves a consistent performance boost and can effectively tackle the MNAR problem.
Besides, as MCAR is a special case of MNAR, our method maintains the competitive performance in the conventional SSL setting.
The key contributions of this work are summarized as follows:
• We propose a realistic and challenging label Missing Not At Random (MNAR) problem for semisupervised learning, which is not extensively studied in previous work.
We systematically analyze the bias caused by non-random missing labels.
• We proposes a unified doubly robust framework called Class-Aware Doubly Robust (CADR) estimator to remove the bias from both the supervised model training end—by using Class-Aware Prospesity (CAP), and the unlabeled data imputation end—by using Class-Aware Imputation (CAI).
• Our proposed CADR achieves competitive performances in both MNAR and conventional label Missing Completely At Random (MCAR).
R ELATED W ORKS
Missing Not At Random .
Missing data problems are ubiquitous across the analysis in social, behavioral, and medical sciences (Enders, 2010; Heckman, 1977).
When data is not missing at random, estimations based on the observed data only results in bias (Heckman, 1977).
The solutions are three-fold: 1) Inverse probability weighting (IPW) assigns weights to each observed datum based on the propensity score, i.e. , the probability of being observed.
A missing mechanism-based model is used to estimate the sample propensity, such as a logistic regression model (Rosset et al., 2005;
Wang et al., 1997) or a robit regression model (Kang & Schafer, 2007; Liu, 2005).
2) Imputation methods aim to fill in the missing values to produce a complete dataset (Kenward & Carpenter, 2007;
Little & Rubin, 2014).
An imputation model is regressed to predict the incomplete data from the observed data in a deterministic (Kovar & Whitridge., 1995) or stochastic (Lo et al., 2019) way.
As the propensity estimation or the regressed imputation are easy to be biased, doubly robust estimators propose to integrate IPW and imputation with double robustness (Seaman & Vansteelandt, 2018;
Vansteelandt et al., 2010; Jonsson Funk et al., 2011): the capability to maintain unbiased when either the propensity or imputation model is biased.
In this work, as the image labels are missing not at random, we design the class-aware propensity and imputation considering the causal role of label classes in missing, avoiding the uncertain model specification and generating unbiased estimation.
These two class-aware modules can be naturally integrated into a doubly robust estimator.
Semi-Supervised Learning (SSL) .
It aims to exploit unlabeled data to improve the model learned on labeled data.
Prevailing SSL methods (Sohn et al., 2020; Berthelot et al., 2019b;a) share a similar strategy: training a model with the labeled data and generating pseudo-labels for unlabeled data based on the model predictions.
Pseudo-labeling methods (Lee, 2013; Xie et al., 2020; Rizve et al., 2021) predict pseudo-labels for unlabeled data and add them to the training data for re-training.
Consistency-regularization methods (Sajjadi et al., 2016; Laine & Aila, 2016; Berthelot et al., 2019b) apply a random perturbation to an unlabeled image and then use the prediction as the pseudo-label of the same image under a different augmentation.
Recent state-of-the-art SSL methods (Sohn et al., 2020; Berthelot et al., 2019a) combine the two existing techniques and predict improved pseudolabels.
However, when labels are missing at random, these methods can be inefficient as the model learned on labeled data is biased and significantly harms the quality of pseudo labels.
Though some works also notice the model bias (Wei et al., 2021; Kim et al., 2020), they neglect the causal relationship between the bias and the missing label mechanism in SSL, which is systematically discussed in our work.
M ISSING L ABELS IN S EMI -S UPERVISED L EARNING
In semi-supervised learning (SSL), the training dataset D is divided into two disjoint sets: a labeled dataset D L and an unlabeled data D U .
We denote D L as { ( x ( i ) , y ( i ) )
} N L i =1 (usually N L ≪ N ) where x ( i ) ∈
R d is a sample feature and y ( i ) ∈ { 0 , 1 } C is its one-hot label over C classes, and D U as { ( x ( i ) )
} N i = N L +1 where the remaining N − N L labels are missing in D U .
We further review SSL as a label missing problem, and define a label missing indicator set M with m ( i ) ∈ { 0 , 1 } , where m ( i ) =
1 denotes the label is missing ( i.e. , unlabeled) and m ( i ) =
0 denotes the label is not missing ( i.e. , labeled).
In this way, we can rewrite the dataset as D =( X, Y, M )=
{ ( x ( i ) , y ( i ) , m ( i ) ) }
Ni =1 .
Traditional SSL methods assume that M is independent with Y , i.e. , the labels are Missing Completely At Random (MCAR).
Under this assumption, prevailing SSL algorithms can be summarized as the following multi-task optimization of the supervised learning and the unlabeled imputation:
[Equation]
where L s and L u are the loss function designed separately for supervised learning on labeled data ( M = 0 ) and regression imputation on unlabeled data ( M = 1 ).
In general, SSL methods first train a model with parameters ˆ θ using L s on D L , then impute the missing labels (Van Buuren, 2018) by predicting the pseudo-labels ˆ y = f
( x ; ˆ θ ) , and finally combine the true- and pseudo-labels to further improve the model for another SSL loop, e.g. , another optimization iteration of Eq.
The implementation of L s and L u are open.
For example, L s is normally the standard cross-entropy loss, and L u can be implemented as squared L 2 loss (Berthelot et al., 2019b) or cross-entropy loss (Berthelot et al., 2019a; Sohn et al., 2020).
As you may expect, the key to stable SSL methods is the unbiasedness of the imputation model.
When M is independent with Y , we have P ( Y | X
= x, M =0) =
P ( Y | X = x ) , and thus
[Equation]
which indicates that the model is an ideally unbiased estimator, and the imputed labels are unbiased.
However, such MCAR assumption is too strong and impractical.
For example, annotators may tend to tag a specific group of samples due to their preferences or experiences.
This motivates us to ask the following realistic but overlooked question: what if M is dependent with Y , i.e. , the labels are missing not at random (MNAR)?
In this case, we have P ( Y | X
= x, M = 0) ̸
= P ( Y | X = x ) , and thus
[Equation]
We can see it is no longer unbiased, further leading to biased imputed labels.
As a result, the bias will be increasingly enhanced with the joint optimization loop in Eq.
(1) moving forward.
This analysis indicates that previous SSL methods are not applicable for the MNAR problem as they overlook the role of “class” Y in causing the non-randomness of M .
In the following, we will introduce our
Class-Aware Doubly Robust framework that targets at an unbiased estimator.
C LASS -A WARE D OUBLY R OBUST F RAMEWORK
As discussed above, both the supervised learning and regression imputation in previous SSL methods suffer from the biased label estimation for MNAR.
In this section, we proposed a doubly robust estimator that explicitly incorporates “class” into SSL and mitigates the bias from both sides.
4. C LASS -A WARE P ROPENSITY FOR L ABELED D ATA
Traditional SSL methods estimate the model parameters via maximum likelihood estimation on the labeled data:
[Equation]
Since ˆ θ is estimated over D L rather than the entire D , the difference between D L and D determines the unbiasedness of ˆ θ .
As analyzed in Section 3, ˆ θ is an unbiased estimator if the labels Y and their missing indicators M are independent, i.e. , MCAR.
However, when
M is dependent with Y , i.e. , MNAR, we have P ( y | x ; M = 0) ̸ =
P ( y | x ) .
In this case, ˆ θ is no longer unbiased.
Inspired by causal inference (Pearl et al., 2016), we establish a structural causal model for MNAR to analyze the causal relations between X , Y and M shown in Figure 2.
First, X → Y denotes that each image determines its own label.
Second, Y → M denotes that the missing of label is dependent with its category.
For example, popular pets like cat and dog attracts people to tag more often.
We observe a chain structure of X → Y → M , which obtains an important conditional independence rule (Pearl et al., 2016):
Variables A and B are conditionally independent given C , if there is only one unidirectional path between A and B , and C is any set of variables intercepting the path.
Based on this rule, we have
P ( X | Y, M = 0) =
P ( X | Y ) when conditioning on Y , i.e. , the path between X and M is blocked by
Y .
In this way, we can obtain the unbiased ˆ θ on the labeled subset of data by maximizing P ( X | Y ) :
[Equation]
Eq.
(5) to Eq.
(6) holds by Bayes’ rule; Eq.
(6) to Eq.
(7) holds because x is sampled from an empirical distribution and p ( x ; θ ) is thus constant.
Eq.(7) to Eq.(9) shows the connection with
Inverse Probability Weighting, where we estimate the propensity score s ( x, y ) for image x as log P
( y | x ; θ ) log P ( y | x ; θ ) − log P ( y ; θ ) .
Comparing the training objectives in Eq.
(9) with Eq.
(4), our estimation can be understood as adjusting the original P ( y | x ; θ ) by a class-aware prior P ( y ; θ ) for each labeled data ( x, y ) .
To estimate P ( y ; θ ) , i.e. , the marginal distribution P ( Y = y ) with the current parameters θ , a straightforward solution is to go through the whole dataset and calculate the mean probability of all the data.
However, during the training stage, θ is updated step by step, and it is impractical to go thorough the whole dataset in each iteration considering the memory and computational cost.
An alternative solution is to estimate the propensity within a mini-batch.
Although the cost is saved, it faces a risk of high-variance estimation.
To resolve the dilemma, we propose to use a moving averaging strategy over all the minibatches.
Specifically, we keep a buffer ˆ P ( Y ) to estimate P ( Y ; θ t ) and continuously update it by
P ( Y ; B t , θ t ) , which is estimated with the current mini-batch of B t samples and parameters θ t at t -th iteration:
[Equation]
where µ ∈ [0 , 1) is a momentum coefficient.
4. C LASS -A WARE I MPUTATION FOR U NLABELED D ATA
As discussed in Section 1, the imputation model in traditional SSL is biased towards popular classes, and so are the imputed labels.
Although our proposed CAP can mitigate the preference bias during the supervised learning stage, the model can still tend to impute popular class that achieves a high confidence score.
To further alleviate the bias in the imputation stage, we propose to remove the inferior imputed labels from ˆ Y .
Traditional models like FixMatch adopt a fixed threshold for imputation samples selection, which aims to reserve the accurate imputed labels and discard the noisy ones.
However, as shown in Figure 1 (c), for specific classes, an extremely small number of imputed labels can meet the requirement while most are discarded.
In fact, as observed in previous works (Wei et al., 2021), these discarded predictions still hold nearly perfect precision.
Based on these observations, we believe that a fixed threshold is too coarse to impute missing labels.
To tackle this challenge, we propose a Class-Aware Imputation (CAI) strategy that dynamically adjusts the pseudo-label assignment threshold for different classes.
Let C x
denote
the potential imputed label for image x , i.e. , C x =arg max y P ( y | x ; θ ) .
We use a class-aware threshold τ ( x ) for image x as:
[Equation]
where τ o is the conventional threshold, β is a hyper-parameter, and ˆ P ( y ) is the class-aware propensity for class y estimated by CAP.
Intuitively, our class-aware threshold would set a higher requirement for popular classes and a lower requirement for rare classes, allowing more samples from rare classes to be imputed.
With the gradual removal of bias from the training process, the performance gap between classes also shrinks, and both popular and rare classes can be fairly treated.
Furthermore, when labels are missing completely at random (MCAR), our CIA can be degraded to the conventional imputation like previous works, as ˆ P ( Y ) is uniform over all classes.
4. C LASS -A WARE D OUBLY R OBUST E STIMATOR
Note that CAP and CAI mitigate the bias in supervised learning and imputation regression.
Thanks to the theory of double robustness (Seaman & Vansteelandt, 2018; Vansteelandt et al., 2010), we can incorporate CAP and CAI into a Class-Aware Doubly Robust (CADR) estimator.
Theoretically, the CADR combination has a lower tail bound than applying each of the components alone (Wang et al., 2019).
Besides, it provides our learning system with double robustness: the capability to remain unbiased if either CAP or CAI is unbiased.
Following the formulation of DR estimator, we first rewrite the training objective of CAP and CAI in semi-supervised learning as:
[Equation]
where m ( i ) is the missing state, p ( i ) is the propensity score, q ( i ) is the imputed label with confidence con ( q ( i ) ) , and I ( · ) is the indicator function.
As introduced in Section 4.1, we estimate the propensity score p ( i ) as s ( x ( i ) , y ( i ) ) in CAP.
Then the optimization of CADR estimator is implemented as
[Equation]
where
[Equation]
which is a supplementary loss to guarantee the unbiasedness.
In this design, L CAI + L supp is expected to be 0 given correct CAP, and L CAP + L supp is expected to be 0 given correct CAI.
These results guarantee the double robustness in case that either the propensity or imputation is inaccurate.
E XPERIMENTS 5. E XPERIMENTAL SETUP
Datasets.
We evaluate our method on four image classification benchmark datasets: CIFAR-10,
CIFAR-100 (Krizhevsky, 2012), STL-10 (Coates et al., 2011) and mini-ImageNet (Vinyals et al., 2016).
CIFAR-10(-100) is composed of 60,000 images of size 32 × 32 from 10 (100) classes and each class has 5,000 (500) training images and 1,000 (100) samples for evaluation.
STL-10 dataset has 5,000 labeled images and 100,000 unlabeled images of size 64 × 64 .
mini-ImageNet is a subset of ImageNet (Deng et al., 2009).
It contains 100 classes where each class has 600 images of size 84 × 84 .
Follows previous SSL works (Hu et al., 2021; Iscen et al., 2019), we select 500 images from each class for training and 100 images per class for testing.
MNAR Settings.
Since the training data is class-balanced in all datasets, we randomly select a class-imbalanced subset as the labeled data to mimic the label missing not at random (MNAR).
For a dataset containing C classes, the number of labeled data in each class N i are calculated as
N i = N max · γ − i − 1 C − 1 .
N 1 = N max is the maximum number of labeled data among all the classes, and γ describes the imbalance ratio.
γ = 1 when the labeled data is balanced over classes, and larger γ indicates more imbalanced class distribution.
Figure 1(a) shows an example of the data distribution when γ = N max = 50 in CIFAR-10.
We also consider other MNAR settings where the unlabeled data is also imbalanced, i.e. , the imbalance ratio of unlabeled data γ u does not equal to 1.
To ensure the number of unlabeled data is much larger than the labeled, we set the least number of labeled data over all classes as 1, and the largest number of unlabeled data to its original size.
Training Details.
Following previous works (Berthelot et al., 2019b; Sohn et al., 2020; Hu et al., 2021), we used Wide ResNet (WRN)-28-2 for CIFAR-10, WRN-28-8 for CIFAR-100, WRN-37for STL-10 and ResNet-18 for mini-Imagenet.
Since our methods are implemented as a plug-in module to FixMatch, common network hyper-parameters, e.g. , learning rates, batch-sizes, are the same as their original settings (Sohn et al., 2020).
For each dataset, our model and FixMatch are trained 2 17 iterations in MNAR and 2 20 steps in ordinary SSL cases ( γ = 1 ).
[Table]
+ CREST
[Table]
– – – – + DARP
[Table]
– – + CADR (
[Table]
[Table]
the unlabeled data balanced ( γ u = 1 ).
We keep N max = γ so that the least number of labeled data among all the classes is always 1.
5. R ESULTS AND A NALYSES
Comparison with State-of-The-Art Methods.
To demonstrate the effectiveness of our method, we compared it with multiple baselines using the same network architecture, including CREST (Wei et al., 2021) and DARP (Kim et al., 2020) that aims to handle unbalanced semi-supervised learning.
All the methods are implemented based on their public codes.
Table 1 shows the results on all the datasets with different levels of imbalanced label ratios.
While CREST and DARP fail to correct the severely biased learning process, our method outperforms all the baselines by large margins across all the settings.
The reason is that though CREST and DARP handle the unbalanced labeled data, they mainly focus on the case where the unlabeled data is equally unbalanced.
As the labeled and the unlabeled data share the same class distribution, the pseudo-labels they used are not as misleading as in our cases.
Thus the bias removal is not critically emphasized by their algorithms.
On CIFAR10 and CIFAR-100, we boost the mean accuracy of FixMatch by 24.41% and 11.26% on average.
On STL-10 and more challenging mini-ImageNet, our improvements are also substantial (21.38% and 5.86%).
Specifically, Figure 3 shows the convergence trend and confusion matrices under one
MNAR scenario ( γ = N max = 50 ).
As one can observe, the performance of FixMatch is boosted mainly due to the de-biased estimation for the labeled-data-rare classes.
Individual Effectiveness of Each Component.
Table 2 shows the results of using Class-Aware
Propensity (CAP) and Class-Aware Imputation (CAI) alone and together in trivial ( L CAP + L CAI , w/o CADR) and our DR combination ( L CAP + L CAI + L supp , w/ CADR).
We can observe that the improvement of using either CAP and CAI alone is distinguishable, demonstrating that both
CAP and CAI achieve bias-removed estimation.
Combining CAI and CAP, our CADR exhibits steady improvement over the baseline.
We found that our CADR is consistently among the best or second-best performance, agnostic to the value of γ and the performance gap between individual
[Figure]
s t i n g A cc u r a c y
[Figure]
u e L a
[Figure]
l
[Figure]
o l o r B a r
[Table]
components.
When the label data is more imbalanced, e.g. , γ = 100 , CAP outperforms CAI by large margins on CIFAR-100 (63.07% vs. 58.26%), STL-10 (77.69% vs. 71.21%), and mini-ImageNet (32.23% vs. 30.94%), and our CADR outperforms the trivial combination by 0.52% ∼ 1.26%.
The special case is γ = 100 on CIFAR-10, where the trivial combination slightly beats CADR (93.97% vs. 94.10%), and the gap between CAP and CAI is also small (93.96% vs. 93.86%).
These results empirically verified our theoretical analysis and the robustness of our CADR, that is , CADR maintains unbiasedness when either CAP or CAI is unbiased.
To sum up, CADR can provide a reliable solution with robust performance and bring convenience to real applications.
[Table]
Apart from
CREST and DARP, we further compare with some re-balancing techniques adopted in long-tailed fullysupervised learning in Table 3.
They are 1) reweighting (Cui et al., 2019): re-weighting labeled sample according to the inverse of the number of labeled data in each class, 2) re-sampling : re-sampling the labeled data to construct a balanced labeled set, and 3)
LA (Menon et al., 2020): using logits adjusted softmax cross-entropy loss that applies a label-dependent offset to logits for each labeled sample.
However, under MNAR, where the model is biased by supervised learning and unlabeled data imputation together, only dealing with the supervised process without considering the whole data distribution gives inferior results.
A recent work, DASH (Xu et al., 2021) also proposed to use a dynamic threshold in filtering imputed labels.
Compared to their work, our threshold is both dynamic and classaware, and our CADR outperforms DASH by a large margin.
More MNAR Settings.
In Figure 4, we show more different MNAR scenarios.
Figure 4(a) shows the case where the unlabeled data is imbalanced.
γ u denotes the imbalance ratio of unlabeled data, and those unlabeled data is inversely imbalanced (the rarest class in labeled data is the most fre- [Figure] quent in unlabeled data) when γ u < 1.
Apart from the artificially designed imbalanced distribution,
Figure 4(b) describes the results under the case where we label a random subset of data unaware of class distribution of the whole data.
Without strictly selecting labeled data following the class distribution, the class distribution of labeled data can also be different from the unlabeled data.
To show our effectiveness in producing unbiased predictions, we also report geometric mean accuracy (GM Accu) (Kubat, 2000; Kim et al., 2020).
The overall results demonstrate that our method is robust to the distribution perturbation of both labeled and unlabeled data.
Specifically, the significantly improved GM Accuracy shows that the class-wise accuracy obtained through our unbiased estimation is both high and in good balance.
Compatibility with Conventional Settings.
As discussed in Section 3, our method degrades to the original FixMatch implementation when labels are missing completely at random (MCAR).
Tabledemonstrates the compatibility to MCAR of our methods: our results are consistent with the performance of FixMatch when labeled data are uniformly sampled from each class.
Particularly, when provided with the extremely limited number of labeled data (CIFAR-10, #labels = 40), our performance is better than FixMatch (94.41% to 91.96%).
The reason is that compared to FixMatch, our method is more robust to the data variance among mini-batches with the moving averaged propensity in such few-shot labeled data learning.
[Table]
completely at random (MCAR).
Performances of Fixmatch are the reported results in their paper (Sohn et al., 2020).
#labels denotes the overall number of the labeled data.
Ablation Studies of Hyper-parameters.
We conducted ablation studies on two hyper-parameters: the moving average momentum µ for calculating class-aware propensity (Table 5), and the threshold scaling coefficient β for class-aware imputation (Table 6).
Table 5 shows the effectiveness of the moving average strategy comparing results of different momentum values, where“0” denotes no moving average.
As shown in Table 6, when β is too small (0.2), the requirement of confidence for rare classes lowers marginally, and thus the selected imputed labels still bear bias.
Specifically, using the distribution of the labeled data to re-scale the threshold is not promising, while our method dynamically updating the threshold with training is adaptive and effective.
[Table]
[Table]
CONCLUSION
In this work, we presented a principled approach to handle the non-random missing label problem in semi-supervised learning.
First, we proposed Class-Aware Propensity (CAP) to train an improved classifier using the biased labeled data.
Our CAP exploits the class distribution information of the unlabeled data to achieve unbiased missing label imputation.
Second, we proposed Class-Aware Imputation (CAI) that dynamically adjusts the threshold in filtering pseudo-labels of different classes.
Finally, we combined CAP and CAI into a doubly robust estimator (CADR) for the overall SSL model.
Under various label missing not at random (MNAR) settings for several image classification benchmarks, we demonstrated that our method gains a significant and robust improvement over existing baselines.
For future work, we are interested in incorporating 1) the selection bias theory (Bareinboim et al., 2014) of the missing data mechanisms and 2) the causal direction between label and data (K¨ugelgen et al., 2020), into semi-supervised learning.
A A PPENDIX A. I MPLEMENTATION D ETAILS
As mentioned in Section 5.1, we used almost identical hyper-parameters as FixMatch (Sohn et al., 2020) on CIFAR-10, CIFAR-100, STL-10 and mini-ImageNet.
Here, we provide a complete list of hyper-parameters in Table 7, where xx/xx denote the parameters in FixMatch/
Ours and xx is the common parameter. Notation
[Table]
For DARP (Kim et al., 2020), the original implementation needs abundant labeled data as the validation set to estimate the unlabeled data distribution, which is not practical in our case (the minimal number of labeled data of classes is 1).
As an alternative, we provide DARP with the ground truth unlabeled data distribution.
A. A LGORITHM
Algorithm 1: Input : D L , D U ▷ labeled and unlabeled data 2: Input : model θ 0 , strong augmentation A , threshold τ o 3:
Output : θ 4: Initialize θ 0 randomly, Iteration i = 0 , P ( Y ) is uniform 5: for i <
MaxIter do 6: { X L , Y L } ← D L , { X U } ← D U ▷ sample a mini-batch 7:
Output L ( θ ) ∪ Output U ( θ ) ← f θ ( { X L , Y L } ∪ { X U } )
▷ Model prediction 8:
P ( Y | X ; θ ) ← S OFTMAX ( Output L ( θ ) ∪ Output U ( θ ))
▷ softmax probability 9: # For supervised labeled data: 10:
P ( Y ) , P ( Y ; θ ) ← CAP ( Output L ( θ ) ∪ Output U ( θ ) , P ( Y ))
▷ CAP 11:
L s ( X L , Y L ) ← CAP-L OSS ( P ( Y L | X L ; θ ) , P ( Y ; θ ))
▷ Eq.
(9) 12: # For imputed unlabeled data: 13:
Q, con(Q) ← M AX ( Output U ( θ ))
▷ imputed label with confidence 14: τ ← τ o , P ( Y ) , Q ▷ Eq.
L u ( X U ) ← CAI-L OSS ( Output U ( θ ) , τ ) ▷ Eq.
# Model update: 17: θ ← θ − ∇ θ DR-L OSS ( L s ( X L , Y L ) , L u ( X U ) , X L , P ( Y ; θ ) , τ ) ▷ Eq.
(15) 18: end for 19: Return θ
A. D OUBLE R OBUSTNESS OF THE DR ESTIMATOR
Scenario 1 : CAP is correct, i.e. , the propensity p ( i ) successfully reflects the data missing mechanism.
We have:
[Equation]
The expectation of it over M is written as:
[Equation]
As E m ( i ) [1 − m ( i ) ]
= p ( i ) is the probability that a data is labeled, we have Eq.
18 equals 0.
Scenario 2 : CAI is correct, i.e. , the imputed label plays the same role as the true label:
[Equation]
I ( con ( q ( i ) ) > τ ) = 1 N (cid:88) i =1 , ··· ,N (1 − m ( i ) p ( i ) − (1 − m ( i ) )
L s ( x ( i ) , y ( i ) ))
− 1 N (cid:88) i =1 , ··· ,N (1 − m ( i ) p
( i ) − (1 − m ( i ) )
L u ( x ( i ) , q ( i ) )
I ( con ( q ( i ) ) > τ ))
= (cid:88) i
=1 , ··· ,N (1 − m ( i ) p
( i ) − (1 − m ( i ) )) ( L s ( x ( i ) , y ( i ) ) − L u ( x ( i ) , q ( i ) ) I ( con ( q ( i ) ) > τ )) , whose expectation equals to 0 as the second term in summation is expected to be 0 when imputation is ideal.
A. M ORE E XPERIMENTS A.4. M ORE C OMPARISONS OF L ABELING D EPENDENCE ON C LASSES
In this section, we performed experiments with the dependence of labeling on class varying in the intermediate range by increasing the labeling imbalance ration γ from 1 to 200 gradually.
As shown in Table 8, our method can consistently boost the performance of baseline FixMatch under different levels of label dependence.
Besides, the improvement is more significant with larger γ , i.e. , the high dependence of labeling on class.
This observation is reasonable since the baseline method gradually fails to handle the data distribution shift between the labeled and unlabeled data in challenging
MNAR problems.
Methods
[Table]
=
[Table]
[Table]
[Table]
[Table]
under different labeling dependence on the class.
We alter the imbalance ratio γ of the labeled data in the intermediate range from 1 to 200, where γ =is the case where the label is missing completely at random.
The experiments are conducted on
CIFAR-100, and we keep N max
= 200 across all settings.
[Figure]
[Figure]
A.4. A PPLICATION TO I N ATURALIST
To apply our methods to real-data MNAR occasions, we conducted experiments on the subsets of iNaturalist (Van Horn et al., 2018), a real-world dataset comprised of the natural images and labels collected from a citizen science website 1 .
iNaturalist has two popular versions, iNaturalist-2018 for long-tailed recognition 2 and iNaturalist-2021 for nearly balanced data recognition 3 .
As iNaturalist2021 supplements iNaturalist-2018 with abundant additional data in their overlapped classes, these two versions can be naturally used for our MNAR setting in SSL.
Specifically, we sampled N classes from the iNaturalist dataset, where the data in iNaturalist-2018 are used as the labeled data and the additionally released data in iNaturalist-2021 as the unlabeled.
Figure 5 depicts the details of our 20class subset, and the data distribution of 20/50-class subsets are shown in Figure 5(b) and Figureseparately, where an obvious imbalanced distribution over the labeled data is observed.
N =
Supervised 25.
[Table]
To train the dataset, we resize the images to 64 × 64 , remove the overlapped images and sample 50 images per class from iNaturalist2021 for performance evaluation, and train each network with
Wide ResNet (WRN)-28-2 by 2 15 iterations.
Other hyper-parameters are shown in Table 7.
The performance comparisons are shown in Table 9.
It shows that our proposed methods outperform the baseline FixMatch by large margins, demonstrating the effectiveness of our methods in handling real-world imbalanced labeled data.
A.4. M ORE A BLATION ON O THER B ASELINES
In this section, we perform the ablation experiments on more baseline methods, MixMatch (Berthelot et al., 2019b) and RemixMatch (Berthelot et al., 2019a).
As they do not have a threshold in label imputation, CAI is not directly applicable, and we only apply CAP on them.
As shown in
Table 10, our method consistently boosts the performance, especially outperforming the baselines by large margins on CIFAR-10 and STL-10.
[Table]