PUUPL: P OSITIVE -U NLABELED L EARNING WITH U NCERTAINTY - AWARE P SEUDO - LABEL S ELECTION
Anonymous authors
Paper under double-blind review
A BSTRACT
Positive-unlabeled (PU) learning aims at learning a binary classiﬁer from only positive and unlabeled training data.
Recent approaches addressed this problem via cost-sensitive learning by developing unbiased loss functions, and their performance was later improved by iterative pseudo-labeling solutions.
However, such two-step procedures are vulnerable to incorrectly estimated pseudo-labels, as errors are propagated in later iterations when a new model is trained on erroneous predictions.
To mitigate this issue we propose PUUPL , a new loss-agnostic training procedure for PU learning that incorporates epistemic uncertainty in pseudolabeling.
Using an ensemble of neural networks and assigning pseudo-labels based on high conﬁdence predictions improves the reliability of pseudo-labels, increasing the predictive performance of our method and leading to new state-ofthe-art results in PU learning.
With extensive experiments, we show the effectiveness of our method over different datasets, modalities, and learning tasks, as well as improved robustness over misspeciﬁcations of hyperparameters and biased positive data.
The source code of the method and all the experiments are available in the supplementary material.
I NTRODUCTION
Many real-world applications involve positive and unlabeled (PU) datasets in which only some of the data is labeled positive while the majority is unlabeled and contains both positives and negatives.
PU learning aims to learn a binary classiﬁer in this challenging setting without any labeled negative examples.
Learning from PU data can reduce deployment costs in many deep learning applications that otherwise require annotations from experts such as medical image diagnosis (Armenian and Lilienfeld, 1974) and protein function prediction (Gligorijevi ´ c et al., 2021), and it can even enable applications in settings where the measurement technology itself can not detect negative examples (Purcell et al., 2019).
Some recent approaches such as unbiased PU (Du Plessis et al., 2014, uPU) and non-negative PU (Kiryo et al., 2017, nnPU) formulate this problem as cost-sensitive learning.
Others approach PU learning as a two-step procedure ﬁrst identifying and labeling some reliable negative examples, and then re-training the model based on this newly constructed labeled dataset (Bekker and Davis, 2020).
These approaches show similarities with pseudo-labeling in semi-supervised classiﬁcation settings (Lee, 2013).
Such pseudo-labeling techniques are however especially vulnerable to incorrectly assigned labels of the selected examples as these errors will propagate and magnify in the retrained model, resulting in a negative feedback loop.
Worse yet, since the true labels are unavailable in PU learning, this situation is hard to detect by any metrics computed on the training set.
This erroneous selection of unreliable pseudo-labels occurs when wrong model predictions are associated with excessive model conﬁdence.
Such poor model calibration is accompanied by a distortion of the signal for the pseudo label selection (Van Engelen and Hoos, 2020).
In recent literature on pseudo-labeling, this problem is recognized and successfully addressed by explicitly estimating the prediction uncertainty (Abdar et al., 2021; Rizve et al., 2021; Arazo et al., 2020).
While this is the case for semi-supervised classiﬁcation, there does not yet exist a method that explores the use of uncertainty quantiﬁcation for pseudo-labeling in a PU learning context.
[Figure]
Contributions: Motivated by this, we propose a novel, uncertainty-aware pseudo-labeling framework for PU learning that uses established uncertainty quantiﬁcation techniques to identify reliable examples to pseudo-label (Fig. 1).
In particular, our contributions are: (1) We introduce PUUPL (Positive-Unlabeled, Uncertainty-Aware Pseudo-Labeling), a simple uncertainty-aware pseudolabeling framework for PU learning.
(2) PUUPL can use any loss function for PU learning, improving model performance while being robust to the speciﬁc data biases that the respective loss considers.
(3) We evaluate our methods on a wide range of benchmarks and PU datasets, achieving state-of-the-art performance in PU learning.
(4) Our extensive ablation studies provide new insights into uncertainty-aware pseudo-labeling for PU learning.
Further, they show that our method is robust to the choices of hyperparameters, with 1% or less variability in test accuracy among different choices as well as distribution shifts between labeled positives in the train and test datasets.
To the best of our knowledge, PUUPL is the ﬁrst framework for PU learning which leverages uncertainty information during pseudo-labeling.
R ELATED WORK
PU Learning PUL was introduced as a variant of binary classiﬁcation (Liu et al., 2003) and is related to one-class learning (Ruff et al., 2018; Li et al., 2010), multi-positive learning (Xu et al., 2017), multi-task learning (Kaji et al., 2018), and semi-supervised learning (Chapelle et al., 2009).
There exist three main research branches for PUL: two-step techniques, class prior incorporation, and biased PUL (Bekker and Davis, 2020).
In this work, we combine Pseudo-Labeling which has similarities to two-step techniques, with biased PUL, also coined as reweighting methods, and refer to Bekker and Davis (2020) for a comprehensive overview of the ﬁeld.
In this context, Du Plessis et al.
(2014) introduced the unbiased risk estimator uPU.
Kiryo et al.
(2017) showed this loss function is prone to overﬁtting in deep learning contexts as it lacks a lower bound and proposed the non-negative risk estimator nnPU as a remedy.
Follow-up work on loss functions for PUL has focused on robustness w.r.t biases in the sampling process such as PUSB (Kato et al., 2019)
, PUbN (Hsieh et al., 2019) or PULNS (Luo et al., 2021).
Uncertainty-aware Pseudo-Labeling Pseudo-labeling follows the rationale that the model leverages its own predictions on unlabeled data as pseudo training targets to enable iterative semisupervised model training.
The ﬁrst such approach for deep learning was introduced by Lee (2013), simply selecting the class with the highest predicted probability as a pseudo label.
One weakness of pseudo-labeling is that erroneously selected pseudo-labels can amplify for training, potentially leading to model degradation.
This is grounded in poor model calibration distorting the signal for the pseudo label selection (Van Engelen and Hoos, 2020).
Iscen et al.
(2019) try to mitigate this issue using conﬁdence and class weights.
Shi et al.
(2018) use conﬁdent scores based on the geometric neighborhood of the unlabeled samples while Arazo et al.
(2020) effectively tackle this conﬁrmation bias using Mixup (Zhang et al., 2017), Label Noise (Tanaka et al., 2018), and Entropy Regularization (Grandvalet et al., 2005).
Rizve et al.
(2021) introduced a pseudo-labeling framework using a weighting scheme for class balancing and MC dropout (Gal and Ghahramani, 2016) for calibration, while Beluch et al.
(2018) found deep ensembles (Lakshminarayanan et al., 2017a) to yield the best model calibration in an active learning context, especially in low-label regimes.
The commonality of these works is the explicit consideration of model uncertainty to improve pseudo-label selection, which motivates us to apply this in the context of PU learning.
Pseudo-Labeling for PU Learning Two-step approaches in PU learning ﬁrst identify negative samples from the unlabeled dataset, and then train a binary classiﬁcation model on the original dataset augmented with the newly identiﬁed negatives (Bekker and Davis, 2020).
These approaches share similarities with pseudo-labeling but lack an iterative feedback loop after the completion of the second step.
A ﬁrst attempt to combine pseudo-labeling with PU learning was made with Self-PU
(Chen et al., 2020b), where self-paced learning, a conﬁdence-weighting scheme based on the model predictions and a teacher-student distillation approach are combined.
Via this complex training scheme, Self-PU was shown to marginally outperform recent baselines.
With PUUPL , we propose an alternative PL strategy for PU learning that performs better in a simpler and more principled way using implicitly well-calibrated models to improve the pseudo-label selection.
Uncertainty-aware pseudo
-labeling for PU learning To the best of our knowledge, we are the ﬁrst to introduce an uncertainty-aware pseudo-labeling paradigm to PU learning.
Although our method shares the same motivation as that from Rizve et al. (2021) for semi-supervised classiﬁcation, we differ in several important aspects: (1) we speciﬁcally target PU data with a PU loss, (2) we quantify uncertainty with an ensemble instead of Monte Carlo dropout, (3) we use epistemic uncertainty instead of the predicted class probabilities for the selection, (4) we do not use temperature scaling, and (5) we use soft labels.
M ETHOD
We propose PUUPL (Positive Unlabeled, Uncertainty-aware Pseudo-Labeling), an iterative pseudolabeling procedure to progressively select and label the most conﬁdent examples from unlabeled data.
The pseudo-code for PUUPL is shown as Algorithm 1.
Our method separates the training set
X tr into the sets P , U , and L which contain the initial positives, the currently unlabeled, and the pseudo-labeled samples respectively.
The set L is initially empty.
At each pseudo-labeling iteration, we ﬁrst train our model using all samples in P , U , and L until some convergence condition is met (Section 3.2).
Then, samples in U are predicted and ranked w.r.t their predictive uncertainty (Section 3.3) and samples with the most conﬁdent score are assigned the predicted label and moved into the set L (Section 3.4).
Similarly, samples in L are also predicted and the most uncertain samples are moved back to the unlabeled set U (Section 3.5).
Next, the model is re-initialized to the same initial weights and a new pseudo-labeling iteration starts.
In the following, we ﬁrst describe the notation used in this paper and then explain in detail the training procedure of PUUPL .
3. N OTATION
Consider input samples X with label y and superscripts · tr , · va and · te for training, validation, and test data respectively.
The initial training labels y tr are set to one for all samples in P and zero for all others in U .
We group the indices of original positives, unlabeled, and pseudo-labeled samples
Input
: end in X tr into the sets P , U , and L respectively.
Our proposed model is an ensemble of K deep neural networks whose random initial weights are collectively denoted as θ 0 .
The predictions of the k -th network for sample i are indicated with ˆ p ik =
σ ( ˆ f ik ) , with σ ( · ) the logistic function and ˆ f ik the predicted logits.
The logits and predictions for a sample averaged across the networks in the ensemble are denoted by ˆ f i and ˆ p i respectively.
We subscript data and predictions with i to index individual samples, and use an index set in the subscript to index all samples in the set (e.g., X trU =
X tri | i ∈ U } denotes the features of all unlabeled samples).
We denote the total, epistemic and aleatoric uncertainty of sample i as ˆ u ti , ˆ u ei , and ˆ u ai , respectively.
3. L OSS FUNCTION
We train our proposed model with a loss function L that is a convex combination of a loss L PU for the samples in the positive and unlabeled set ( P ∪ U ) and a loss L L for the samples in the pseudo-labeled set ( L ):
[Equation]
with λ ∈ (0 , 1) .
The loss L L is the binary cross-entropy computed w.r.t the assigned pseudo-labels y .
Our model is agnostic to the speciﬁc PU loss L PU used.
This allows our method to be easily adapted to different scenarios for which a PU loss was proposed and improve over its performance, for example when coping with a selection bias in the positive examples (Kato et al., 2019) or the availability of a biased negative set (Hsieh et al., 2019).
For the standard setting of PU learning, we use the non-negative correction nnPU of the PU loss (Kiryo et al., 2017):
[Equation]
With π the prior probability of a sample being positive, which we assume known and can be estimated from PU data (du Plessis et al., 2016), and (cid:96) ( S, y )
the expected sigmoid loss of samples in the set S with label y :
[Equation]
3. M ODEL U NCERTAINTY
To quantify the predictive uncertainty, we utilize a deep ensemble with K networks with the same architecture, each trained on the full training dataset (Lakshminarayanan et al., 2017b).
Given the predictions ˆ p i 1 , . . .
, ˆ p iK for a sample x i , we associate three types of uncertainties to x i ’s predictions (H¨ullermeier and Waegeman, 2021): the aleatoric uncertainty as the mean of the entropy of the predictions (Eq. 4), the total uncertainty as the entropy of the mean prediction (Eq. 5), and the epistemic uncertainty formulated as the difference between the two (Eq. 6).
[Equation]
Epistemic uncertainty corresponds to the mutual information between the parameters of the model and the true label of the sample.
Low epistemic uncertainty thus means that that the model parameters would not change signiﬁcantly if trained on the true label, suggesting that the prediction is indeed correct.
Using such a prediction as target in the cross entropy loss would in turn provide a stronger, more explicit learning signal to the model, so that a correctly pseudo-labeled example provides a larger decrease in risk compared to using the same example without any label within the positive-unlabeled loss.
3. P SEUDO - LABELING
The estimated epistemic uncertainty (Eq. 6) is used to rank samples of the unlabeled set U and to select reliable samples for pseudo-labeling.
Next, the predictions of the unlabeled samples U are ranked according to their epistemic uncertainty (Eq. 6).
Let ρ ( i ) denote the rank of sample i , then the set L new of newly pseudo-labeled samples is formed by taking the T samples with lowest uncertainty from U , ensuring that it is lower than the threshold t l :
[Equation]
Previous works have shown that balancing the pseudo-label selection between the two classes, i.e., ensuring that the ratio of newly labeled positives and negatives is close to a given target ratio r , is beneﬁcial (Rizve et al., 2021).
In this case, the set L new should be partitioned according to the model’s predictions into a set L new + of predicted positives and L new − of predicted negatives, and the most uncertain samples in the larger set should be discarded to reach the desired ratio r , which we ﬁx to 1 .
We then assign soft pseudo-labels, i.e., the average prediction in the open interval (0 , 1) , to these samples:
[Equation]
3. P SEUDO - UNLABELING
Similar to the way that low uncertainty on an unlabeled example indicates that the prediction can be trusted, high uncertainty on a pseudo-labeled example indicates that the assigned pseudo-label might not be correct after all.
To avoid training on such possibly incorrect pseudo-labels, we move the pseudo-labeled examples with uncertainty above a threshold t u back into the unlabeled set:
[Equation]
[Figure]
V a li d a t i o n
A cc u r a c y 0.
E x p e c t e d
C a li b r a t
i
o n
E rr o r
E XPERIMENTS 4. E XPERIMENTAL PROTOCOL
To empirically compare our proposed framework to existing state-of-the-art losses and models, we followed standard protocols for PU learning (Chen et al., 2020b; Kiryo et al., 2017; Kato et al., 2019).
Datasets: We evaluated our method in the standard setting of MNIST (Deng, 2012) and CIFAR
(Krizhevsky et al., 2009) datasets, as well as Fashion MNIST (F-MNIST)
(Xiao et al., 2017),
STL-10 (Coates et al., 2011) and IMDb (Maas et al., 2011) datasets to show the applicability to different data modalities.
Similar to previous studies (Chen et al., 2020b; Kiryo et al., 2017; Kato et al., 2019), positives were deﬁned as odd digits in MNIST, vehicles in CIFAR-10 and STL-10, and we used trousers, dress, sandals, sneaker, and ankle boots for F-MNIST and positive reviews for IMDb.
For STL-10 we used all available labeled and unlabeled data and the ofﬁcial ten crossvalidation folds.
For all other datasets, we reserved a validation set of 5,000 samples and use all other samples for training with 1,000 randomly chosen labeled positives, as is common practice, and evaluated on the canonical test set of each dataset.
More details are provided in Appendix B
Network architectures:
To ensure a fair comparison with other works in PU learning (Chen et al., 2020b; Kiryo et al., 2017) we used the same architectures on the same datasets, namely a 13-layers convolutional neural network for the experiments on CIFAR-10 (Table A.3) and a MLP with four fully connected hidden layers of 300 neurons each and ReLU activation for MNIST and F-MNIST.
For IMDb we used a bidirectional LSTM network with a MLP head whose architecture was optimized as part of the hyperparameter search.
Training: We trained all models with the Adam optimizer (Kingma and Ba, 2015) with β 1 = 0 .and
β 2 = 0 .
999 , and an exponential learning rate decay with γ = 0 .
We further used the nnPU loss (Kiryo et al., 2017) as L PU (Eq. 1) unless otherwise stated.
As is common in the pseudolabeling literature (Chen et al., 2020b; Rizve et al., 2021;
Kato et al., 2019; Tanaka et al., 2018;
Hu et al., 2021), we assume that a positive and negative labeled validation set is available and use this validation set for early stopping, i.e., stop the pseudo-labeling loop when the model’s accuracy on this set has stopped improving, and use the parameters that achieved the highest validation accuracy to compute the test performance.
An experiment will show how this requirement can be relaxed in practice.
Hyperparameter tuning: We used the Hyperband algorithm (Li et al., 2017) to optimize all hyperparameters on the CIFAR-10 dataset with η = 3 and S = 4 , using the validation accuracy as the criterion to be optimized.
The conﬁguration that achieved the highest validation accuracy was
[Table]
then used as a basis for the ablation studies and ﬁne-tuned on the remaining datasets to show that the pseudo-labeling hyperparameters (Table A.1) do not require tuning when transferred to other datasets and data modalities.
Speciﬁcally, on the other datasets we only tuned hyperparameters related to network training such batch size, learning rate, weight decay, number of training epochs, dropout probability and other details of the network architecture by running only the ﬁrst bracket of
Hyperband with η = 3 and S = 3 .
Evaluation: the best conﬁguration found by Hyperband was trained ﬁve times with different random training/validation splits and evaluated on the test set to produce the ﬁnal results, except for STLwhere we used the ofﬁcial ten cross-validation folds.
4. R ESULTS
The best performance achieved by our method is shown in Table 1 and compared with SelfPU (Chen et al., 2020b), DAN (Liu et al., 2019) and PAN (Hu et al., 2021).
PUUPL was always able to improve over the baseline nnPU loss, with larger gap for more difﬁcult datasets such as CIFAR-10 (+0.98%) and IMDb (+1.89%) as well as over SelfPU (Chen et al., 2020b) and DAN (Liu et al., 2019), setting a new state-of-the-art for PU learning.
Moreover, PUUPL is naturally very well calibrated despite the absence of explicit calibration on labeled data (Fig. 2), making its predictions inherently reliable.
The best pseudo-labeling hyperparameters constitute the defaults we suggested in Algorithm 1 and are K = 2 , T = 1000 , t l = 0 .
05 and t u = 0 .
Note that the baseline nnPU scores reported in Table 1 were also obtained by training an ensemble of two networks with the nnPU loss, thus possibly explaining the discrepancy observed with SelfPU.
The best network architecture for IMDb is shown in Table A.2.
4. A BLATION STUDIES
We performed ablation studies on the CIFAR-10 dataset by changing one parameter at a time of the best conﬁguration found by Hyperband, training and evaluating with ﬁve different splits and reporting the test accuracy corresponding to the best validation score for each run.
To limit the computational resources needed, we used at most 15 pseudo-labeling iterations.
Weights initialization:
We conﬁrmed the observation that it is beneﬁcial to re-initialize the weights after each pseudo-labeling step (Arazo et al., 2020), with slightly better performance ( +0 . 052% ) achieved when the weights are re-initialized to the same values before every pseudo-labeling iteration (Fig. 3a).
We believe this encourages the model to be consistent across pseudo-labeling rounds.
Pseudo-label assignment: Soft pseudo-labels were preferred over hard ones ( +0 . 75% ).
We found that our model was very well calibrated with ECEs as low as 0.05 on the labeled validation data (Fig. 2), indicating that the soft pseudo-labels they estimated were reliable training targets and that post hoc calibration was not necessary.
Contrary to expectation, however, re-assigning all pseudolabels at every iteration harmed performance ( − 0 .
12% ); instead, pseudo-labels should be kept ﬁxed after being assigned for the ﬁrst time.
A possible explanation is that ﬁxed pseudo-labels prevent the [Figure] [Figure] model’s predictions from drifting too far away from the initial pseudo-labeling towards an incorrect assignment.
It was also beneﬁcial to assign the same number of positive and negative pseudo-labels (Fig. 3b) compared to keeping the same ratio π of positives and negatives found in the whole dataset ( − 0 .
20% ) or not balancing the selection at all ( − 0 .
Uncertainty: Ranking predictions by aleatoric performance was almost as good as ranking by epistemic uncertainty ( − 0 .
08% ), while total uncertainty produced moderately worse rankings ( − 0 .
Fig.
3c).
An ensemble with only two networks achieved the best performance, while larger ensembles performed worse, and Monte Carlo dropout ( − 0 .
85% ) was better than ensembles of ﬁve ( − 1 . 00% ) and ten networks ( − 1 .
Early stopping: Finally, performing early stopping on the validation PU loss resulted in worse accuracy ( − 1 .
compared to using the accuracy on positive-negative labels (Fig. 3d).
Although considerable when compared to the impact of other algorithmic choices, such performance drop indicates that PUUPL can be used effectively in real-world scenarios with no labeled validation data available.
4. R OBUSTNESS
Following the same protocol as the ablation studies in Section 4.3, we tested the robustness of our method with respect to misspeciﬁcations of the continuous hyperparameters (Fig. 4).
Pseudo-labeling: Our method was fairly robust to the maximum number T of assigned pseudolabels and the maximum uncertainty threshold t l for the pseudo-labels, with almost constant performance up to T = 1000 and t l = 0 .
The best performance was achieved by the combination having T = 1000 and t l = 0 .
05 , but both of these experiments were performed while disabling the other constraint (i.e., setting T = inf when testing t l and vice-versa).
Using only a constraint on T resulted in a reduction of − 0 .
11% , while constraining t l alone resulted in a reduction of − 1 .
The results for t u were less conclusive as for the general trend, possibly because values lower than 0.35 require more than the 15 pseudo-labeling iterations we used for the experiment, and values above 0.4 did not show signiﬁcant differences.
Misspeciﬁcation of the class prior: The performance of our framework slowly degraded as the prior π moved further from the true value of 0.4 with a performance reduction of less than 2.5% in the range [0 . 3 , 0 . 6] (Fig. 4a).
Furthermore, the performance gap between PUUPL and nnPU widens as π is more severly misspeciﬁed.
Modern losses for PU learning such as uPU and nnPU rely on the [Table] correct estimation of the positive class prior π from domain knowledge or a priori estimation of π , which constitutes a whole research branch in PU learning (Chen et al., 2020a) and is a signiﬁcant challenge in any practical PU application (Bekker and Davis, 2020; Chen et al., 2020a).
We believe that the inclusion of epistemic uncertainty, the usage of soft labels and the convex combination of two losses enables PUUPL to be considerably more robust to signiﬁcant misspeciﬁcation of the class prior π .
Loss combination: The best performing combination had λ = 0 .
15 , with modest performance reduction until λ = 0 .
5 ( − 0 .
25% , Fig. 4b).
Values of 0.05 and below resulted in the same performance reduction of -0.5%, similarly to λ = 0 . 75 , and performance was 1.09% worst at λ = 0 .
Too large λ might facilitate the emergence of a harmful conﬁrmation bias, but it is nonetheless important to train on the pseudo-labels, too, to avoid losing the information contained therein.
Number of training labeled positives:
The performance of our method steadily increased and seemed to plateau at 91.4% between 3,000 and 6,000 labeled positives.
The gap between nnPU and
PUUPL is largest in the low labeled data region with a 1.44% gap at 250 labels, where we achieved 87.59% accuracy, shrinking to a gap of 0.52% with 3,000 labels, where our performance was 91.44% (Fig. 4c).
This supports our intuition about the importance of uncertainty because, as the amount of labeled data decreases, uncertainty becomes more important to detect overﬁtting and to prevent the model from assigning incorrect pseudo-labels.
Positive bias: The most general assumption of PU learning is that the labeled examples are a biased sample from the positive distribution (Bekker and Davis, 2020).
We tested PUUPL in such a biased setting where positives in the training and validation sets were with 50% chance an airplane, 30% chance an automobile, 15% chance ship and 5% chance truck, while in previous experiments the positives were evenly composed of airplanes, automobiles, ships and trucks.
The test distribution was unchanged, meaning that test samples are half as likely to be airplanes compared to the training set, and ﬁve times more likely to be to be truck images.
We also ﬁxed all hyperparameters to the values identiﬁed previously, except for the loss L PU where we used the nnPUSB loss (Kato et al., 2019) to handle the positive bias.
The baseline with nnPUSB loss performed better than the nnPU loss (+0.26%), but worse than PUUPL with the nnPU loss (-0.39%), highlighting the beneﬁt of our uncertainty-aware approach.
The best performance was however achieved with PUUPL on top of the nnPUSB loss (+0.21% compared to nnPU and -2.27% compared to the unbiased setting), showing that PUUPL can leverage the advantages of different PU losses and further improve on them (Table 2).
C ONCLUSIONS
In this paper, we proposed an uncertainty-aware pseudo-labeling framework for PU learning which quantiﬁes the epistemic uncertainty of an ensemble of networks and selects which examples to pseudo-label based on their predictive uncertainty.
We demonstrated the beneﬁts of our approach on different data modalities and biased settings, achieving state-of-the-art performance in all our benchmarks.
We further conducted extensive ablation studies and investigated the robustness of our approach, showing it to be reliable in settings that are likely to be encountered in the real world, such as a bias in the positive data, the unavailability of labeled negatives as validation data and the misspeciﬁcation of the class prior π .
Ethics statement: Most of the ethical concerns stem from the speciﬁc application and dataset.
Here we have shown a certain robustness towards biased positive labels without providing a comprehensive assessment, therefore practitioners should always ensure, insofar as possible, that the obtained predictions are ”fair” (with ”fairness” deﬁned appropriately w.r.t.
the target application) and do not systematically affect particular subsets of the population of interest.
Reproducibility statement: The source code for the framework is available in the supplementary material.
A PPENDIX A N ETWORK ARCHITECTURE AND HYPERPARAMETERS
Table A.1 reports the hyperparameters related to pseudo-labeling and their ranges.
Table A.2 reports the network architecture used in the IMDb experiments, while Table A.3 reports the network used with CIFAR-10.
B D ATASET INFORMATION
Table B.1 reports the number of samples for each split and each dataset.
For the image datasets, we subtracted the mean pixel intensity in the training set and divided by the standard deviation.
For
IMDb we used pre-trained GloVe embeddings of size 200 on a corpus of six billion tokens.
[Table] [Table]