Original: O N THE I MPORTANCE OF C ALIBRATION IN S EMI -S UPERVISED L EARNING
Revised: O N THE I MPORTANCE OF C ALIBRATION IN S EMI -S UPERVISED L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: State-of-the-art (SOTA) semi-supervised learning (SSL) methods have been highly successful in leveraging a mix of labeled and unlabeled data by combining techniques of consistency regularization and pseudo-labeling.
Revised: State-of-the-art (SOTA) semi-supervised learning (SSL) methods have been highly successful in leveraging a mix of labeled and unlabeled data by combining techniques of consistency regularization and pseudo-labeling.

Original: During pseudo-labeling, the model’s predictions on unlabeled data are used for training and thus, model calibration is important in mitigating confirmation bias.
Revised: During pseudo-labeling, the model’s predictions on unlabeled data are used for training and thus, model calibration is important in mitigating confirmation bias.

Original: Yet, many SOTA methods are optimized for model performance, with little focus directed to improve model calibration.
Revised: Yet, many SOTA methods are optimized for model performance, with little focus directed to improve model calibration.

Original: In this work, we empirically demonstrate that model calibration is strongly correlated with model performance and propose to improve calibration via approximate Bayesian techniques.
Revised: In this work, we empirically demonstrate that model calibration is strongly correlated with model performance and propose to improve calibration via approximate Bayesian techniques.

Original: We introduce a family of new SSL models that optimizes for calibration and demonstrate their effectiveness across standard vision benchmarks of CIFAR-10, CIFAR-100 and ImageNet, giving up to 15.9% improvement in test accuracy.
Revised: We introduce a family of new SSL models that optimizes for calibration and demonstrate their effectiveness across standard vision benchmarks of CIFAR-10, CIFAR-100 and ImageNet, giving up to 16.2% improvement in test accuracy on the CIFAR-100-400-labels benchmark.

Original: Furthermore, we also demonstrate their effectiveness in additional realistic and challenging problems, such as class-imbalanced datasets and in photonics science.
Revised: Furthermore, we also demonstrate their effectiveness in additional realistic and challenging problems, such as class-imbalanced datasets and in photonics science.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: While deep learning has achieved unprecedented success in recent years, its reliance on vast amounts of labeled data remains a long standing challenge.
Revised: While deep learning has achieved unprecedented success in recent years, its reliance on vast amounts of labeled data remains a long standing challenge.

Original: Semi-supervised learning (SSL) aims to mitigate this by leveraging unlabeled samples in combination with a limited set of annotated data.
Revised: Semi-supervised learning (SSL) aims to mitigate this by leveraging unlabeled samples in combination with a limited set of annotated data.

Original: In computer vision, two powerful techniques that have emerged are pseudo-labeling (also known as self-training)
Revised: In computer vision, two powerful techniques that have emerged are pseudo-labeling (also known as self-training)

Original: (Rosenberg et al., 2005; Xie et al., 2019b) and consistency regularization (Bachman et al., 2014; Sajjadi et al., 2016).
Revised: (Rosenberg et al., 2005; Xie et al., 2019b) and consistency regularization (Bachman et al., 2014; Sajjadi et al., 2016).

Original: Broadly, pseudo-labeling is the technique where artificial labels are assigned to unlabeled samples, which are then used to train the model.
Revised: Broadly, pseudo-labeling is the technique where artificial labels are assigned to unlabeled samples, which are then used to train the model.

Original: Consistency regularization enforces that random perturbations of the unlabeled inputs produce similar predictions.
Revised: Consistency regularization enforces that random perturbations of the unlabeled inputs produce similar predictions.

Original: These two techniques are typically combined by minimizing the cross-entropy between pseudo-labels and predictions that are derived from differently augmented inputs, and have led to strong performances on vision benchmarks (Sohn et al., 2020; Assran et al., 2021).
Revised: These two techniques are typically combined by minimizing the cross-entropy between pseudo-labels and predictions that are derived from differently augmented inputs, and have led to strong performances on vision benchmarks (Sohn et al., 2020; Assran et al., 2021).

Original: Intuitively, given that pseudo-labels (i.e. the model’s predictions for unlabeled data) are used to drive training objectives, the calibration of the model should be of paramount importance.
Revised: Intuitively, given that pseudo-labels (i.e. the model’s predictions for unlabeled data) are used to drive training objectives, the calibration of the model should be of paramount importance.

Original: Model calibration (Guo et al., 2017) is a measure of how a model’s output truthfully quantifies its predictive uncertainty, i.e. it can be understood as the alignment between its prediction confidence and its groundtruth accuracy.
Revised: Model calibration (Guo et al., 2017) is a measure of how a model’s output truthfully quantifies its predictive uncertainty, i.e. it can be understood as the alignment between its prediction confidence and its groundtruth accuracy.

Original: In some SSL methods, the model’s confidence is used as a selection metric (Lee, 2013;
Revised: In some SSL methods, the model’s confidence is used as a selection metric (Lee, 2013;

Original: Sohn et al., 2020) to determine pseudo-label acceptance, further highlighting the need for proper confidence estimates.
Revised: Sohn et al., 2020) to determine pseudo-label acceptance, further highlighting the need for proper confidence estimates.

Original: Even outside this family of methods, the use of cross-entropy minimization objectives common in SSL implies that models will naturally be driven to output high-confidence predictions (Grandvalet & Bengio, 2004).
Revised: Even outside this family of methods, the use of cross-entropy minimization objectives common in SSL implies that models will naturally be driven to output high-confidence predictions (Grandvalet & Bengio, 2004).

Original: Having high-confidence predictions is highly desirable in
Revised: Having high-confidence predictions is highly desirable in

Original: SSL since we want the decision boundary to lie in low-density regions of the data manifold, i.e. away from labeled data points (Murphy, 2022).
Revised: SSL since we want the decision boundary to lie in low-density regions of the data manifold, i.e. away from labeled data points (Murphy, 2022).

Original: However, without proper calibration, a model would easily become over-confident.
Revised: However, without proper calibration, a model would easily become over-confident.

Original: This is highly detrimental as the model would be encouraged to reinforce its mistakes, resulting in the phenomenon commonly known as confirmation bias (Arazo et al., 2019).
Revised: This is highly detrimental as the model would be encouraged to reinforce its mistakes, resulting in the phenomenon commonly known as confirmation bias (Arazo et al., 2019).

Original: Despite the fundamental importance of calibration in SSL, many state-of-the-art (SOTA) methods have thus far been empirically driven and optimized for performance, with little focus on techniques that specifically target improving calibration to mitigate confirmation bias.
Revised: Despite the fundamental importance of calibration in SSL, many state-of-the-art (SOTA) methods have thus far been empirically driven and optimized for performance, with little focus on techniques that specifically target improving calibration to mitigate confirmation bias.

Original: In this work, we explore the generality of the importance of calibration in SSL by focusing on two broad families of SOTA
Revised: In this work, we explore the generality of the importance of calibration in SSL by focusing on two broad families of SOTA

Revised: [Figure]

Original: SSL methods that both use pseudo-labeling and consistency regularization: 1) threshold-mediated methods (Sohn et al., 2020;
Revised: SSL methods that both use pseudo-labeling and consistency regularization: 1) threshold-mediated methods (Sohn et al., 2020;

Original: Xie et al., 2019a; Lee, 2013) where the model selectively accepts pseudolabels whose confidence exceed a threshold and 2) “representation learning” methods adopted from self-supervised learning (Assran et al., 2021)
Revised: Xie et al., 2019a; Lee, 2013) where the model selectively accepts pseudolabels whose confidence exceed a threshold and 2) “representation learning” methods adopted from self-supervised learning (Assran et al., 2021)

Original: where pseudo-labels are non-selective and training consists of two sequential stages of representation learning and fine-tuning.
Revised: where pseudo-labels are non-selective and training consists of two sequential stages of representation learning and fine-tuning.

Original: To motivate our work, we first empirically show that strong baselines like FixMatch (Sohn et al., 2020) and PAWS (Assran et al., 2021), from each of the two families, employ a set of indirect techniques to implicitly maintain calibration and that achieving good calibration is strongly correlated to improved performance.
Revised: To motivate our work, we first empirically show that strong baselines like FixMatch (Sohn et al., 2020) and PAWS (Assran et al., 2021), from each of the two families, employ a set of indirect techniques to implicitly maintain calibration and that achieving good calibration is strongly correlated to improved performance.

Original: Furthermore, we demonstrate that it is not straightforward to control calibration via such indirect techniques.
Revised: Furthermore, we demonstrate that it is not straightforward to control calibration via such indirect techniques.

Original: To remedy this issue, we propose techniques that are directed to explicitly improve calibration, by leveraging approximate Bayesian techniques, such as approximate Bayesian neural networks (Blundell et al., 2015) and weight-ensembling approaches (Izmailov et al., 2018). 
Revised: To remedy this issue, we propose techniques that are directed to explicitly improve calibration by using approximate Bayesian techniques that are designed to capture model uncertainty. In our work, we explored approximate Bayesian neural networks (Blundell et al., 2015) and weight-ensembling approaches (Izmailov et al., 2018). Such

Original: Our modification forms a new family of SSL methods that improve upon the SOTA on both standard benchmarks and real-world applications.
Revised: Our modification forms a new family of SSL methods that improve upon the SOTA on both standard benchmarks and real-world applications.

Original: Our contributions are summarized as follows:
Revised: Our contributions are summarized as follows:

Original: Using SOTA SSL methods as case studies, we empirically show that maintaining good calibration is strongly correlated to better model performance in SSL.
Revised: Using SOTA SSL methods as case studies, we empirically show that maintaining good calibration is strongly correlated to better model performance in SSL.

Original: We propose to use approximate Bayesian techniques to directly improve calibration and provide theoretical results on generalization bounds for SSL to motivate our approach.
Revised: We propose to use approximate Bayesian techniques to directly improve calibration and provide theoretical results on generalization bounds for SSL to motivate our approach.

Original: We introduce a new family of methods that improves calibration via Bayesian model averaging and weight-averaging techniques and demonstrate their improvements upon a variety of SOTA SSL methods on standard benchmarks including CIFAR-10, CIFARand ImageNet, notably giving up to 15.9% gains in test accuracy. 
Revised: We introduce a new family of methods, BAM-, that improves calibration via BAyesian model averaging (see Fig. 1) and demonstrate their improvements upon a variety of SOTA SSL methods on standard benchmarks, notably giving up to 16.2% gains in test accuracy. We further explored weight averaging techniques, one of which (i e. EMA) being wellestablished in SSL and show that their effectiveness can be understood from improving pseudo-label calibration.

Original: We further demonstrate the efficacy of our proposed model calibration methods in more challenging and realistic scenarios, such as class-imbalanced datasets and a real-world application in photonic science.
Revised: We further demonstrate the efficacy of BAM- in more challenging and realistic scenarios, such as class-imbalanced datasets and a real-world application in photonic science.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Semi-supervised learning (SSL) and confirmation bias.
Revised: Semi-supervised learning (SSL) and confirmation bias.

Original: A fundamental problem in SSL methods based on pseudo-labeling (Rosenberg et al., 2005) is that of confirmation bias (Tarvainen & Valpola, 2017; Murphy, 2022), i.e. the phenomenon where a model overfits to incorrect pseudo-labels.
Revised: A fundamental problem in SSL methods based on pseudo-labeling (Rosenberg et al., 2005) is that of confirmation bias (Tarvainen & Valpola, 2017; Murphy, 2022), i.e. the phenomenon where a model overfits to incorrect pseudo-labels.

Original: Several strategies have emerged to tackle this problem; Guo et al.
Revised: Several strategies have emerged to tackle this problem; Guo et al.

Original: and Ren et al.
Revised: and Ren et al.

Original: (2020) looked into weighting unlabeled samples, Thulasidasan et al.
Revised: (2020) looked into weighting unlabeled samples, Thulasidasan et al.

Original: and Arazo et al.
Revised: and Arazo et al.

Original: (2019) proposes to use augmentation strategies like MixUp (Zhang et al., 2017), while Cascante-Bonilla et al.
Revised: (2019) proposes to use augmentation strategies like MixUp (Zhang et al., 2017), while Cascante-Bonilla et al.

Original: (2020) proposes to re-initialize the model before every iteration to overcome confirmation bias.
Revised: (2020) proposes to re-initialize the model before every iteration to overcome confirmation bias.

Original: Another popular technique is to impose a selection metric (Yarowsky, 1995) to retain only the highest quality pseudo-labels, commonly realized via a fixed threshold on the maximum class probability (Xie et al., 2019a; Sohn et al., 2020).
Revised: Another popular technique is to impose a selection metric (Yarowsky, 1995) to retain only the highest quality pseudo-labels, commonly realized via a fixed threshold on the maximum class probability (Xie et al., 2019a; Sohn et al., 2020).

Original: Recent works have further extended such selection metrics to be based on dynamic thresholds, either in time (Xu et al., 2021) or class-wise (Zou et al., 2018; Zhang et al., 2021).
Revised: Recent works have further extended such selection metrics to be based on dynamic thresholds, either in time (Xu et al., 2021) or class-wise (Zou et al., 2018; Zhang et al., 2021).

Original: Different from the above approaches, our work proposes to overcome confirmation bias in
Revised: Different from the above approaches, our work proposes to overcome confirmation bias in

Original: SSL by directly improving the calibration of the model through approximate Bayesian techniques.
Revised: SSL by directly improving the calibration of the model through approximate Bayesian techniques.

Original: Model calibration and uncertainty quantification.
Revised: Model calibration and uncertainty quantification.

Original: Proper estimation of a network’s prediction uncertainty is of practical importance (Amodei et al., 2016) and has been widely studied.
Revised: Proper estimation of a network’s prediction uncertainty is of practical importance (Amodei et al., 2016) and has been widely studied.

Original: A common approach to improve uncertainty estimates is via Bayesian marginalization (Wilson & Izmailov, 2020), i.e. by weighting solutions by their posterior probabilities.
Revised: A common approach to improve uncertainty estimates is via Bayesian marginalization (Wilson & Izmailov, 2020), i.e. by weighting solutions by their posterior probabilities.

Original: Since exact Bayesian inference is computationally intractable for neural networks, a series of approximate Bayesian methods have emerged, such as variational methods (Graves, 2011; Blundell et al., 2015; Kingma et al., 2015),
Revised: Since exact Bayesian inference is computationally intractable for neural networks, a series of approximate Bayesian methods have emerged, such as variational methods (Graves, 2011; Blundell et al., 2015; Kingma et al., 2015),

Original: Hamiltonian methods (Springenberg et al., 2016) and Langevin diffusion methods (Welling & Teh, 2011).
Revised: Hamiltonian methods (Springenberg et al., 2016) and Langevin diffusion methods (Welling & Teh, 2011).

Original: Other methods to achieve Bayesian marginalization also exist, such as deep ensembles (Lakshminarayanan et al., 2016) and efficient versions of them (Wen et al., 2020; Gal & Ghahramani, 2015), which have been empirically shown to improve uncertainty quantification.
Revised: Other methods to achieve Bayesian marginalization also exist, such as deep ensembles (Lakshminarayanan et al., 2016) and efficient versions of them (Wen et al., 2020; Gal & Ghahramani, 2015), which have been empirically shown to improve uncertainty quantification.

Original: The concept of uncertainty and calibration are inherently related, where calibration is commonly interpreted as the frequentist notion of uncertainty.
Revised: The concept of uncertainty and calibration are inherently related, where calibration is commonly interpreted as the frequentist notion of uncertainty.

Original: In our work, we will adopt some of these techniques specifically for the context of semi-supervised learning in order to improve model calibration during pseudo-labeling.
Revised: In our work, we will adopt some of these techniques specifically for the context of semi-supervised learning in order to improve model calibration during pseudo-labeling.

Original: While other methods for improving model calibration exists (Platt, 1999; Zadrozny & Elkan, 2002; Guo et al., 2017), these are most commonly achieved in a post-hoc manner using a held-out validation set; instead, we seek to improve calibration during training and with a scarce set of labels.
Revised: While other methods for improving model calibration exists (Platt, 1999; Zadrozny & Elkan, 2002; Guo et al., 2017), these are most commonly achieved in a post-hoc manner using a held-out validation set; instead, we seek to improve calibration during training and with a scarce set of labels.

Original: Finally, in the intersection of SSL and calibration, Rizve et al.
Revised: Finally, in the intersection of SSL and calibration, Rizve et al.

Original: (2021) proposes to leverage uncertainty to select a better calibrated subset of pseudo-labels.
Revised: (2021) proposes to leverage uncertainty to select a better calibrated subset of pseudo-labels.

Original: Our work builds on a similar motivation, however, in addition to improving the selection metric with uncertainty estimates, we further seek to directly improve calibration via Bayesian marginalization (i.e. averaging predictions).
Revised: Our work builds on a similar motivation, however, in addition to improving the selection metric with uncertainty estimates, we further seek to directly improve calibration via Bayesian marginalization (i.e. averaging predictions).

Original:  N OTATION AND B ACKGROUND
Revised:  N OTATION AND B ACKGROUND

Original: Given a small amount of labeled data L = { ( x l , y l ) } N l l =1 (here, y l ∈ { 0 , 1 } K , are one-hot labels) and a large amount of unlabeled data U = { x u } N u u =1 , i.e. N u ≫ N l , in SSL, we seek to perform a K -class classification task.
Revised: Given a small amount of labeled data L = { ( x l , y l ) } N l l =1 (here, y l ∈ { 0 , 1 } K , are one-hot labels) and a large amount of unlabeled data U = { x u } N u u =1 , i.e. N u ≫ N l , in SSL, we seek to perform a K -class classification task.

Original: Let f ( · , θ f ) be a backbone encoder (e.g. ResNet or WideResNet) with trainable parameters θ f , h ( · , θ h ) be a linear classification head, and H denote the standard cross-entropy loss.
Revised: Let f ( · , θ f ) be a backbone encoder (e.g. ResNet or WideResNet) with trainable parameters θ f , h ( · , θ h ) be a linear classification head, and H denote the standard cross-entropy loss.

Original: Threshold-mediated methods.
Revised: Threshold-mediated methods.

Original: Threshold-mediated methods such as Pseudo-Labels (Lee, 2013),
Revised: Threshold-mediated methods such as Pseudo-Labels (Lee, 2013),

Original: UDA (Xie et al., 2019a) and FixMatch (Sohn et al., 2020) minimizes a cross-entropy loss on augmented copies of unlabeled samples whose confidence exceeds a pre-defined threshold.
Revised: UDA (Xie et al., 2019a) and FixMatch (Sohn et al., 2020) minimizes a cross-entropy loss on augmented copies of unlabeled samples whose confidence exceeds a pre-defined threshold.

Original: Let αand α 2 denote two augmentation transformations and their corresponding network predictions for sample x to be q 1 = h ◦ f
Revised: Let αand α 2 denote two augmentation transformations and their corresponding network predictions for sample x to be q 1 = h ◦ f

Original: ( α 1 ( x )) and q 2 = h ◦ f
Revised: ( α 1 ( x )) and q 2 = h ◦ f

Original: ( α 2 ( x )) , the total loss on a batch of unlabeled data has the following form:
Revised: ( α 2 ( x )) , the total loss on a batch of unlabeled data has the following form:

Original: [Equation]
Revised: [Equation]

Original: where B denotes the batch-size of labeled examples, µ a scaling hyperparameter for the unlabeled batch-size, τ ∈ [0 , 1] is a threshold parameter often set close to 1 and ρ t is either a sharpening operation on the pseudo-labels, i.e. [ ρ t ( q )]
Revised: where B denotes the batch-size of labeled examples, µ a scaling hyperparameter for the unlabeled batch-size, τ ∈ [0 , 1] is a threshold parameter often set close to 1 and ρ t is either a sharpening operation on the pseudo-labels, i.e. [ ρ t ( q )]

Original: k :=
Revised: k :=

Original: [ q ] 1 /tk / (cid:80) Kc =1 [ q ] 1 /tc or an argmax operation (i.e. t → 0 ).
Revised: [ q ] 1 /t k / (cid:80) K c =1 [ q ] 1 /tc or an argmax operation (i.e. t → 0 ).

Original: ρ t also implicitly includes a “stop-gradient” operation, i.e. gradients are not back-propagated from predictions of pseudo-labels.
Revised: ρ t also implicitly includes a “stop-gradient” operation, i.e. gradients are not back-propagated from predictions of pseudo-labels.

Original: L u is combined with the expected cross-entropy loss on labeled examples, L l = 1 B (cid:80)
Revised: L u is combined with the expected cross-entropy loss on labeled examples, L l = 1 B (cid:80)

Original: Bl =1 H ( y l , q 1 ,l ) to form the combined loss L l + λL u , with hyperparameter λ .
Revised: Bl =1 H ( y l , q 1 ,l ) to form the combined loss L l + λL u , with hyperparameter λ .

Original: Differences between Pseudo-Labels, UDA and FixMatch are detailed in Appendix C.1.
Revised: Differences between Pseudo-Labels, UDA and FixMatch are detailed in Appendix D.1.

Original: Representation learning based methods.
Revised: Representation learning based methods.

Original: We use PAWS (Assran et al., 2021) as a canonical example for this family.
Revised: We use PAWS (Assran et al., 2021) as a canonical example for this family.

Original: A key difference from threshold-mediated methods is the lack of the parametric classifier h , which is replaced by a non-parametric soft-nearest neighbour classifier ( π d ) based on a labeled support set { z s } Bs =1 .
Revised: A key difference from threshold-mediated methods is the lack of the parametric classifier h , which is replaced by a non-parametric soft-nearest neighbour classifier ( π d ) based on a labeled support set { z s } Bs =1 .

Original: Let z 1 = f
Revised: Let z 1 = f

Original: ( α 1 ( x )) and z 2 = f
Revised: ( α 1 ( x )) and z 2 = f

Original: ( α 2 ( x )) be the representations for the two views from the backbone encoder, their pseudo-labels ( q 1 , q 2 ) and the unlabeled loss are given by:[Equation] , { z s } )
Revised: ( α 2 ( x )) be the representations for the two views from the backbone encoder, their pseudo-labels ( q 1 , q 2 ) and the unlabeled loss are given by:

Original: [Equation]
Revised: [Equation]

Original: where d ( a, b ) =
Revised: where d ( a, b ) =

Original: exp( a · b/ ( ∥ a ∥∥ b ∥ τ p )) is a similarity metric with temperature hyperparameter τ p and all other symbols have the same meanings defined before.
Revised: exp( a · b/ ( ∥ a ∥∥ b ∥ τ p )) is a similarity metric with temperature hyperparameter τ p and all other symbols have the same meanings defined before.

Revised: The combined loss is L u + L me-max

Revised: [Table]

Original: The combined loss is L u + L me-max where the latter is a regularization term L me-max = H (¯ q ) that seeks to maximize the entropy of the average of predictions ¯ q :=
Revised:  [Figure] where the latter is a regularization term L me-max = H (¯ q ) that seeks to maximize the entropy of the average of predictions ¯ q :=

Original: (1 / (2 µB )) (cid:80) µBu =1 ( ρ t ( q 1 ,u ) + ρ t ( q 2 ,u )) .
Revised: (1 / (2 µB )) (cid:80) µBu =1 ( ρ t ( q 1 ,u ) + ρ t ( q 2 ,u )) .

Original: Calibration metrics.
Revised: Calibration metrics.

Original: A popular empirical metric to measure a model’s calibration is via the
Revised: A popular empirical metric to measure a model’s calibration is via the

Original: Expected Calibration Error ( ECE ) .
Revised: Expected Calibration Error ( ECE ) .

Original: Following (Guo et al., 2017; Minderer et al., 2021), we focus on a slightly weaker condition and consider only the model’s most likely class-prediction, which can be computed as follows.
Revised: Following (Guo et al., 2017; Minderer et al., 2021), we focus on a slightly weaker condition and consider only the model’s most likely class-prediction, which can be computed as follows.

Original: Let ρ 0 ( q ) denote the model’s prediction (where ρ 0 is the argmax operation) as defined before, the model predictions on a batch of N samples are grouped into M equal-interval bins, i.e. B m contains the set of samples with ρ 0 ( q ) ∈ ( m − 1 M , mM ] .
Revised: Let ρ 0 ( q ) denote the model’s prediction (where ρ 0 is the argmax operation) as defined before, the model predictions on a batch of N samples are grouped into M equal-interval bins, i.e. B m contains the set of samples with ρ 0 ( q ) ∈ ( m − 1 M , mM ] .

Original: ECE is then computed as the expected difference between the accuracy and confidence of each bin over all N samples:
Revised: ECE is then computed as the expected difference between the accuracy and confidence of each bin over all N samples:

Original: [Equation]
Revised: [Equation]

Original: [Table]  [Figure] where acc( B m ) = (1 / |B m | ) (cid:80) i ∈B m 1 ( ρ 0 ( q i )
Revised: where acc( B m ) = (1 / |B m | ) (cid:80) i ∈B m 1 ( ρ 0 ( q i )

Original: =
Revised: =

Original: y i ) and conf( B m ) =
Revised: y i ) and conf( B m ) =

Original: (1 / |B m | ) (cid:80) i ∈B m max q i with y i the true label of sample i .
Revised: (1 / |B m | ) (cid:80) i ∈B m max q i with y i the true label of sample i .

Original: In this work, we estimate ECE using M = 10 bins.
Revised: In this work, we estimate ECE using M = 10 bins.

Original: We also caveat here that while ECE is not free from biases (Minderer et al., 2021), we chose ECE over alternatives (Brier, 1950; DeGroot & Fienberg, 1983) due to its simplicity and widespread adoption.
Revised: We also caveat here that while ECE is not free from biases (Minderer et al., 2021), we chose ECE over alternatives (Brier, 1950; DeGroot & Fienberg, 1983) due to its simplicity and widespread adoption.

Original:  C ALIBRATION IN S EMI - SUPERVISED LEARNING
Revised:  C ALIBRATION IN S EMI - SUPERVISED LEARNING

Original: Better calibration is correlated to better performances.
Revised: Better calibration is correlated to better performances.

Original: To motivate our work, we first perform detailed ablation studies on SOTA SSL methods.
Revised: To motivate our work, we first perform detailed ablation studies on SOTA SSL methods.

Original: In particular, we used FixMatch (Sohn et al., 2020) and PAWS (Assran et al., 2021), each from the two families, due to their strong performance.
Revised: In particular, we used FixMatch (Sohn et al., 2020) and PAWS (Assran et al., 2021), each from the two families, due to their strong performance.

Original: FixMatch relies on using a high value of the selection threshold τ to mitigate confirmation bias by accepting only the most credible pseudo-labels; on the other hand, PAWS uses multiple techniques that seem to have an effect on shaping the output prediction distribution and implicitly controlling calibration — these include label smoothing (Müller et al., 2019), mean-entropy maximization (Joulin & Bach, 2012), sharpening (Berthelot et al., 2019b;a; Xie et al., 2019a) and temperature scaling (Guo et al., 2017).
Revised: FixMatch relies on using a high value of the selection threshold τ to mitigate confirmation bias by accepting only the most credible pseudo-labels; on the other hand, PAWS uses multiple techniques that seem to have an effect on shaping the output prediction distribution and implicitly controlling calibration — these include label smoothing (Müller et al., 2019), mean-entropy maximization (Joulin & Bach, 2012), sharpening (Berthelot et al., 2019b;a; Xie et al., 2019a) and temperature scaling (Guo et al., 2017).

Original: In Fig.
Revised: In Fig.

Original: 1, we ablated on all of these parameters and observe an overall trend, for both methods, that model performance is strongly correlated to better calibration (i.e. lower ECE).
Revised: 2, we ablated on all of these parameters and observe an overall trend, for both methods, that model performance is strongly correlated to better calibration (i.e. lower ECE).

Original: However, these trends are inevitably noisy since none of these parameters tunes for calibration in isolation but are instead optimized towards performance.
Revised: However, these trends are inevitably noisy since none of these parameters tunes for calibration in isolation but are instead optimized towards performance.

Original: Therefore, in this work, we aim to explore techniques that predominantly adjusts for model calibration in these methods to clearly demonstrate the direct effect of improving calibration towards model accuracy.
Revised: Therefore, in this work, we aim to explore techniques that predominantly adjusts for model calibration in these methods to clearly demonstrate the direct effect of improving calibration towards model accuracy.

Original: 4. I MPROVING CALIBRATION WITH B AYESIAN M ODEL A VERAGING
Revised: 4. I MPROVING CALIBRATION WITH B AYESIAN M ODEL A VERAGING

Original: Bayesian techniques have been widely known to produce well-calibrated uncertainty estimates (Wilson & Izmailov, 2020), thus in our work, we explored the use of approximate Bayesian Neural
Revised: Bayesian techniques have been widely known to produce well-calibrated uncertainty estimates (Wilson & Izmailov, 2020), thus in our work, we explored the use of approximate Bayesian Neural

Original: Networks (BNN).
Revised: Networks (BNN).

Original: To minimize the additional computational overhead, we propose to only replace the final layer of the network with a BNN layer.
Revised: To minimize the additional computational overhead, we propose to only replace the final layer of the network with a BNN layer.

Revised: We propose a novel family of SSL methods termed

Original:  For threshold-mediated methods this is the linear classification head h and for representation learning methods this is the finallinear layer of the projection network (i.e. immediately before the representations).
Revised: BAM- (for BAyesian Model averaging) and provide an illustration of this approach in Fig. For threshold-mediated methods the BNN layer is the linear classification head h and for representation learning methods this is the final layer of the projection head.

Original: For brevity we will simply denote this BNN layer to be h and an input embedding to this layer to be v in this section.
Revised: For brevity we will simply denote this BNN layer to be h and an input embedding to this layer to be v in this section.

Original: Following a
Revised: Following a

Original: Bayesian approach, we first assume a prior distribution on weights p ( θ h ) .
Revised: Bayesian approach, we first assume a prior distribution on weights p ( θ h ) .

Original: Given some training data
Revised: Given some training data

Original: D X :=
Revised: D X :=

Original: ( X, Y ) , we seek to calculate the posterior distribution of weights, p ( θ h |D X ) , which can then be used to derive the posterior predictive p ( y | v, D X )
Revised: ( X, Y ) , we seek to calculate the posterior distribution of weights, p ( θ h |D X ) , which can then be used to derive the posterior predictive p ( y | v, D X )

Original: = (cid:82) p ( y | v, θ h ) p ( θ h |D X ) dθ h .
Revised: = (cid:82) p ( y | v, θ h ) p ( θ h |D X ) dθ h .

Original: This process is also known as “Bayesian model averaging” or “Bayesian marginalization” (Wilson & Izmailov, 2020).
Revised: This process is also known as “Bayesian model averaging” or “Bayesian marginalization” (Wilson & Izmailov, 2020).

Original: Since exact Bayesian inference is computationally intractable for neural networks, we adopt a variational approach following (Blundell et al., 2015), where we learn a Gaussian variational approximation to the posterior q ϕ
Revised: Since exact Bayesian inference is computationally intractable for neural networks, we adopt a variational approach following (Blundell et al., 2015), where we learn a Gaussian variational approximation to the posterior q ϕ

Original: ( θ h | ϕ ) , parameterized by ϕ , by maximizing the evidence lowerbound (ELBO) (see Appendix B.1 for details).
Revised: ( θ h | ϕ ) , parameterized by ϕ , by maximizing the evidence lowerbound (ELBO) (see Appendix C.1 for details).

Original: The ELBO =
Revised: The ELBO =

Original: E q log p
Revised: E q log p

Original: ( Y | X ; θ ) − KL ( q ( θ | ϕ )
Revised: ( Y | X ; θ ) − KL ( q ( θ | ϕ )

Original: ∥ p ( θ )) consists of a log-likelihood (data-dependent) term and a KL (prior-dependent) term.
Revised: ∥ p ( θ )) consists of a log-likelihood (data-dependent) term and a KL (prior-dependent) term.

Original: Theoretical results.
Revised: Theoretical results.

Original: We motivate our approach by using Corollary 1 below, derived from the
Revised: We motivate our approach by using Corollary 1 below, derived from the

Original: PAC-Bayes framework (see Appendix A for the proof).
Revised: PAC-Bayes framework (see Appendix B for the proof).

Original: The statement shows the generalization error bounds on the variational posterior in the SSL setting and suggests that this generalization error is upper bounded by the negative ELBO .
Revised: The statement shows the generalization error bounds on the variational posterior in the SSL setting and suggests that this generalization error is upper bounded by the negative ELBO .

Original: This motivates our approach, i.e. by maximizing the ELBO we improve generalization by minimizing the upper bound to the generalization error.
Revised: This motivates our approach, i.e. by maximizing the ELBO we improve generalization by minimizing the upper bound to the generalization error.

Original: The second term on the right side of the inequality characterises thegeneralization bound for the SSL setting, which vanishes in the supervised setting.
Revised: The second term on the right side of the inequality characterises the SSL setting, and vanishes in the supervised setting.

Original: Corollary 1 Let D be a data distribution
Revised: Corollary 1 Let D be a data distribution

Original: where i.i.d.
Revised: where i.i.d.

Original: training samples are sampled, of which N l are labeled, ( x, y ) ∼ D N l and N u are unlabeled, ( x, ˆ y ) ∼ D N u where ˆ y i
Revised: training samples are sampled, of which N l are labeled, ( x, y ) ∼ D N l and N u are unlabeled, ( x, ˆ y ) ∼ D N u where ˆ y i

Original: ( y i ) denotes the modelassigned pseudo-labels (true labels) for input x i .
Revised: ( y i ) denotes the modelassigned pseudo-labels (true labels) for input x i .

Original: For the negative log likelihood loss function ℓ , assuming that ℓ is sub-Gaussian (see Germain et al.
Revised: For the negative log likelihood loss function ℓ , assuming that ℓ is sub-Gaussian (see Germain et al.

Original: with variance factor s 2 , then with probability at least 1 − δ , the generalization error  of the variational posterior q ( θ | ϕ ) is given by,
Revised: with variance factor s 2 , then with probability at least 1 − δ , the generalization error (denoted as L ℓ D ) of the variational posterior q ( θ | ϕ ) is given by,

Original: [Equation]
Revised: [Equation]

Original: Incorporating Bayesian model averaging techniques involves two main modifications during pseudolabeling: 1) M weights from the BNN layer are sampled and predictions are derived from the Monte 
Revised: As depicted in Fig. 1, pseudo-labeling proceeds in two-stages: 1) M weights from the BNN layer are sampled and predictions are derived from the Monte Carlo estimated posterior predictive,

Original: Carlo estimated posterior predictive, i.e. ˆ q = (1 /M )
Revised: i.e. ˆ q = (1 /M )

Original: (cid:80)
Revised: (cid:80)

Original: Mm h ( v, θ ( m ) h ) , and 2) the selection criteria, if present, is based upon their variance, σ 2 c = (1 /M )
Revised: Mm h ( v, θ ( m ) h ) , and 2) the selection criteria, if present, is based upon their variance, σ 2 c = (1 /M )

Original: (cid:80)
Revised: (cid:80)

Original: Mm ( h ( v, θ ( m ) h ) − ˆ q ) 2 , at the predicted class c = argmax c ′ [ˆ q ] c ′ .
Revised: Mm ( h ( v, θ ( m ) h ) − ˆ q ) 2 , at the predicted class c = argmax c ′ [ˆ q ] c ′ .

Original: This constitutes a better uncertainty measure as compared to the maximum logit value used in threshold-mediated methods and is highly intuitive — if the model’s prediction has a large variance, it is highly uncertain and the pseudo-label should not be accepted.
Revised: This constitutes a better uncertainty measure as compared to the maximum logit value used in threshold-mediated methods and is highly intuitive — if the model’s prediction has a large variance, it is highly uncertain and the pseudo-label should not be accepted.

Original: In practice, as σ 2 c decreases across training, we use a simple quantile Q over the batch to define the threshold where pseudo-labels of samples with σ 2 c < Q are accepted, with Q as a hyperparameter (see Appendix B. for pseudocode).
Revised: In practice, as σ 2 c decreases across training, we use a simple quantile Q over the batch to define the threshold where pseudo-labels of samples with σ 2 c < Q are accepted, with Q as a hyperparameter (see Appendix C.1 for pseudocode).

Original: In representation learning methods where predictions are computed using a non-parametric nearest neighbour classifier, we use the “Bayesian marginalized” versions of the representations, i.e. q i = π d
Revised: In representation learning methods where predictions are computed using a non-parametric nearest neighbour classifier, we use the “Bayesian marginalized” versions of the representations, i.e. q i = π d

Original: (ˆ z i , { ˆ z s } ) in Eq.
Revised: (ˆ z i , { ˆ z s } ) in Eq.

Original: (2) where ˆ z
Revised: (2) where ˆ z

Original: =
Revised: =

Original: (1 /M )
Revised: (1 /M )

Original: (cid:80)
Revised: (cid:80)

Original: Mm h ( v, θ ( m ) h ) .
Revised: Mm h ( v, θ ( m ) h ) .

Original: Rested on the two modifications detailed above, in this work we introduce a new family of SSL methods that builds upon SOTA SSL methods and name them “BAM-X”, which incorporates approximate BAyesian Model averaging (BAM) during pseudo-labeling for SSL method X.
Revised: We explore the effectiveness of BAM by modifying upon SOTA SSL methods and name them “BAM-X”, which incorporates approximate BAyesian Model averaging (BAM) during pseudo-labeling for SSL method X.

Original: 4. I MPROVING CALIBRATION WITH WEIGHT AVERAGING TECHNIQUES
Revised: 4. I MPROVING CALIBRATION WITH WEIGHT AVERAGING TECHNIQUES

Original: A BNN classifier has two desirable characteristics: 1)rather than a single prediction, multiple predictions areaveraged or “Bayesian marginalized” and 2)we obtain a better measure of uncertainty (i.e. confidence estimates) from the variance across predictions which serves as a better selection metric.
Revised: A BNN classifier has two desirable characteristics: 1) multiple predictions are “Bayesian marginalized” and 2) better measure of uncertainty (i.e. confidence estimates) using the variance across predictions.

Original:  SSL methods that do not use a selection metric, e.g. representation learning methods like PAWS, cannot explicitly benefit from (2) and in these cases, instead of aggregating over just one layer, one could seek to aggregate over the entire network.
Revised: Representation learning SSL methods that do not use a selection metric cannot explicitly benefit from (2) and in these cases, instead of aggregating over just one layer, one could seek to aggregate over the entire network.

Original: This can be achieved by ensembling approaches (Lakshminarayanan et al., 2016; Wen et al., 2020), which have been highly successful at improving uncertainty estimation; however, training multiple networks would add immense computational overhead.
Revised: To explore the generality of the importance of calibration in SSL beyond a

Original: In this work, we propose and explore weight averaging approaches such as Stochastic Weight Averaging (Izmailov et al., 2018)
Revised: Bayesian layer approach, we further explored well-established weight averaging approaches such as Stochastic Weight Averaging (Izmailov et al., 2018)

Original: (SWA) and Exponential Moving Averaging (EMA) (Tarvainen & Valpola, 2017;
Revised: (SWA) and Exponential Moving Averaging (EMA) (Tarvainen & Valpola, 2017;

Original: He et al., 2020; Grill et al., 2020a) to improve calibration during pseudo-labeling.
Revised: He et al., 2020; Grill et al., 2020a) and studied their role in calibration during pseudo-labeling.

Original: Weight averaging differs from ensembling in that model weights are averaged instead of predictions; however, the approximation of SWA to Fast Geometric Ensembles (Garipov et al., 2018) has been justified by previous work (Izmailov et al., 2018).

Original: To the best of our knowledge, the connection of EMA to ensembling has not been formally shown.

Original: Incorporating weight averaging to pseudo-labeling requires one key modification – we maintain a separate set of non-trainable weights θ g containing the aggregated weight average which are used throughout training to produce better calibrated pseudo-labels.
Revised: To do so, we maintain a separate set of non-trainable weights θ g containing the aggregated weight average which are used throughout training to produce better calibrated pseudo-labels.

Original: In SWA, these weights are updated via θ g ← ( n a θ g + θ f ) / ( n a + 1) at every iteration, where n a represents the total number of models in the aggregate and θ f are the trainable parameters of our backbone encoder as before.
Revised: In SWA, these weights are updated via θ g ← ( n a θ g + θ f ) / ( n a + 1) at every iteration, where n a represents the total number of models in the aggregate and θ f are the trainable parameters of our backbone encoder as before.

Original: The update of EMA only differs slightly: θ g ← γθ g +(1 − γ ) θ f , where γ is a momentum hyperparameter controlling how much memory θ g should retain at each iteration (see Appendix B.2 for pseudocode).
Revised: The update of EMA only differs slightly: θ g ← γθ g + (1 − γ ) θ f , where γ is a momentum hyperparameter controlling how much memory θ g should retain at each iteration (see Appendix C.2 for pseudocode).

Original: Interestingly, most SSL methods that rely on pseudo-labeling possesses a “stop-gradient” operation on the branch generating pseudo-labels (see the description of ρ t in Section 3). This makes them easily amendable to weight averaging; for e.g. the “stop-gradient” operation canalready be interpreted as an EMA with a trivial decay parameter of γ = 0 or an SWA with trivial n a = 0 .
Revised: While EMA has been previously explored in the context of SSL (Tarvainen & Valpola, 2017), to the best of our knowledge SWA has not been explored in SSL and more importantly, their link to pseudo-label calibration has not been explicitly shown. In this work, we demonstrate that the effectiveness of such weight averaging approaches can be understood from an improved calibration of pseudo-labels.

Original:  E XPERIMENTAL S ETUP
Revised:  E XPERIMENTAL S ETUP

Original: In all our experiments, we begin with and modify upon the original implementations of the baseline
Revised: In all our experiments, we begin with and modify upon the original implementations of the baseline

Original: SSL methods.
Revised: SSL methods.

Original: The backbone encoder f is a Wide ResNet-28-2, Wide ResNet-28-8 and ResNetfor the CIFAR-10, CIFAR-100 and ImageNet benchmarks respectively.
Revised: The backbone encoder f is a Wide ResNet-28-2, Wide ResNet-28-8 and ResNetfor the CIFAR-10, CIFAR-100 and ImageNet benchmarks respectively.

Original: Where possible, the default hyperparameters and dataset-specific settings (learning rates, batch size, optimizers and schedulers) recommended by the original authors were used and where unavailable (for e.g. CIFAR-100 on
Revised: Where possible, the default hyperparameters and dataset-specific settings (learning rates, batch size, optimizers and schedulers) recommended by the original authors were used and where unavailable (for e.g. CIFAR-100 on

Original: PAWS), the baseline was first optimized to the best we could.
Revised: PAWS), the baseline was first optimized to the best we could.

Original: To ensure fairness, we use the same configuration and same set of hyperparameters when comparing our methods against the baselines.
Revised: To ensure fairness, we use the same configuration and same set of hyperparameters when comparing our methods against the baselines.

Original: Bayesian model averaging with a BNN final layer.
Revised: Bayesian model averaging with a BNN final layer.

Original: We set the weight priors as unit Gaussians and use a separate Adam optimizer for the BNN layer with learning rate 0.01, no weight decay and impose the same cosine learning rate scheduler as the backbone.
Revised: We set the weight priors as unit Gaussians and use a separate Adam optimizer for the BNN layer with learning rate 0.01, no weight decay and impose the same cosine learning rate scheduler as the backbone.

Original: We set Q = 0 .
Revised: We set Q = 0 .

Original: 75 for the CIFARbenchmark and Q = 0 .
Revised: 75 for the CIFARbenchmark and Q = 0 .

Original: 95 for the CIFAR-10 benchmark; which are both linearly warmed-up from 0 .
Revised: 95 for the CIFAR-10 benchmark; which are both linearly warmed-up from 0 .

Original: 1 in the first 10 epochs.
Revised: 1 in the first 10 epochs.

Original: As Q is computed across batches, we improve stability by using a moving average of the last 50 thresholds.
Revised: As Q is computed across batches, we improve stability by using a moving average of the last 50 thresholds.

Original: For representation learning methods without a parametric classifier, we do not use a separate optimizer and simply impose a one-minus-cosine scheduler (i.e. scheduler (2) from the next paragraph) for the coefficient to the KL-divergence loss that goes from 0 to 1 in
Revised: For representation learning methods without a parametric classifier, we do not use a separate optimizer and simply impose a one-minus-cosine scheduler (i.e. scheduler (2) from the next paragraph) for the coefficient to the KL-divergence loss that goes from 0 to 1 in

Original: T epochs where we simply picked the best from T ∈ { 50 , 100 } .
Revised: T epochs where we simply picked the best from T ∈ { 50 , 100 } .

Original: Weight averaging.
Revised: Weight averaging.

Original: We delay the onset of SWA to after some amount of training has elapsed (i.e.
Revised: We delay the onset of SWA to after some amount of training has elapsed (i.e.

Original: T swa epochs) since it is undesirable to include the randomly initialized weights to the aggregate.
Revised: T swa epochs) since it is undesirable to include the randomly initialized weights to the aggregate.

Original: We set T swa = 200 and T swa = 100 for CIFAR-10 and CIFAR-100 respectively.
Revised: We set T swa = 200 and T swa = 100 for CIFAR-10 and CIFAR-100 respectively.

Original: In contrast, EMA has a natural curriculum to “forget” older model parameters since more recent parameters are given more weight in the aggregate.
Revised: In contrast, EMA has a natural curriculum to “forget” older model parameters since more recent parameters are given more weight in the aggregate.

Original: We experimented with two schedules for γ : 1) using a linear warm-up from 0 to 0.996 in 50 epochs and then maintaining γ at 0.996 for the rest of training and 2) using a one-minus-cosine scheduler starting from 0.05 and decreasing to 0 resulting in a 0.95 to 1 range for γ .
Revised: We experimented with two schedules for γ : 1) using a linear warm-up from 0 to 0.996 in 50 epochs and then maintaining γ at 0.996 for the rest of training and 2) using a one-minus-cosine scheduler starting from 0.05 and decreasing to 0 resulting in a 0.95 to 1 range for γ .

Original: We visualize these schedulers and include ablation studies on them and on T swa in Appendix F.3.
Revised: We visualize these schedulers and include ablation studies on them and on T swa in Appendix G.3.

Original: ECE and test accuracy evaluation.
Revised: ECE and test accuracy evaluation.

Original: In our experiments, we found that the test accuracy exhibits a considerable amount of noise across training, especially in label-scarce settings.
Revised: In our experiments, we found that the test accuracy exhibits a considerable amount of noise across training, especially in label-scarce settings.

Original: Sohn et al.
Revised: Sohn et al.

Original: (2020) proposes to take the median accuracy of the last 20 checkpoints, while Zhang et al.
Revised: (2020) proposes to take the median accuracy of the last 20 checkpoints, while Zhang et al.

Original: (2021) argues that this fixed training budget approach is not suitable when convergence speeds of the algorithms are different (as the faster converging algorithm would over-fit more severely at the end) and thus report also the overall best accuracy.
Revised: (2021) argues that this fixed training budget approach is not suitable when convergence speeds of the algorithms are different (as the faster converging algorithm would over-fit more severely at the end) and thus report also the overall best accuracy.

Original: In our experiments, we adopt a balance between the two aforementioned approaches: we consider the median of 20 checkpoints around the best accuracy checkpoint as the convergence criteria , and report this value as the test accuracy.
Revised: In our experiments, we adopt a balance between the two aforementioned approaches: we consider the median of 20 checkpoints around the best accuracy checkpoint as the convergence criteria , and report this value as the test accuracy.

Original: The ECE is reported when the model reaches this convergence criteria (One could also also aggregate the ECEs up till convergence — we found this gave similar trends and thus report the simpler metric.)
Revised: The ECE is reported when the model reaches this convergence criteria (One could also also aggregate the ECEs up till convergence — we found this gave similar trends and thus report the simpler metric.)

Original: For PAWS, we follow Assran et al.
Revised: For PAWS, we follow Assran et al.

Original: and report the test accuracy and ECE from the soft nearest neighbour classifier on the CIFAR benchmarks and after fine-tuning a linear head for the ImageNet benchmark.
Revised: and report the test accuracy and ECE from the soft nearest neighbour classifier on the CIFAR benchmarks and after fine-tuning a linear head for the ImageNet benchmark.

Original:  R ESULTS
Revised:  R ESULTS

Original: Bayesian model averaging.
Revised: Bayesian model averaging.

Original: Our main results on CIFAR-10 and CIFAR-100 in Table 1 show the use of a BNN layer to improve calibration for all methods in our study.
Revised: Our main results on CIFAR-10 and CIFAR-100 in Table 1 show the use of a BNN layer to improve calibration for all methods in our study.

Original: Table 1 demonstrates that incorporating a BNN final layer successfully reduces the ECE over the baselines across all the benchmarks and as a result, we also attained significant improvements in test accuracies, notably up to 15.9% (for UDA on CIFAR-100-400 labels).
Revised: Table 1 demonstrates that incorporating a BNN final layer successfully reduces the ECE over the baselines across all the [Table]

Original: Interestingly, while the baseline FixMatch outperforms UDA across all the benchmarks, improving calibration in UDA (i.e. BAM-UDA) allows it to outperfom FixMatch and even the calibrated version of FixMatch (i.e. BAM-FM). A key difference between FixMatch and UDA is the use of hard pseudo-labels in FixMatch (i.e. t → 0 in ρ t
Revised: benchmarks and as a result

Original: [Table]
Revised: [Table]

Original:  defined in Section 3) versus soft pseudo-labels in UDA (with t = 0 . 4 ); this suggests that a Bayesian classifier is more effective in conjunction with soft pseudo-labels.
Revised: we also attained significant improvements in test accuracies, notably up to 16.2% (for UDA on CIFAR-100-400 labels). Interestingly, while the baseline FixMatch outperforms UDA across all the benchmarks, improving calibration in UDA (i.e. BAM-UDA) allows it to outperfom FixMatch and even the calibrated version of FixMatch (i.e. BAM-FM). A key difference between FixMatch and UDA is the use of hard pseudo-labels in FixMatch (i.e. t → 0 in ρ t defined in Section 3) versus soft pseudo-labels in UDA (with t = 0 . 4 ); this suggests that a Bayesian classifier is more effective in conjunction with soft pseudo-labels.

Original: Intuitively, this makes sense, since better calibration could potentially allow the model to leverage on information about its prediction on classes apart from the one it’s most confident of.
Revised: Intuitively, this makes sense, since better calibration could potentially allow the model to leverage on information about its prediction on classes apart from the one it’s most confident of.

Original: This further underscores the importance of calibration — while information about the model’s prediction on other classes is useful, the lack of proper calibration prohibits the model from using this information, resulting in stronger confirmation bias compared to the case where this information is not used (i [Table] e. UDA vs FixMatch) [Table]
Revised: This further underscores the importance of calibration — while information about the model’s prediction on other classes is useful, the lack of proper calibration prohibits the model from using this information, resulting in stronger confirmation bias compared to the case where this information is not used (i.e. UDA vs FixMatch).

Original: Leveraging on this insight [Table] we significantly reduce the sharpening of the pseudo-labels in BAM-UDA by setting t = 0 .
Revised: Leveraging on this insight, we significantly reduce the sharpening of the pseudo-labels in BAM-UDA by setting t = 0 .

Original: 9 (see Appendix F.2 for ablations on t ).
Revised: 9 (see Appendix G.2 for ablations on t ).

Original: BAM-UDA consistently outperforms all threshold-mediated baselines across all CIFAR benchmarks .
Revised: BAM-UDA consistently outperforms all threshold-mediated baselines across all CIFAR benchmarks .

Original: Overall, we found that the improvements in both calibration and test accuracy are more significant for label-scarce settings, as expected, since the problem of confirmation bias is more acute there and thus calibration can provide greater benefits by mitigating this.
Revised: Overall, we found that the improvements in both calibration and test accuracy are more significant for label-scarce settings, as expected, since the problem of confirmation bias is more acute there and thus calibration can provide greater benefits by mitigating this.

Original: The additional computation overhead incurred from our methods is minimal, adding approximately only 2-5% in wall-clock time (see Appendix G).
Revised: The additional computation overhead incurred from our methods is minimal, adding approximately only 2-5% in wall-clock time (see Appendix H).

Original: While previous works (Sohn et al., 2020) also include extreme low label settings such as CIFAR-10-40 labels, we found this benchmark to be highly sensitive to random initialization and thus exclude them in our study.
Revised: While previous works (Sohn et al., 2020) also include extreme low label settings such as CIFAR-10-40 labels, we found this benchmark to be highly sensitive to random initialization and thus exclude them in our study.

Original: Why does improving calibration improve performance?
Revised: Why does improving calibration improve performance?

Original: To further understand the dynamics between calibration and test accuracies, we track the test accuracy and ECE over the course of training for UDA, FixMatch and BAM-UDA and plot them in Fig.
Revised: To further understand the dynamics between calibration and test accuracies, we track the test accuracy and ECE over the course of training for UDA, FixMatch and BAM-UDA and plot them in Fig.

Original: While the baselines learn effectively for the initial stages of training, learning is eventually hindered — as the model is encouraged to output increasingly confident predictions due to the entropy minimization objective, in the absence of explicit calibration, the baselines inevitably become over-confident, resulting in a drop in test accuracy.
Revised: While the baselines learn effectively for the initial stages of training, learning is eventually hindered — as the model is encouraged to output increasingly confident predictions due to the entropy minimization objective, in the absence of explicit calibration, the baselines inevitably become over-confident, resulting in a drop in test accuracy.

Original: This phenomenon of confirmation bias is evident from that 1) the baselines’ output predictions become increasingly over-confident (Fig. 2d) and 2) they make increasingly more mistakes on the accepted pseudo-labels (Fig. 2c).
Revised: This phenomenon of confirmation bias is evident from that 1) the baselines’ output predictions become increasingly over-confident (Fig. 3d) and 2) they make increasingly more mistakes on the accepted pseudo-labels (Fig. 3c).

Original: In contrast, explicit calibration via Bayesian model averaging allows BAM-UDA to improve its calibration throughout training, resulting in a constantly improving purity rate and effectively mitigating confirmation bias, thus promoting learning for longer periods to result in better final performances.
Revised: In contrast, explicit calibration via Bayesian model averaging allows BAM-UDA to improve its calibration throughout training, resulting in a constantly improving purity rate and effectively mitigating confirmation bias, thus promoting learning for longer periods to result in better final performances.

Original: Further ablation studies are discussed in Appendix F.2.
Revised: Further ablation studies are discussed in Appendix G.2.

Original: Weight averaging techniques.
Revised: Weight averaging techniques.

Original: The effect of better calibration is also clearly demonstrated in the weight averaging approaches (Fig. 3).
Revised: The effect of better calibration is also clearly demonstrated in the weight averaging approaches (Fig. 4).

Original: This is particularly evident through PAWS-SWA—when
Revised: This is particularly evident through PAWS-SWA—when

Original: SWA was switched on (at T swa = 200 epochs), the ECE quickly dips and the test accuracy also correspondingly spikes.
Revised: SWA was switched on (at T swa = 200 epochs), the ECE quickly dips and the test accuracy also correspondingly spikes.

Original: Furthermore, we also observe a significant improvement in convergence rates resulting from the weight averaging techniques, i.e. better test accuracies can be attained in just a third of the number of training iterations 
Revised: Furthermore, we also observe a significant improvement in convergence rates resulting from the weight averaging techniques, i.e. better test accuracies can be attained in just a third of the number of training iterations of the baseline. Ablations on T swa and momentum

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: schedules are presented in Appendix F.3.
Revised: schedules are presented in Appendix G.3.

Original: Our motivation for exploring weight averaging techniques in PAWS was attributed to its lack of a selection metric; however, given that almost all SSL methods have a “stop-gradient” operation on pseudo-labels, one could potentially also explore the use of weight averaging techniques in threshold-mediated methods.
Revised: Our motivation for exploring weight averaging techniques in PAWS was attributed to its lack of a selection metric; however, given that almost all SSL methods have a “stop-gradient” operation on pseudo-labels, one could potentially also explore the use of weight averaging techniques in threshold-mediated methods.

Original: Results are shown in Appendix F.1, where we studied the use of EMA on the best performing threshold-mediated baseline,
Revised: Results are shown in Appendix G.1, where we studied the use of EMA on the best performing threshold-mediated baseline,

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: e. FixMatch.
Revised: e. FixMatch.

Original: Notably, we found that EMA consistently under-performs Bayesian model averaging, highlighting the superiority of a BNN classifier in threshold-mediated methods.
Revised: Notably, we found that EMA consistently under-performs Bayesian model averaging, highlighting the superiority of a BNN classifier in threshold-mediated methods.

Original: This could be attributed to the fact that, in addition to averaging predictions, a BNN classifier provides better uncertainty estimates to improve the selection metric in these family of methods (see Appendix F.1 for further discussion) [Figure]
Revised: This could be attributed to the fact that, in addition to averaging predictions, a BNN classifier provides better uncertainty estimates to improve the selection metric in these family of methods (see Appendix G.1 for further discussion).

Original: 6. L ARGE - SCALE & REAL - WORLD DATASETS
Revised: 6. L ARGE - SCALE & REAL - WORLD DATASETS

Original: [Table]
Revised: [Table]

Original: ImageNet.
Revised: ImageNet.

Original: We further verified the effectiveness of calibration on the large-scale dataset ImageNet.
Revised: We further verified the effectiveness of calibration on the large-scale dataset ImageNet.

Original: Out of all the baseline methods in our study, PAWS reports the best performance on the ImageNet-10% benchmark (i.e. 73.9% for 100 epochs vs 71.5% on the best performing threshold-mediated method, FixMatch).
Revised: Out of all the baseline methods in our study, PAWS reports the best performance on the ImageNet-10% benchmark (i.e. 73.9% for 100 epochs vs 71.5% on the best performing threshold-mediated method, FixMatch).

Original: Due to computational resource limitations, we focus our experiments only on the best baseline PAWS and explored calibration improvements using SWA and EMA.
Revised: Due to computational resource limitations, we focus our experiments only on the best baseline PAWS and explored calibration improvements using SWA and EMA.

Original: We use the 10% labels setting of Assran et al.
Revised: We use the 10% labels setting of Assran et al.

Original: (2021) with slight modifications to the default setting in order to fit on our system (see Appendix C.2 for implementation details).
Revised: (2021) with slight modifications to the default setting in order to fit on our system (see Appendix D.2 for implementation details).

Original: In particular, we maintain the exact same configuration across the baseline and our methods.
Revised: In particular, we maintain the exact same configuration across the baseline and our methods.

Original: Results are shown in Table 2, where our methods provide consistent improvements in calibration, and correspondingly, we also obtain consistent improvements on the test accuracy.[Table]
Revised: Results are shown in Table 2, where our methods provide consistent improvements in calibration, and correspondingly, we also obtain consistent improvements on the test accuracy.

Original: Long-tailed datasets.
Revised: Long-tailed datasets.

Original: Datasets in the real world are often long-tailed or class-imbalanced, where some classes are more commonly observed while others are rare.
Revised: Datasets in the real world are often long-tailed or class-imbalanced, where some classes are more commonly observed while others are rare.

Original: We curate long-tailed versions from the CIFAR datasets following Cao et al.
Revised: We curate long-tailed versions from the CIFAR datasets following Cao et al.

Original: (2019), where α indicates the imbalance ratio (i.e. ratio between the sample sizes of the most frequent and least frequent classes).
Revised: (2019), where α indicates the imbalance ratio (i.e. ratio between the sample sizes of the most frequent and least frequent classes).

Original: We randomly select 10%  of the samples in each class to form the labeled set; see further details in Appendix D.
Revised: We randomly select 10% [Table] of the samples in each class to form the labeled set; see further details in Appendix E. We use the best performing baseline method from Table 1,

Original: We use the best performing baseline method from Table 1, i.e. FixMatch (FM), as our baseline and compare against
Revised: i [Table] e. FixMatch (FM), as our baseline and compare against

Original: BAM-UDA.
Revised: BAM-UDA.

Original: Results are shown in Table 3, where BAM-UDA achieves consistent improvements over
Revised: Results are shown in Table [Table] where BAM-UDA achieves consistent improvements over

Original: FM in both calibration and accuracy across all benchmarks.
Revised: FM in both calibration and accuracy across all benchmarks.

Original: Notably, gains from improving calibration are more significant than those in the class-balanced settings (for e.g. BAM-UDA improves upon FM by a smaller margin of 1.5% on the CIFAR-100-4000 labels benchmark which is also approximately 10%), further underscoring the utility of improving calibration in more challenging SSL problems.
Revised: Notably, gains from improving calibration are more significant than those in the class-balanced settings (for e.g. BAM-UDA improves upon FM by a smaller margin of 1.5% on the CIFAR-100-4000 labels benchmark which is also approximately 10%), further underscoring the utility of improving calibration in more challenging SSL problems.

Original: Further results are in Appendix D.2, where test samples were separated into three groups depending on the number of samples per class and test accuracies are plotted for each group.
Revised: Further results are in Appendix E.2, where test samples were separated into three groups depending on the number of samples per class and test accuracies are plotted for each group

Original: [Table] [Table]
Revised: [Table] [Table]

Original:  A practical example of a real-world domain where long-tailed datasets are prevalent is that of science – samples with the desired properties are often much rarer than trivial samples.
Revised: Photonics science. A practical example of a real-world domain where long-tailed datasets are prevalent is that of science – samples with the desired properties are often much rarer than trivial samples.

Original: Further, SSL is highly important in scientific domains since labeled data is particularly scarce (owing to the high resource cost needed for data collection).
Revised: Further, SSL is highly important in scientific domains since labeled data is particularly scarce (owing to the high resource cost needed for data collection).

Original: To demonstrate the effectiveness of calibration, we adopt a problem in photonics (Loh et al., 2022), where the task is a 5-way classification of photonic crystals (PhCs) based on their band gap sizes.
Revised: To demonstrate the effectiveness of calibration, we adopt a problem in photonics (Loh et al., 2022), where the task is a 5-way classification of photonic crystals (PhCs) based on their band gap sizes.

Original: A brief summary and visualization of this dataset are detailed in Appendix E. We explored an approach similar to FixMatch for the baseline and similar to BAM-UDA for ours (with modifications needed in the augmentation strategies; see Appendix E).
Revised: A brief summary and visualization of this dataset are detailed in Appendix F. We explored an approach similar to FixMatch for the baseline and similar to BAM-UDA for ours (with modifications needed in the augmentation strategies; see Appendix F).

Original: Results are shown in Table 4, where we demonstrate the consistency of
Revised: Results are shown in Table 4, where we demonstrate the consistency of

Original: BAM-UDA’s effectiveness in improving calibration and accuracies in this real-world problem.
Revised: BAM-UDA’s effectiveness in improving calibration and accuracies in this real-world problem.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we demonstrate that calibration plays a crucial role in SSL methods – specifically, since confirmation bias is a fundamental problem in SSL, it is imperative for the model to be wellcalibrated to mitigate this problem.
Revised: In this work, we demonstrate that calibration plays a crucial role in SSL methods – specifically, since confirmation bias is a fundamental problem in SSL, it is imperative for the model to be wellcalibrated to mitigate this problem.

Original: In particular, we demonstrated the strong correspondence between calibration and model performance in SSL methods and proposed to use approximate Bayesian techniques to directly improve calibration in state-of-the-art SSL methods.
Revised: In particular, we demonstrated the strong correspondence between calibration and model performance in SSL methods and proposed to use approximate Bayesian techniques to directly improve calibration in state-of-the-art SSL methods.

Original: Notably, we demonstrate their effectiveness across a broad range of SSL methods and further underscore their importance in more challenging real-world datasets.
Revised: Notably, we demonstrate their effectiveness across a broad range of SSL methods and further underscore their importance in more challenging real-world datasets.

Original: We hope that our findings can motivate future research directions to incorporate techniques targeted for optimizing calibration during the development of new SSL methods.
Revised: We hope that our findings can motivate future research directions to incorporate techniques targeted for optimizing calibration during the development of new SSL methods.

Original: Furthermore, while the primary goal of improving calibration is to mitigate confirmation bias during pseudo-labeling, an auxiliary benefit brought about by our approach is a better calibrated network, i.e. one that can better quantify its uncertainty, which is highly important for real-world applications.
Revised: Furthermore, while the primary goal of improving calibration is to mitigate confirmation bias during pseudo-labeling, an auxiliary benefit brought about by our approach is a better calibrated network, i.e. one that can better quantify its uncertainty, which is highly important for real-world applications.

Original: A potential limitation in our work lies in the use of ECE as a metric to measure calibration which, while commonly used across literature, are not free from flaws (Nixon et al., 2019).
Revised: A potential limitation in our work lies in the use of ECE as a metric to measure calibration which, while commonly used across literature, are not free from flaws (Nixon et al., 2019).

Original: However, in our work, we empirically demonstrate that despite their flaws, the ECE metric still provides good correlations to measuring confirmation bias and test accuracy.
Revised: However, in our work, we empirically demonstrate that despite their flaws, the ECE metric still provides good correlations to measuring confirmation bias and test accuracy.

Original:  R EPRODUCIBILITY S TATEMENT
Revised:  R EPRODUCIBILITY S TATEMENT

Original: To ensure that our reported results are reproducible, we provide our anonymized source codein the supplementary materials, which will be made available  publicly upon acceptance.
Revised: To ensure that our reported results are reproducible, all source code will be made available in a publicly hosted repository soon.

Original: We further list all details on training configurations and hyperparameter settings in Section 5 and Appendix C
Revised: We further list all details on training configurations and hyperparameter settings in Section 5 and Appendix D.

Revised: A C OMPARISON WITH P RIOR A RT IN SSL

Revised: [Table]

Original: A P ROOF OF C OROLLARY
Revised: B P ROOF OF C OROLLARY

Original: PAC-Bayesian theory aims to provide bounds on the generalization risk under the assumption that samples are i.i.d.. While PAC-Bayesian bounds typically apply to bounded losses, Germain et al.
Revised: PAC-Bayesian theory aims to provide bounds on the generalization risk under the assumption that samples are i.i.d.. While PAC-Bayesian bounds typically apply to bounded losses, Germain et al.

Original: (2016) extends them to unbounded losses (which is necessary in our work since it uses the unbounded log-loss).
Revised: (2016) extends them to unbounded losses (which is necessary in our work since it uses the unbounded log-loss).

Original: Their theoretical result is reproduced below in Theorem 1, under assumptions for subGaussian losses.
Revised: Their theoretical result is reproduced below in Theorem 1, under assumptions for subGaussian losses.

Original: These bounds commonly assume the supervised setting; in this proof, our goal is to extend them to the semi-supervised learning setting where in addition to labels from an empirical set, our loss is also mediated by pseudo-labels assigned by the model.
Revised: These bounds commonly assume the supervised setting; in this proof, our goal is to extend them to the semi-supervised learning setting where in addition to labels from an empirical set, our loss is also mediated by pseudo-labels assigned by the model.

Original: Let ( X, Y ) ∼ D N denote N i.i.d.
Revised: Let ( X, Y ) ∼ D N denote N i.i.d.

Original: training samples from the data distribution D , where ( X, Y )
Revised: training samples from the data distribution D , where ( X, Y )

Original: = (cid:110) { ( x i , y i ) } N l i =1 , { ( x i , ˆ y i ) }
Revised: = (cid:110) { ( x i , y i ) } N l i =1 , { ( x i , ˆ y i ) }

Original: N u i =1 (cid:111) represents the set of N l labeled input-output pairs and N u pairs of unlabeled input and their model-assigned pseudo-labels ˆ y . 
Revised: N u i =1 (cid:111) represents the set of N l labeled input-output pairs and N u pairs of unlabeled input and their model-assigned pseudo-labels ˆ y . Here, N l + N u = N .

Original: Let the loss function be ℓ ( f, x, y )
Revised: Let the loss function be ℓ ( f, x, y )

Original: → R , the generalization risk on distribution D , i.e. L ℓ D ( f ) , and the empirical risk on the training set, i.e. L ℓX,Y ( f ) , is given by:
Revised: → R , the generalization risk on distribution D , i.e. L ℓ D ( f ) , and the empirical risk on the training set, i.e. L ℓX,Y ( f ) , is given by:

Original: [Equation]
Revised: [Equation]

Original: On assumptions that the loss is sub-Gaussian (see Boucheron et al.
Revised: On assumptions that the loss is sub-Gaussian (see Boucheron et al.

Original: (2013) section 2.3), i.e. a loss function ℓ is sub-Gaussian with variance s 2 under a prior π and D if it can be described by a sub-Gaussian random variable v =
Revised: (2013) section 2.3), i.e. a loss function ℓ is sub-Gaussian with variance s 2 under a prior π and D if it can be described by a sub-Gaussian random variable v =

Original: [Equation] x, y ) , the generalization error bounds are given by Theorem 1 (Germain et al., 2016) below.
Revised: L ℓ D ( f ) − ℓ ( f, x, y ) , the generalization error bounds are given by Theorem 1 (Germain et al., 2016) below.

Original: Theorem 1 (Corollary 4 from Germain et al.
Revised: Theorem 1 (Corollary 4 from Germain et al.

Original: Let π be the prior distribution and ˆ ρ be the posterior.
Revised: Let π be the prior distribution and ˆ ρ be the posterior.

Original: If the loss is sub-Gaussian with variance factor s 2 , with probability at least 1 − δ over the choice of ( X, Y ) ∼ D N ,
Revised: If the loss is sub-Gaussian with variance factor s 2 , with probability at least 1 − δ over the choice of ( X, Y ) ∼ D N ,

Original: [Equation]
Revised: [Equation]

Original: Beginning from Theorem 1, the generalization error of the variational posterior q ( θ | ϕ ) under the negative log likelihood loss and prior p ( θ ) in our SSL setting can be found by replacing ˆ ρ with q ( θ | ϕ ) and ℓ with ℓ nll ( f, x i , y i ) :
Revised: Beginning from Theorem 1, the generalization error of the variational posterior q ( θ | ϕ ) under the negative log likelihood loss and prior p ( θ ) in our SSL setting can be found by replacing ˆ ρ with q ( θ | ϕ ) and ℓ with ℓ nll [Equation]

Original: = − log p ( y i | x i ; θ ) ;
Revised: [Equation] , y i [Equation]

Original: [Equation]
Revised: [Equation]

Revised: ˆ y

Original: N
Revised: i

Original: [Equation]
Revised: [Equation]

Original: where we define ELBO =
Revised: where we define ELBO =

Original: E q ( θ | ϕ ) [log p
Revised: E q ( θ | ϕ ) [log p

Original: ( Y | X ; θ )] − KL( q ( θ | ϕ )
Revised: ( Y | X ; θ )] − KL( q ( θ | ϕ )

Original: ∥ P ( θ )) .
Revised: ∥ P ( θ )) .

Original: In the next section,
Revised: In the next section,

Original: Appendix B.1, we will see that this is the term we are maximizing in our loss objective.
Revised: Appendix C.1, we will see that this is the term we are maximizing in our loss objective.

Original: By maximizing the ELBO , i.e. minimizing the negative ELBO , we are minimizing the upper bound to the generalization error of our variational posterior.
Revised: By maximizing the ELBO , i.e. minimizing the negative ELBO , we are minimizing the upper bound to the generalization error of our variational posterior.

Original: The first term in the last line adds a divergence measure between the pseudo-label prediction distribution and the ground truth distribution — in the fully supervised setting ˆ y i
Revised: The first term in the last line adds a divergence measure between the pseudo-label prediction distribution and the ground truth distribution — in the fully supervised setting ˆ y i

Original: → y i and this term vanishes.
Revised: → y i and this term vanishes.

Original: B F ORMULATION DETAILS AND PSEUDOCODE OF OUR METHODS B. B AYESIAN MODEL AVERAGING VIA A BNN FINAL LAYER
Revised: C F ORMULATION DETAILS AND PSEUDOCODE OF OUR METHODS C. B AYESIAN MODEL AVERAGING VIA A BNN FINAL LAYER

Original: Following the notations in Section 4.1, we denote the BNN layer to be h and an input embedding to this layer to be v in this section.
Revised: Following the notations in Section 4.1, we denote the BNN layer to be h and an input embedding to this layer to be v in this section.

Original: We assume a prior distribution on weights P ( θ h ) and seek to calculate the posterior distribution of weights given the empirical/training data, P ( θ h |D X ) , where D X :=
Revised: We assume a prior distribution on weights P ( θ h ) and seek to calculate the posterior distribution of weights given the empirical/training data, P ( θ h |D X ) , where D X :=

Original: ( X, Y ) , which can then be used to compute the posterior predictive during inference.
Revised: ( X, Y ) , which can then be used to compute the posterior predictive during inference.

Original: As exact Bayesian inference is intractable for neural networks, we adopt a variational approach following (Blundell et al., 2015) to approximate the posterior with a Gaussian distribution parameterized by ϕ , q ( θ | ϕ ) .
Revised: As exact Bayesian inference is intractable for neural networks, we adopt a variational approach following (Blundell et al., 2015) to approximate the posterior with a Gaussian distribution parameterized by ϕ , q ( θ | ϕ ) .

Original: From now, we will drop the h index in θ h for brevity.
Revised: From now, we will drop the h index in θ h for brevity.

Original: To learn the variational approximation, we seek to minimize the Kullback-Leibler (KL) divergence between the Gaussian variational approximation and the posterior:
Revised: To learn the variational approximation, we seek to minimize the Kullback-Leibler (KL) divergence between the Gaussian variational approximation and the posterior:

Original: [Equation]
Revised: [Equation]

Original: where in the last line, ELBO =
Revised: where in the last line, ELBO =

Original: E q ( θ | ϕ ) [log p
Revised: E q ( θ | ϕ ) [log p

Original: ( Y | X ; θ )] − KL( q ( θ | ϕ )
Revised: ( Y | X ; θ )] − KL( q ( θ | ϕ )

Original: ∥ p ( θ )) is the evidence lowerbound which consists of a log-likelihood (data-dependent) term and a KL (prior-dependent) term.
Revised: ∥ p ( θ )) is the evidence lowerbound which consists of a log-likelihood (data-dependent) term and a KL (prior-dependent) term.

Revised: [Table]

Original: Since KL ( q ( θ | ϕ ) ∥ p ( θ |D X )) is intractable, we maximize the ELBO which is equivalent to minimizing the former up to the constant, log p ( Y | X ) .
Revised: Since KL ( q ( θ | ϕ ) ∥ p ( θ |D X )) is intractable, we maximize the ELBO which is equivalent to minimizing the former up to the constant, log p ( Y | X ) .

Original: Each variational posterior parameter of the Gaussian distribution, ϕ , consists of the mean ( µ ) and the standard deviation (which is parametrized as σ = log(1 + exp( ρ )) so that σ is always positive (Blundell et al., 2015)), i.e. ϕ = ( µ, ρ ) .
Revised: Each variational posterior parameter of the Gaussian distribution, ϕ , consists of the mean ( µ ) and the standard deviation (which is parametrized as σ = log(1 + exp( ρ )) so that σ is always positive (Blundell et al., 2015)), i.e. ϕ = ( µ, ρ ) .

Original: To obtain a sample of the weights θ , we use the reparametrization trick (Kingma et al., 2015) and sample ϵ ∼ N (0 , I ) to get θ = µ + log(1 + exp( ρ )) ◦ ϵ where ◦ denotes elementwise multiplication.
Revised: To obtain a sample of the weights θ , we use the reparametrization trick (Kingma et al., 2015) and sample ϵ ∼ N (0 , I ) to get θ = µ + log(1 + exp( ρ )) ◦ ϵ where ◦ denotes elementwise multiplication.

Original: In other words, we double the number of learnable parameters in the layer compared to a non-Bayesian approach; however, this does not add a huge computational cost since only the last layer is Bayesian and the dense backbone remains non-Bayesian.
Revised: In other words, we double the number of learnable parameters in the layer compared to a non-Bayesian approach; however, this does not add a huge computational cost since only the last layer is Bayesian and the dense backbone remains non-Bayesian.

Original: Algorithm 1 shows the PyTorch-style pseudo-code for BAM- in threshold-mediated methods (here showing asymmetric augmentation applicable for UDA or FixMatch).
Revised: Algorithm 1 shows the PyTorch-style pseudo-code for BAM- in threshold-mediated methods (here showing asymmetric augmentation applicable for UDA or FixMatch).

Original: The main modifications upon the baseline methods include 1) computation of an additional KL loss term (between two Gaussians, i.e. the variational approximation and the prior), 2) taking multiple samples of weights to derive predictions and 3) replacing the acceptance criteria from using maximum probability to using standard deviation of predictions.
Revised: The main modifications upon the baseline methods include 1) computation of an additional KL loss term (between two Gaussians, i.e. the variational approximation and the prior), 2) taking multiple samples of weights to derive predictions and 3) replacing the acceptance criteria from using maximum probability to using standard deviation of predictions.

Original: B. A LGORITHM FOR SWA & EMA IN PAWS
Revised: C. A LGORITHM FOR SWA & EMA IN PAWS

Original: Algorithm 2 shows the pseudocode for the implementation of PAWS+SWA and PAWS+EMA in
Revised: Algorithm 2 shows the pseudocode for the implementation of PAWS+SWA and PAWS+EMA in

Original: PyTorch.
Revised: PyTorch.

Original: For brevity, we leave out details of the multicrop strategy, mean entropy maximization regularization and soft nearest neighbour classifier formulation which are all replicated from the original implementation.
Revised: For brevity, we leave out details of the multicrop strategy, mean entropy maximization regularization and soft nearest neighbour classifier formulation which are all replicated from the original implementation.

Original: We defer readers to the original paper (Assran et al., 2021) for these details.
Revised: We defer readers to the original paper (Assran et al., 2021) for these details.

Original: [Table]

Original: [Table]
Revised: [Table]

Original: C F URTHER IMPLEMENTATION DETAILS C. H YPERPARAMETERS FOR VARIOUS THRESHOLD - MEDIATED METHODS
Revised: D F URTHER IMPLEMENTATION DETAILS D. H YPERPARAMETERS FOR VARIOUS THRESHOLD - MEDIATED METHODS

Original: All threshold-mediated methods in this study uses an optimization loss function of the form of Eq.
Revised: All threshold-mediated methods in this study uses an optimization loss function of the form of Eq.

Original: (1), with differences in the hyperparameters µ , λ , τ , ρ t and α .
Revised: (1), with differences in the hyperparameters µ , λ , τ , ρ t and α .

Original: We use the hyperparameters from the original implementations.
Revised: We use the hyperparameters from the original implementations.

Original: For Pseudo-Labels (Lee, 2013), µ = 1 , τ = 0 .
Revised: For Pseudo-Labels (Lee, 2013), µ = 1 , τ = 0 .

Original: 95 and ρ t =

Revised: 95 and ρ t =

Original: (i.e. hard labels); for UDA (Xie et al., 2019a), µ = 7 , τ = 0 .
Revised: (i.e. hard labels); for UDA (Xie et al., 2019a), µ = 7 , τ = 0 .

Original: 8 , ρ t =0 .
Revised: 8 , ρ t =0 .

Original: 4 (i.e. soft pseudo-labels sharpened with temperature of 0.4); for FixMatch (Sohn et al., 2020), µ = 7 , τ = 0 .
Revised: 4 (i.e. soft pseudo-labels sharpened with temperature of 0.4); for FixMatch (Sohn et al., 2020), µ = 7 , τ = 0 .

Original: 95 , ρ t =

Revised: 95 , ρ t =

Original: (i.e. hard labels).
Revised: (i.e. hard labels).

Original: All methods use λ = 1 .
Revised: All methods use λ = 1 .

Original: In addition, UDA and FixMatch uses asymmetric transforms for the two legs of sample and pseudo-label prediction, i.e. α 1 is a weak transform (based on the standard flip-and-shift augmentation) and α 2 is a strong transform (based on RandAugment (Cubuk et al., 2019)).
Revised: In addition, UDA and FixMatch uses asymmetric transforms for the two legs of sample and pseudo-label prediction, i.e. α 1 is a weak transform (based on the standard flip-and-shift augmentation) and α 2 is a strong transform (based on RandAugment (Cubuk et al., 2019)).

Original: Pseudo-Labels uses symmetric weak transforms for both legs.
Revised: Pseudo-Labels uses symmetric weak transforms for both legs.

Original: In our calibrated versions of all these methods (i.e. “BAM-X”), we maintained the exact same hyperparameter configuration as its corresponding baseline.
Revised: In our calibrated versions of all these methods (i.e. “BAM-X”), we maintained the exact same hyperparameter configuration as its corresponding baseline.

Original: The only exception is the sharpening temperature of BAM-UDA, which uses t = 0 .
Revised: The only exception is the sharpening temperature of BAM-UDA, which uses t = 0 .

Original: 9 instead of t = 0 .
Revised: 9 instead of t = 0 .

Original: 4 in UDA, as we found that calibration enables, and was highly effective with, the use of soft pseudo-labels (see discussion in Section 6).
Revised: 4 in UDA, as we found that calibration enables, and was highly effective with, the use of soft pseudo-labels (see discussion in Section 6).

Original: C. I MPLEMENTATION DETAILS FOR I MAGE N ET EXPERIMENTS
Revised: D. I MPLEMENTATION DETAILS FOR I MAGE N ET EXPERIMENTS

Original: We maintain the default 64 GPU training configuration recommended by the authors and make slight modifications to the default implementations to fit on our hardware.
Revised: We maintain the default 64 GPU training configuration recommended by the authors and make slight modifications to the default implementations to fit on our hardware.

Original: On our set up (which does not use
Revised: On our set up (which does not use

Original: Nvidia’s Apex package due to installations issues), we found training to be unstable on half-precision and had to use full-precision training.
Revised: Nvidia’s Apex package due to installations issues), we found training to be unstable on half-precision and had to use full-precision training.

Original: In order to fit into memory, we had to decrease the number of images per class from 7 to 6, resulting in a slightly lower baseline performance from the reported for 200 epochs of training (see Table 4 in Assran et al.
Revised: In order to fit into memory, we had to decrease the number of images per class from 7 to 6, resulting in a slightly lower baseline performance from the reported for 200 epochs of training (see Table 4 in Assran et al.

Original: (2021) for the study on the correlation between the number of images per class and the final accuracy).
Revised: (2021) for the study on the correlation between the number of images per class and the final accuracy).

Original: On all ImageNet experiments on PAWS, we follow the validation and testing procedure of PAWS (Assran et al., 2021) and swept over the same set of hyperparameters during fine-tuning of the linear head.
Revised: On all ImageNet experiments on PAWS, we follow the validation and testing procedure of PAWS (Assran et al., 2021) and swept over the same set of hyperparameters during fine-tuning of the linear head.

Original: We report the ECE at the final checkpoint.
Revised: We report the ECE at the final checkpoint.

Original: D L ONG - TAILED CIFAR-10 AND CIFARD. D ATASET PREPARATION
Revised: E L ONG - TAILED CIFAR-10 AND CIFARE. D ATASET PREPARATION

Original: We create long-tailed versions of CIFAR-10 and CIFAR-100 following the procedure from Cao et al.
Revised: We create long-tailed versions of CIFAR-10 and CIFAR-100 following the procedure from Cao et al.

Original: (2019), i.e. by removing the number of training examples per class from the standard training set with 50,000 samples.
Revised: (2019), i.e. by removing the number of training examples per class from the standard training set with 50,000 samples.

Original: We create the class-imbalance unlabeled dataset with an exponential decay where the severity of the imbalance is given by the imbalance ratio α = max i ( n u,i ) /
Revised: We create the class-imbalance unlabeled dataset with an exponential decay where the severity of the imbalance is given by the imbalance ratio α = max i ( n u,i ) /

Original: min i ( n u,i ) ∈ { 10 , 100 } , where n u,i is the number of unlabeled examples for class i .
Revised: min i ( n u,i ) ∈ { 10 , 100 } , where n u,i is the number of unlabeled examples for class i .

Original: The number of samples in the most frequent class is 5,000 for CIFAR-10 and 500 for CIFAR-100.
Revised: The number of samples in the most frequent class is 5,000 for CIFAR-10 and 500 for CIFAR-100.

Original: To create the labeled set, we randomly select 10% of samples from each class , under the constrain that at least 1 sample for each class is included in the labeled set, i.e. n l,i = min(1 , 0 .
Revised: To create the labeled set, we randomly select 10% of samples from each class , under the constrain that at least 1 sample for each class is included in the labeled set, i.e. n l,i = min(1 , 0 .

Original: 1 ∗ n u,i ) , where n l,i is the number of labeled examples for class i .
Revised: 1 ∗ n u,i ) , where n l,i is the number of labeled examples for class i .

Original: The total number of labeled and unlabeled examples for the different benchmarks in
Revised: The total number of labeled and unlabeled examples for the different benchmarks in

Original: CIFAR-10-LT and CIFAR-100-LT are summarized in Table 5.
Revised: CIFAR-10-LT and CIFAR-100-LT are summarized in Table 6.

Original: The test set remains unchanged, i.e. we use the original (class-balanced) test set of the CIFAR datasets with 10,000 samples.
Revised: The test set remains unchanged, i.e. we use the original (class-balanced) test set of the CIFAR datasets with 10,000 samples.

Original: [Table]
Revised: [Table]

Original: -100-LT α =α

Original: =α =α = Labeled 2, 1, 1, 1, Total 20, 12, 19, 10, We used the exact same configuration and hyperparameters as the original (non-long-tailed) CIFAR benchmarks, i.e. the architecture is WideResNet-28-2 for CIFAR-10-LT and WideResNet-28for CIFAR-100-LT.
Revised: We used the exact same configuration and hyperparameters as the original (non-long-tailed) CIFAR benchmarks, i.e. the architecture is WideResNet-28-2 for CIFAR-10-LT and WideResNet-28for CIFAR-100-LT.

Original: All training hyperparameters used to obtain the results for FM and BAM-UDA in Table 3 also follow the ones from the original CIFAR benchmarks.
Revised: All training hyperparameters used to obtain the results for FM and BAM-UDA in Table 3 also follow the ones from the original CIFAR benchmarks.

Original: D. A DDITIONAL CLASSWISE RESULTS
Revised: E. A DDITIONAL CLASSWISE RESULTS

Original: Fig.
Revised: Fig.

Original: 4 shows the test accuracies for the baseline (FM) and ours (BAM-UDA) after separating samples from the test set into three groups — the “Many” group which contains classes with more than 100 samples, the “medium” group which contains classes between 10 and 100 samples and the “few” group which contains classes less than 10 samples, for the benchmark of CIFAR-100-LT with α = 10 and 10% labels.
Revised: 5 shows the test accuracies for the baseline (FM) and ours (BAM-UDA) after separating samples from the test set into three groups — the “Many” group which contains classes with more than 100 samples, the “medium” group which contains classes between 10 and 100 samples and the “few” group which contains classes less than 10 samples, for the benchmark of CIFAR-100-LT with α = 10 and 10% labels.

Original: We see that BAM-UDA outperforms FM on every group; in addition, the gap between BAM-UDA and FM increases as the number of samples are more scarce, highlighting
Revised: We see that BAM-UDA outperforms FM on every group; in addition, the gap between BAM-UDA and FM increases as the number of samples are more scarce, highlighting

Original: BAM-UDA’s utility in improving accuracy over the baseline in the more difficult classes.
Revised: BAM-UDA’s utility in improving accuracy over the baseline in the more difficult classes.

Original: [Figure]
Revised: [Figure]

Original: E S EMI - SUPERVISED L EARNING IN P HOTONICS S CIENCE
Revised: F S EMI - SUPERVISED L EARNING IN P HOTONICS S CIENCE

Original: Semi-supervised learning is highly important and applicable to domains like Science where labeled data is particularly scarce, owing to the need for computationally expensive simulations and laborintensive laboratory experiments for data collection.
Revised: Semi-supervised learning is highly important and applicable to domains like Science where labeled data is particularly scarce, owing to the need for computationally expensive simulations and laborintensive laboratory experiments for data collection.

Original: To demonstrate the effectiveness of our proposed techniques in the Science domain, we use an example problem in Photonics, adopting the datasets from Loh et al.
Revised: To demonstrate the effectiveness of our proposed techniques in the Science domain, we use an example problem in Photonics, adopting the datasets from Loh et al.

Original: The task we studied in this work is a 5-way classification of photonic crystals (PhC) based on their band gap sizes.
Revised: The task we studied in this work is a 5-way classification of photonic crystals (PhC) based on their band gap sizes.

Original: PhCs are periodically-structured materials engineered for wide ranging applications by manipulating light waves (Joannopoulos et al., 2008) and an important property of these crystals is the size of their band gap (often, engineers seek to design photonic crystals that host a substantial band gap (Christensen et al., 2020)).
Revised: PhCs are periodically-structured materials engineered for wide ranging applications by manipulating light waves (Joannopoulos et al., 2008) and an important property of these crystals is the size of their band gap (often, engineers seek to design photonic crystals that host a substantial band gap (Christensen et al., 2020)).

Original: We adopt the dataset of PhCs from Loh et al.
Revised: We adopt the dataset of PhCs from Loh et al.

Original: (2022), which consists of 32,000 PhC samples and their corresponding band gaps which had been pre-computed through numerical simulations (Johnson & Joannopoulos, 2001).
Revised: (2022), which consists of 32,000 PhC samples and their corresponding band gaps which had been pre-computed through numerical simulations (Johnson & Joannopoulos, 2001).

Original: Examples of PhCs and an illustration of band gap from the dataset is shown in Fig.
Revised: Examples of PhCs and an illustration of band gap from the dataset is shown in Fig.

Original: We binned all samples in the dataset into 5 classes based on their band gap sizes; since there was a preponderance (about 25,000) of samples without a band gap, we only selected 5,000 of them in order to limit the severity of class imbalance and form a new dataset with just 11,375 samples (see Fig. 5c).
Revised: We binned all samples in the dataset into 5 classes based on their band gap sizes; since there was a preponderance (about 25,000) of samples without a band gap, we only selected 5,000 of them in order to limit the severity of class imbalance and form a new dataset with just 11,375 samples (see Fig. 6c).

Original: Notably the long-tailed distribution of this dataset is characteristic of many problems in science (and other real-world datasets), where samples with the desired properties (larger band gap) are much rarer than trivial samples.
Revised: Notably the long-tailed distribution of this dataset is characteristic of many problems in science (and other real-world datasets), where samples with the desired properties (larger band gap) are much rarer than trivial samples.

Original: From this reduced dataset, we created two PhC benchmarks with 10% labels (PhC-10%) and 1% labels (PhC-1%), where 10% and 1% of samples from each class are randomly selected to form the labeled set respectively.
Revised: From this reduced dataset, we created two PhC benchmarks with 10% labels (PhC-10%) and 1% labels (PhC-1%), where 10% and 1% of samples from each class are randomly selected to form the labeled set respectively.

Original: The class-balanced test set is fixed with 300 samples per class (total of 1,500 samples), the unlabeled[Figure] set consists of 9,876 samples and the labeled sets consists of 1,136 samples and 113 samples for
Revised: The class-balanced test set is fixed with 300 samples per class (total of 1,500 samples), the unlabeled set consists of 9,876 samples and the labeled sets consists of 1,136 samples and 113 samples for

Original: PhC-10% and PhC-1% respectively.
Revised: PhC-10% and PhC-1% respectively.

Original: For these benchmarks, we used a WideResNet-28-2 architecture, with a single channel for the first CNN layer, and made the following changes upon FixMatch and BAM-UDA.
Revised: For these benchmarks, we used a WideResNet-28-2 architecture, with a single channel for the first CNN layer, and made the following changes upon FixMatch and BAM-UDA.

Original: We set µ =and instead of 2 20 iterations, we trained for 300 epochs (where each epoch is defined as iterating through the unlabeled set once).
Revised: We set µ =and instead of 2 20 iterations, we trained for 300 epochs (where each epoch is defined as iterating through the unlabeled set once).

Original: For BAM-UDA, we set Q = 0 .
Revised: For BAM-UDA, we set Q = 0 .

Original: 9 and use a one-minus-cosine warm-up scheduler of 50 epochs to the KL loss coefficient (resulting in a coefficient going fromto 1 in 50 epochs).
Revised: 9 and use a one-minus-cosine warm-up scheduler of 50 epochs to the KL loss coefficient (resulting in a coefficient going fromto 1 in 50 epochs).

Original: For each benchmark, we swept the learning rate across { 0 . 01 , 0 . 001 } and select the best model for both the baseline and ours.
Revised: For each benchmark, we swept the learning rate across { 0 . 01 , 0 . 001 } and select the best model for both the baseline and ours.

Original: The standard image augmentations used in vision problems cannot be applied to this problem, since it destroys the scientific integrity of the data (e.g. cropping the PhC input would result in a completely different band gap profile).
Revised: The standard image augmentations used in vision problems cannot be applied to this problem, since it destroys the scientific integrity of the data (e.g. cropping the PhC input would result in a completely different band gap profile).

Original: Instead, we used the augmentations proposed in Loh et al.
Revised: Instead, we used the augmentations proposed in Loh et al.

Original: (periodic translations, rotations and mirrors) and applied them symmetrically (i.e. there is no distinction of strong and weak augmentations).
Revised: (periodic translations, rotations and mirrors) and applied them symmetrically (i.e. there is no distinction of strong and weak augmentations).

Original: In addition, we included these augmentations to the labeled samples as well.
Revised: In addition, we included these augmentations to the labeled samples as well.

Revised: [Figure]

Original: F A BLATION S TUDIES F. WEIGHT AVERAGING IN THRESHOLD - MEDIATED METHODS
Revised: G A BLATION S TUDIES G. WEIGHT AVERAGING IN THRESHOLD - MEDIATED METHODS

Original: For our experiments incorporating EMA during pseudo-labeling for FixMatch, we use the second momentum scheduler defined in Section 5, i.e. a one-minus-cosine scheduler starting at (1 − x ) and decreasing to 0, resulting in a [ x, 1] range for γ .
Revised: For our experiments incorporating EMA during pseudo-labeling for FixMatch, we use the second momentum scheduler defined in Section 5, i e. a one-minus-cosine scheduler starting at (1 − x ) and decreasing to 0, resulting in a [ x, 1] range for γ .

Original: While PAWS is effective with high γ values(i.e. x = 0 .
Revised: While PAWS is effective with high γ values

Original: , we found that FixMatch needed much smaller momentum values at the start of training and set x = 0 .
Revised: (i e. x = 0 . 95 ), we found that FixMatch needed much smaller momentum values at the start of training and set x = 0 .

Original: Note that in the default implementation of FixMatch, an EMA of constant γ = 0 .
Revised: Note that in the default implementation of FixMatch, an EMA of constant γ = 0 .

Original: 999 is also used, however it is used only for test evaluation and not during training.
Revised: 999 is also used, however it is used only for test evaluation and not during training.

Original: In our experiments, we use the EMA weight aggregate for pseudo-labeling throughout training and use the same weight aggregate after training for test evaluation.
Revised: In our experiments, we use the EMA weight aggregate for pseudo-labeling throughout training and use the same weight aggregate after training for test evaluation.

Original: Results are shown in Table 6, where we see that using weight averaging was also effective at improving calibration in FixMatch and can lead to improved test accuracies.
Revised: Results are shown in Table 7, where we see that using weight averaging was also effective at improving calibration in FixMatch and can lead to improved test accuracies.

Original: However, EMA weight averaging was not so effective when the labels are less scarce and further, it consistently under-performs the Bayesian model averaging approach.
Revised: However, EMA weight averaging was not so effective when the labels are less scarce and further, it consistently under-performs the Bayesian model averaging approach.

Revised: [Table]

Original: The desirability of Bayesian model averaging over EMA weight averaging for threshold-mediated
Revised: The desirability of Bayesian model averaging over EMA weight averaging for threshold-mediated

Original: SSL methods could be attributed to replacing the selection metric with better uncertainty estimates given by the BNN layer, which is not available for EMA weight averaging approaches.
Revised: SSL methods could be attributed to replacing the selection metric with better uncertainty estimates given by the BNN layer, which is not available for EMA weight averaging approaches.

Original: To further investigate this claim, we perform ablation studies on BAM-UDA to isolate the contribution of averaging predictions from the contribution of replacing the selection metric.
Revised: To further investigate this claim, we perform ablation studies on BAM-UDA to isolate the contribution of averaging predictions from the contribution of replacing the selection metric.

Original: Results are shown 
Revised: Results are shown in Table 8, rows indicated with “BNN no σ 2 ” show experiments using a BNN layer in BAM-UDA only for averaging predictions while maintaining the original selection metric, i.e. pseudo-labels are accepted if the maximum prediction class probability is greater than τ = 0 . Comparing the first two rows, indeed we see that by not using the uncertainty estimates from the BNN, we get marginal improvements especially when labels are not so scarce ( 72 . 9 → 73 . 0 for CIFAR-100-4000 labels), corroborating our observations with FM+EMA. The difference between the last two rows show the effect of replacing the selection metric and indeed we observe consistent gains across the benchmarks from doing so.

Original: [Table]
Revised: [Table]

Original: in Table rows indicated

Original: “BNN σ for averaging predictions while maintaining the original selection metric e. pseudo labels are accepted if the than

Original: = indeed

Original: [Table]
Revised: [Table]

Original: F. A BLATION E XPERIMENTS ON B AYESIAN MODEL AVERAGING
Revised: G. A BLATION E XPERIMENTS ON B AYESIAN MODEL AVERAGING

Original: From our main results in Section 6, we found that our Bayesian model averaging approaches were more effective in conjunction with soft pseudo-labels.
Revised: From our main results in Section [Table] we found that our Bayesian model averaging approaches were more effective in conjunction with soft pseudo-labels.

Original: In Table 8, we show ablation experiments on the temperature of the sharpening operation on the CIFAR-100-400 labels benchmark.
Revised: In Table [Table] we show ablation experiments on the temperature of the sharpening operation on the CIFAR-100-400 labels benchmark.

Original: Overall, we observe a trend that softer pseudo-labels (i.e. reducing the sharpening of pseudo-labels) led to better calibration and improved test performance.
Revised: Overall, we observe a trend that softer pseudo-labels (i [Table] e. reducing the sharpening of pseudo-labels) led to better calibration and improved test performance.

Original: As such, in our experiments we modify upon the original sharpening parameter of UDA and set t = 0 .
Revised: As such, in our experiments we modify upon the original sharpening parameter of UDA and set t = 0 .

Original: 9 for BAM-UDA in all our benchmarks.
Revised: 9 for BAM-UDA in all our benchmarks

Original: [Table]
Revised: [Table]

Revised: [Table]

Original: Bayesian model averaging produces better calibrated predictions through “Bayesian marginalization”, i.e. by averaging over multiple predictions instead of using a single prediction of the model.
Revised: Bayesian model averaging produces better calibrated predictions through “Bayesian marginalization”, i.e. by averaging over multiple predictions instead of using a single prediction of the model.

Original: In Table 9, we show ablation studies on the number of weight samples taken from the variational posterior in the BNN layer used for “Bayesian marginalization”, i.e. for computing the posterior predictive.
Revised: In Table 10, we show ablation studies on the number of weight samples taken from the variational posterior in the BNN layer used for “Bayesian marginalization”, i.e. for computing the posterior predictive.

Original: We observe an overall trend that increasing the number of weight samples lead to better calibration and final test accuracies, providing direct evidence for the effectiveness in using Bayesian model averaging towards improving calibration.
Revised: We observe an overall trend that increasing the number of weight samples lead to better calibration and final test accuracies, providing direct evidence for the effectiveness in using Bayesian model averaging towards improving calibration.

Original: In order to limit the computational overhead arising from taking a large number of weight samples, given that pseudo-labeling happens at every iteration, we limit M to 50 in our study.
Revised: In order to limit the computational overhead arising from taking a large number of weight samples, given that pseudo-labeling happens at every iteration, we limit M to 50 in our study.

Original: The computational overhead for M = 50 is discussed in Appendix G, where we see that incorporating Bayesian model averaging incurs negligible computational overhead.
Revised: The computational overhead for M = 50 is discussed in Appendix H, where we see that incorporating Bayesian model averaging incurs negligible computational overhead.

Original: [Table]
Revised: [Table]

Original: F. A BLATION E XPERIMENTS ON PAWS-SWA & PAWS-EMA Training times before switching on SWA.
Revised: G. A BLATION E XPERIMENTS ON PAWS-SWA & PAWS-EMA Training times before switching on SWA.

Original: Fig.
Revised: Fig.

Original: 6 shows the test accuracy and ECE across training for PAWS-SWA and PAWS on the CIFAR-10 benchmark with 4000 labels and explores the effect of varying the time in which SWA was switched on.
Revised: 7 shows the test accuracy and ECE across training for PAWS-SWA and PAWS on the CIFAR-10 benchmark with 4000 labels and explores the effect of varying the time in which SWA was switched on.

Original: In particular, we found that switching on SWA later during training is more desirable.
Revised: In particular, we found that switching on SWA later during training is more desirable.

Original: Unlike EMA which has a natural curriculum to give more weight to the more recent network parameters in the aggregate, SWA weighs all models in the aggregate equally and thus including the poorly optimized weights from the start of training is undesirable.
Revised: Unlike EMA which has a natural curriculum to give more weight to the more recent network parameters in the aggregate, SWA weighs all models in the aggregate equally and thus including the poorly optimized weights from the start of training is undesirable.

Original: [Figure]
Revised: [Figure]

Original: switching on SWA.

Original: We experimented with switching on SWA after { 10 , 50 , 100 , 200 } epochs of training (of the baseline).

Original: Dataset is CIFARwith 4000 labels.

Original: Momentum schedules for PAWS-EMA.
Revised: Momentum schedules for PAWS-EMA.

Original: Table Table 10 explores different schedules for the γ parameter used in PAWS-EMA on the CIFAR-100, 4000 labels benchmark.
Revised: Table Table 11 explores different schedules for the γ parameter used in PAWS-EMA on the CIFAR-100, 4000 labels benchmark.

Original: For brevity, we consider a simple schedule where γ is linearly warmed up from 0 to γ max in the first W epochs, and maintains at γ max thereafter.
Revised: For brevity, we consider a simple schedule where γ is linearly warmed up from 0 to γ max in the first W epochs, and maintains at γ max thereafter.

Original: In Table 10, we varied W ∈ { 0 , 10 , 50 } and γ max ∈ { 0 . 5 , 0 . 9 , 0 . 99 , 0 . 999 } .
Revised: In Table 11, we varied W ∈ { 0 , 10 , 50 } and γ max ∈ { 0 . 5 , 0 . 9 , 0 . 99 , 0 . 999 } .

Original: We observe that using a very high γ (i.e. 0.999) from the start of training is detrimental to both test performance and ECE, while fixing γ to a moderately high value (i.e. 0.9) across training was also less desirable than stepping up γ in the first 50 epochs.
Revised: We observe that using a very high γ (i.e. 0.999) from the start of training is detrimental to both test performance and ECE, while fixing γ to a moderately high value (i.e. 0.9) across training was also less desirable than stepping up γ in the first 50 epochs.

Original: Notably, these observations highlight distinct differences with the EMA commonly used in dual-network methods of self-supervised learning (Grill et al., 2020b; He et al., 2020) where a very high momentum value (0.996 to 0.999) is usually recommended.
Revised: Notably, these observations highlight distinct differences with the EMA commonly used in dual-network methods of self-supervised learning (Grill et al., 2020b; He et al., 2020) where a very high momentum value (0.996 to 0.999) is usually recommended.

Original: With this insight, in this work, we explored two schedules for γ : 1) linear warm-up from 0 to 0.996 for 50 epochs and maintaining at 0.996 (highlighted in cyan in Table 10) and 2) using one-minus-cosine schedule starting from 0.05 and decreasing to 0, resulting in a 0.95 to 1.0 range for γ .
Revised: With this insight, in this work, we explored two schedules for γ : 1) linear warm-up from 0 to 0.996 for 50 epochs and maintaining at 0.996 (highlighted in cyan in Table 11) and 2) using one-minus-cosine schedule starting from 0.05 and decreasing to 0, resulting in a 0.95 to 1.0 range for γ .

Original: These two schedules are visualized in Fig.
Revised: These two schedules are visualized in Fig.

Original: We use (1) for the CIFAR benchmarks and we found (2) to be more effective on the ImageNet benchmark.
Revised: We use (1) for the CIFAR benchmarks and we found (2) to be more effective on the ImageNet benchmark.

Original: [Table] [Figure] 
Revised: [Figure] [Table]

Original: G C OMPUTATIONAL R EQUIREMENTS AND A DDITIONAL C OMPUTATIONAL C OSTS
Revised: H C OMPUTATIONAL R EQUIREMENTS AND A DDITIONAL C OMPUTATIONAL C OSTS

Original: All CIFAR-10 and CIFAR-100 experiments in this work were computed using a single Nvidia
Revised: All CIFAR-10 and CIFAR-100 experiments in this work were computed using a single Nvidia

Original: V100 GPU.
Revised: V100 GPU.

Original: ImageNet experiments were computed using 64 Nvidia V100 GPUs.
Revised: ImageNet experiments were computed using 64 Nvidia V100 GPUs.

Original: A key aspect of our proposed calibration methods is the requirement of adding minimal computational cost to the baseline approaches.
Revised: A key aspect of our proposed calibration methods is the requirement of adding minimal computational cost to the baseline approaches.

Original: In the following paragraphs we list the additional computational cost (based on wall-clock time on the same hardware) for our proposed methods.
Revised: In the following paragraphs we list the additional computational cost (based on wall-clock time on the same hardware) for our proposed methods.

Original: Computational cost of Bayesian model averaging.
Revised: Computational cost of Bayesian model averaging.

Original: The main additional computational cost comes from executing several (in our case, 50) forward passes through the weight samples of the BNN layer when deriving predictions for the unlabeled samples.
Revised: The main additional computational cost comes from executing several (in our case, 50) forward passes through the weight samples of the BNN layer when deriving predictions for the unlabeled samples.

Original: This additional overhead is minimal since only the final layer, which consists of a small fraction of the total network weights, is (approximate)
Revised: This additional overhead is minimal since only the final layer, which consists of a small fraction of the total network weights, is (approximate)

Original: Bayesian.
Revised: Bayesian.

Original: We tested this on the benchmarks of CIFAR-100 and found the BNN versions to take only around 2 − 5 % longer in wall-clock time on the same hardware when compared to the baseline, for the same total number of iterations.
Revised: We tested this on the benchmarks of CIFAR-100 and found the BNN versions to take only around 2 − 5 % longer in wall-clock time on the same hardware when compared to the baseline, for the same total number of iterations.

Original: This justifies the BNN layer as a plug-in calibration approach with low computational overhead.
Revised: This justifies the BNN layer as a plug-in calibration approach with low computational overhead.

Original: Computational cost of SWA & EMA.
Revised: Computational cost of SWA & EMA.

Original: Both SWA and EMA requires storing another network of the same architecture, denoted g in the main text, which maintains the aggregate (exponentially weighted aggregate) of past network weights for SWA (EMA).
Revised: Both SWA and EMA requires storing another network of the same architecture, denoted g in the main text, which maintains the aggregate (exponentially weighted aggregate) of past network weights for SWA (EMA).

Original: Rather than using representations from the original backbone, f , representations from g are used to derive the better calibrated predictions, and thus the main computational overhead comes from the second forward pass needed per iteration.
Revised: Rather than using representations from the original backbone, f , representations from g are used to derive the better calibrated predictions, and thus the main computational overhead comes from the second forward pass needed per iteration.

Original: We timed the methods and found that PAWS+SWA and PAWS+EMA took around 18% longer in wall clock training time (on the same hardware) when compared to the baseline PAWS method for each epoch of training.
Revised: We timed the methods and found that PAWS+SWA and PAWS+EMA took around 18% longer in wall clock training time (on the same hardware) when compared to the baseline PAWS method for each epoch of training.

Original: However, as shown in the main text, PAWS+SWA and PAW+EMA resulted in a significant speed up in convergence, cutting training time by > 60% and additionally giving better test performances.
Revised: However, as shown in the main text, PAWS+SWA and PAW+EMA resulted in a significant speed up in convergence, cutting training time by > 60% and additionally giving better test performances.

Original: Hence rather than an overhead, improved calibration in PAWS+SWA and
Revised: Hence rather than an overhead, improved calibration in PAWS+SWA and

Original: PAWS+EMA in fact reduces the computation cost needed for the original approach.
Revised: PAWS+EMA in fact reduces the computation cost needed for the original approach.

Original: H S OCIETAL I MPACT AND E THICAL C ONSIDERATIONS
Revised: I S OCIETAL I MPACT AND E THICAL C ONSIDERATIONS

Original: Semi-supervised learning is arguably one of the most important deep learning applications, as in real life we often have access to an abundance of unlabeled examples, and only a few labeled datapoints.
Revised: Semi-supervised learning is arguably one of the most important deep learning applications, as in real life we often have access to an abundance of unlabeled examples, and only a few labeled datapoints.

Original: Our work highlights the importance of calibration in semi-supervised learning methods which could yield benefits for real-world applications in two main ways: 1) a better deep learning model that is more data efficient and 2) a model that is better calibrated and can quantify uncertainty better.
Revised: Our work highlights the importance of calibration in semi-supervised learning methods which could yield benefits for real-world applications in two main ways: 1) a better deep learning model that is more data efficient and 2) a model that is better calibrated and can quantify uncertainty better.

Original: The latter is highly important for crucial societal applications such as in healthcare.
Revised: The latter is highly important for crucial societal applications such as in healthcare.

Original: However, as with any deep learning application, there may be biases accumulated during dataset collection or assimilated during the training process.
Revised: However, as with any deep learning application, there may be biases accumulated during dataset collection or assimilated during the training process.

Original: This issue may be more acute in a semi-supervised setting where the small fraction of labels may highly misrepresent the ground truth data.
Revised: This issue may be more acute in a semi-supervised setting where the small fraction of labels may highly misrepresent the ground truth data.

Original: This may lead to unfair and unjust model predictions and give rise to ethical concerns when used for societal applications.
Revised: This may lead to unfair and unjust model predictions and give rise to ethical concerns when used for societal applications.