Original: R O PAWS: R OBUST S EMI - SUPERVISED R EPRESENTATION L EARNING FROM U NCURATED D ATA
Revised: R O PAWS: R OBUST S EMI - SUPERVISED R EPRESENTATION L EARNING FROM U NCURATED D ATA

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Semi-supervised learning aims to train a model using limited labels.
Revised: Semi-supervised learning aims to train a model using limited labels.

Original: State-of-theart semi-supervised methods for image classification such as PAWS rely on self - supervised representations learned with large-scale unlabeled but curated data.
Revised: State-of-theart semi-supervised methods for image classification such as PAWS rely on selfsupervised representations learned with large-scale unlabeled but curated data.

Original: However, PAWS is often less effective when using real-world unlabeled data that is uncurated, e.g., contains out-of-class data.
Revised: However, PAWS is often less effective when using real-world unlabeled data that is uncurated, e.g., contains out-of-class data.

Original: We propose RoPAWS, a robust extension of PAWS that can work with real-world unlabeled data.
Revised: We propose RoPAWS, a robust extension of PAWS that can work with real-world unlabeled data.

Original: We first reinterpret
Revised: We first reinterpret

Original: PAWS as a generative classifier that models densities using kernel density estima- tion
Revised: PAWS as a generative classifier that models densities using kernel density estimation.

Original: From this probabilistic perspective, we calibrate its prediction based on the densities of labeled and unlabeled data, which leads to a simple closed-form solution from the Bayes’ rule.
Revised: From this probabilistic perspective, we calibrate its prediction based on the densities of labeled and unlabeled data, which leads to a simple closed-form solution from the Bayes’ rule.

Original: We demonstrate that RoPAWS significantly improves
Revised: We demonstrate that RoPAWS significantly improves

Original: PAWS for uncurated Semi-iNat by +5.3% and curated ImageNet by +0.4%.
Revised: PAWS for uncurated Semi-iNat by +5.3% and curated ImageNet by +0.4%.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Semi-supervised learning aims to address the fundamental challenge of training models with limited labeled data by leveraging large-scale unlabeled data.
Revised: Semi-supervised learning aims to address the fundamental challenge of training models with limited labeled data by leveraging large-scale unlabeled data.

Original: Recent works exploit the success of selfsupervised learning (He et al., 2020; Chen et al., 2020a) in learning representations from unlabeled data for training large-scale semi-supervised models (Chen et al., 2020b; Cai et al., 2022).
Revised: Recent works exploit the success of selfsupervised learning (He et al., 2020; Chen et al., 2020a) in learning representations from unlabeled data for training large-scale semi-supervised models (Chen et al., 2020b; Cai et al., 2022).

Original: Instead of self-supervised pre-training followed by semi-supervised fine-tuning, PAWS (Assran et al., 2021) proposed a single-stage approach that combines supervised and self-supervised learning and achieves state-of-the-art accuracy and convergence speed.
Revised: Instead of self-supervised pre-training followed by semi-supervised fine-tuning, PAWS (Assran et al., 2021) proposed a single-stage approach that combines supervised and self-supervised learning and achieves state-of-the-art accuracy and convergence speed.

Original: While PAWS can leverage curated unlabeled data, we empirically show that it is not robust to realworld uncurated data, which often contains out-of-class data.
Revised: While PAWS can leverage curated unlabeled data, we empirically show that it is not robust to realworld uncurated data, which often contains out-of-class data.

Original: A common approach to tackle uncurated data in semi-supervised learning is to filter unlabeled data using out-of-distribution (OOD) classification (Chen et al., 2020d;
Revised: A common approach to tackle uncurated data in semi-supervised learning is to filter unlabeled data using out-of-distribution (OOD) classification (Chen et al., 2020d;

Original: Saito et al., 2021; Liu et al., 2022).
Revised: Saito et al., 2021; Liu et al., 2022).

Original: However, OOD filtering methods did not fully utilize OOD data, which could be beneficial to learn the representations especially on large-scale realistic datasets.
Revised: However, OOD filtering methods did not fully utilize OOD data, which could be beneficial to learn the representations especially on large-scale realistic datasets.

Original: Furthermore, filtering OOD data could be ineffective since in-class and out-of-class data are often hard to discriminate in practical scenarios.
Revised: Furthermore, filtering OOD data could be ineffective since in-class and out-of-class data are often hard to discriminate in practical scenarios.

Original: To this end, we propose RoPAWS, a robust semi-supervised learning method that can leverage uncurated unlabeled data.
Revised: To this end, we propose RoPAWS, a robust semi-supervised learning method that can leverage uncurated unlabeled data.

Original: PAWS predicts out-of-class data overconfidently in the known classes since it assigns the pseudo-label to nearby labeled data.
Revised: PAWS predicts out-of-class data overconfidently in the known classes since it assigns the pseudo-label to nearby labeled data.

Original: To handle this, RoPAWS regularizes the pseudolabels by measuring the similarities between labeled and unlabeled data.
Revised: To handle this, RoPAWS regularizes the pseudolabels by measuring the similarities between labeled and unlabeled data.

Original: These pseudo-labels are further calibrated by label propagation between unlabeled data.
Revised: These pseudo-labels are further calibrated by label propagation between unlabeled data.

Original: Figure 1 shows the conceptual illustration of RoPAWS and Figure 4 visualizes the learned representations.
Revised: Figure 1 shows the conceptual illustration of RoPAWS and Figure 4 visualizes the learned representations.

Original: More specifically, RoPAWS calibrates the prediction of PAWS from a probabilistic view.
Revised: More specifically, RoPAWS calibrates the prediction of PAWS from a probabilistic view.

Original: We first introduce a new interpretation of PAWS as a generative classifier, modeling densities over representation by kernel density estimation (KDE) (Rosenblatt, 1956).
Revised: We first introduce a new interpretation of PAWS as a generative classifier, modeling densities over representation by kernel density estimation (KDE) (Rosenblatt, 1956).

Original: The calibrated prediction is given by a closed-form solution from Bayes’ rule, which implicitly computes the fixed point of an iterative propagation formula of labels and priors of unlabeled data.
Revised: The calibrated prediction is given by a closed-form solution from Bayes’ rule, which implicitly computes the fixed point of an iterative propagation formula of labels and priors of unlabeled data.

Original: In addition, RoPAWS explicitly controls out-of-class data by modeling a prior distribution and computing a reweighted loss, making the model robust to uncurated data.
Revised: In addition, RoPAWS explicitly controls out-of-class data by modeling a prior distribution and computing a reweighted loss, making the model robust to uncurated data.

Original: Unlike OOD filtering methods, RoPAWS leverages all of the unlabeled (and labeled) data for representation learning.
Revised: Unlike OOD filtering methods, RoPAWS leverages all of the unlabeled (and labeled) data for representation learning.

Revised: [Figure]

Original: We demonstrate that RoPAWS outperforms PAWS and previous state-of-the-art (SOTA) robust semisupervised learning methods in various scenarios.
Revised: We demonstrate that RoPAWS outperforms PAWS and previous state-of-the-art (SOTA) robust semisupervised learning methods in various scenarios.

Original: RoPAWS improves PAWS in uncurated data with[Figure] a large margin.
Revised: RoPAWS improves PAWS in uncurated data with a large margin.

Original: In addition, PAWS improves PAWS in curated data by calibrating the prediction of uncertain data.
Revised: In addition, PAWS improves PAWS in curated data by calibrating the prediction of uncertain data.

Original: Figure 2 highlights our main results: RoPAWS improves (a) PAWS by +5.3% and previous SOTA by +1.7% on uncurated Semi-iNat (Su& Maji, 2021b) b) PAWS (previous
Revised: Figure 2 highlights our main results: RoPAWS improves (a) PAWS by +5.3% and previous SOTA by +1.7% on uncurated Semi-iNat (Su Maji, 2021b b) PAWS by

Original: ) by +
Revised: (Deng et al c

Original: 4% on
Revised: PAWS

Original: (Deng et al., 2009) using % of c d by et
Revised: (Krizhevsky et al., 2009) using 25 labels for each class using

Revised: CIFAR-100 and Tiny-ImageNet unlabeled

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Semi-supervised learning has been an active area of research for decades (Chapelle et al., 2006;
Revised: Semi-supervised learning has been an active area of research for decades (Chapelle et al., 2006;

Original: Van Engelen & Hoos, 2020).
Revised: Van Engelen & Hoos, 2020).

Original: Classic works directly regularize the classifier’s prediction, which we call the prediction-based approach.
Revised: Classic works directly regularize the classifier’s prediction, which we call the prediction-based approach.

Original: Recent works focus more on the representation that stems from the final classifier, which we call the representation-based approach.
Revised: Recent works focus more on the representation that stems from the final classifier, which we call the representation-based approach.

Original: Leveraging the recent progress of self-supervised learning (He et al., 2020), the representation-based approach has more promise in large-scale scenarios and can be more robust under uncurated data.
Revised: Leveraging the recent progress of self-supervised learning (He et al., 2020), the representation-based approach has more promise in large-scale scenarios and can be more robust under uncurated data.

Original: In the following subsections, we will briefly review the general idea of each approach and how they handle uncurated data.
Revised: In the following subsections, we will briefly review the general idea of each approach and how they handle uncurated data.

Original:  P REDICTION - BASED SEMI - SUPERVISED LEARNING
Revised: 2. P REDICTION - BASED SEMI - SUPERVISED LEARNING

Original: General approach.
Revised: General approach.

Original: Prediction-based approaches regularize the classifier’s prediction of unlabeled data.
Revised: Prediction-based approaches regularize the classifier’s prediction of unlabeled data.

Original: Two objectives are popularly used: pseudo-labeling and consistency regularization.
Revised: Two objectives are popularly used: pseudo-labeling and consistency regularization.

Original: Pseudolabeling (Lee et al., 2013; Yalniz et al., 2019; Xie et al., 2020b; Rizve et al., 2021; Cascante-Bonilla et al., 2021;
Revised: Pseudolabeling (Lee et al., 2013; Yalniz et al., 2019; Xie et al., 2020b; Rizve et al., 2021; Cascante-Bonilla et al., 2021;

Original: Hu et al., 2021) predicts labels of unlabeled data and retrains the classifier using the predicted labels, minimizing the prediction entropy of unlabeled data (Grandvalet & Bengio, 2004).
Revised: Hu et al., 2021) predicts labels of unlabeled data and retrains the classifier using the predicted labels, minimizing the prediction entropy of unlabeled data (Grandvalet & Bengio, 2004).

Original: Consistency regularization (Sajjadi et al., 2016; Laine & Aila, 2017; Tarvainen & Valpola, 2017;
Revised: Consistency regularization (Sajjadi et al., 2016; Laine & Aila, 2017; Tarvainen & Valpola, 2017;

Original: Miyato et al., 2018; Xie et al., 2020a) enforces the prediction that two views of the same image are similar.
Revised: Miyato et al., 2018; Xie et al., 2020a) enforces the prediction that two views of the same image are similar.

Original: Combining two objectives, prediction-based approaches has shown notable results (Berthelot et al., 2019; 2020;
Revised: Combining two objectives, prediction-based approaches has shown notable results (Berthelot et al., 2019; 2020;

Original: Sohn et al., 2020; Kuo et al., 2020; Li et al., 2021).
Revised: Sohn et al., 2020; Kuo et al., 2020; Li et al., 2021).

Original: However, they underperform than representation-based approaches for large-scale scenarios (Chen et al., 2020b).
Revised: However, they underperform than representation-based approaches for large-scale scenarios (Chen et al., 2020b).

Original: Moreover, most prior works assume that unlabeled data are curated, i.e., follow the same distribution of labeled data, and often fail when unlabeled data are uncurated (Oliver et al., 2018;
Revised: Moreover, most prior works assume that unlabeled data are curated, i.e., follow the same distribution of labeled data, and often fail when unlabeled data are uncurated (Oliver et al., 2018;

Original: Su et al., 2021).
Revised: Su et al., 2021).

Original: Handling uncurated data.
Revised: Handling uncurated data.

Original: Numerous works have attempted to make the prediction-based approach robust to uncurated data.
Revised: Numerous works have attempted to make the prediction-based approach robust to uncurated data.

Original: Most prior works assume the unlabeled data are composed of in- and outof-domain (OOD) data and filter OOD data by training an OOD classifier (Chen et al., 2020d;
Revised: Most prior works assume the unlabeled data are composed of in- and outof-domain (OOD) data and filter OOD data by training an OOD classifier (Chen et al., 2020d;

Original: Guo et al., 2020; Yu et al., 2020; Huang et al., 2021a;b; Saito et al., 2021; Killamsetty et al., 2021; Nair et al., 2019; Augustin & Hein, 2020; Park et al., 2021).
Revised: Guo et al., 2020; Yu et al., 2020; Huang et al., 2021a;b; Saito et al., 2021; Killamsetty et al., 2021; Nair et al., 2019; Augustin & Hein, 2020; Park et al., 2021).

Original: From the perspective of the prediction-based approach, it is natural to filter OOD (particularly out-of-class) data since they are irrelevant to inclass prediction.
Revised: From the perspective of the prediction-based approach, it is natural to filter OOD (particularly out-of-class) data since they are irrelevant to in-

Original: However, the filtering approach has two limitations: (a) it ignores the representation

Original: PAWS
Revised: PAWS

Original: CCSSL
Revised: CCSSL

Original: RoPAWS 37.
Revised: RoPAWS 37.

Original: +5.
Revised: +5.

Original: +1.
Revised: +1.

Original: (a) Semi-iNat (uncurated)
Revised: (a) Semi-iNat (uncurated)

Original: PAWS
Revised: PAWS

Original: RoPAWS 66.
Revised: RoPAWS 66.

Original: +0.
Revised: +0.

Original: (b) ImageNet
Revised: (b) ImageNet

Original: PAWS
Revised: PAWS

Original: RoPAWS 78.
Revised: RoPAWS 78.

Original: +6.
Revised: +6.

Original: (c) CF-10 (+ CF-100)
Revised: (c) CF-10 (+ CF-100)

Original: PAWS
Revised: PAWS

Original: OpenCos RoPAWS 60.
Revised: OpenCos RoPAWS 60.

Original: +22.
Revised: +22.

Original: +0.
Revised: +0.

Original: [Figure]
Revised: [Figure]

Original:  learned from OOD data, and (b) the detection of OOD can be artificial.
Revised: class prediction. However, the filtering approach has two limitations: (a) it ignores the representation learned from OOD data, and (b) the detection of OOD can be artificial.

Original: Note that the model cannot distinguish near OOD and in-class data not covered by labels.
Revised: Note that the model cannot distinguish near OOD and in-class data not covered by labels.

Original: Therefore, we do not filter the OOD samples but (a) learn representation from them and (b) consider soft confidence of being in-domain.
Revised: Therefore, we do not filter the OOD samples but (a) learn representation from them and (b) consider soft confidence of being in-domain.

Original: Some recent works reveal that utilizing OOD data can be better than filtering them (Luo et al., 2021;
Revised: Some recent works reveal that utilizing OOD data can be better than filtering them (Luo et al., 2021;

Original: Han et al., 2021; Huang et al., 2022).
Revised: Han et al., 2021; Huang et al., 2022).

Original: RoPAWS is also in this line of work; however, leveraging
Revised: RoPAWS is also in this line of work; however, leveraging

Original: OOD data is more natural from the representation learning perspective.
Revised: OOD data is more natural from the representation learning perspective.

Original: 2. R EPRESENTATION - BASED SEMI - SUPERVISED LEARNING
Revised: 2. R EPRESENTATION - BASED SEMI - SUPERVISED LEARNING

Original: General approach.
Revised: General approach.

Original: Self-supervised learning has shown remarkable success recently, learning a universal representation that can be transferred to various downstream tasks.
Revised: Self-supervised learning has shown remarkable success recently, learning a universal representation that can be transferred to various downstream tasks.

Original: Specifically, the siamese network approach, which makes the representation of two views from the same image invariant, has dramatically improved image classification (He et al., 2020; Chen et al., 2020a).
Revised: Specifically, the siamese network approach, which makes the representation of two views from the same image invariant, has dramatically improved image classification (He et al., 2020; Chen et al., 2020a).

Original: Recent works reveal that fine-tuning (or training a lightweight classifier on top of) the self-supervised model performs well for semi-supervised learning, especially for high-resolution and large-scale datasets (Zhai et al., 2019; Rebuffi et al., 2020;
Revised: Recent works reveal that fine-tuning (or training a lightweight classifier on top of) the self-supervised model performs well for semi-supervised learning, especially for high-resolution and large-scale datasets (Zhai et al., 2019; Rebuffi et al., 2020;

Original: Chen et al., 2020b; Cai et al., 2022).
Revised: Chen et al., 2020b; Cai et al., 2022).

Original: However, self-supervised pretraining is inefficient when the target downstream task is given; one can directly train a task-specific model.
Revised: However, self-supervised pretraining is inefficient when the target downstream task is given; one can directly train a task-specific model.

Original: From this motivation, PAWS (Assran et al., 2021) proposed a semi-supervised extension of the siamese network approach, enforcing the prediction (computed from representation) of two views be invariant.
Revised: From this motivation, PAWS (Assran et al., 2021) proposed a semi-supervised extension of the siamese network approach, enforcing the prediction (computed from representation) of two views be invariant.

Original: Notably, PAWS has shown to be more effective (i.e., performs better) yet efficient (i.e., converges faster) than the two-stage approach of self-supervised pre-training and then fine-tuning.
Revised: Notably, PAWS has shown to be more effective (i.e., performs better) yet efficient (i.e., converges faster) than the two-stage approach of self-supervised pre-training and then fine-tuning.

Original: However, PAWS is often not robust to uncurated data, limiting its applicability to realworld scenarios.
Revised: However, PAWS is often not robust to uncurated data, limiting its applicability to realworld scenarios.

Original: Thus, we propose RoPAWS, a robust extension of PAWS for uncurated data.
Revised: Thus, we propose RoPAWS, a robust extension of PAWS for uncurated data.

Original: Handling uncurated data.
Revised: Handling uncurated data.

Original: Self-supervised learning is robust to uncurated data, learning a generic representation of data, and does not suffer from the issues of labels such as out-of-class, long-tail, or noisy labels (Kang et al., 2020; Tian et al., 2021; Goyal et al., 2021; 2022).
Revised: Self-supervised learning is robust to uncurated data, learning a generic representation of data, and does not suffer from the issues of labels such as out-of-class, long-tail, or noisy labels (Kang et al., 2020; Tian et al., 2021; Goyal et al., 2021; 2022).

Original: Thus, fine-tuning a selfsupervised model is a strong baseline for robust semi-supervised learning.
Revised: Thus, fine-tuning a selfsupervised model is a strong baseline for robust semi-supervised learning.

Original: However, we claim that semi-supervised learning can maintain its efficacy (and efficiency) while enjoying the robustness of self-supervised learning by designing a robust semi-supervised method called RoPAWS.
Revised: However, we claim that semi-supervised learning can maintain its efficacy (and efficiency) while enjoying the robustness of self-supervised learning by designing a robust semi-supervised method called RoPAWS.

Original: On the other hand, some works combine self-supervised and supervised learning, where the supervised loss is only applied to the labeled (or confident) data (Yang et al., 2022; Khosla et al., 2020).
Revised: On the other hand, some works combine self-supervised and supervised learning, where the supervised loss is only applied to the labeled (or confident) data (Yang et al., 2022; Khosla et al., 2020).

Original: It is safe but too conservative as it does not propagate labels over unlabeled data.
Revised: It is safe but too conservative as it does not propagate labels over unlabeled data.

Original: This approach is effective for a related problem called open-world semi-supervised learning, which aims to cluster novel classes while predicting known classes (Cao et al., 2022; Vaze et al., 2022; Rizve et al., 2022; Sun & Li, 2022).
Revised: This approach is effective for a related problem called open-world semi-supervised learning, which aims to cluster novel classes while predicting known classes (Cao et al., 2022; Vaze et al., 2022; Rizve et al., 2022; Sun & Li, 2022).

Original: In contrast, we focus on predicting the known classes, i.e., no extra burden for novel classes.
Revised: In contrast, we focus on predicting the known classes, i.e., no extra burden for novel classes.

Original: Extending our work to open-world setups would be an interesting future direction.
Revised: Extending our work to open-world setups would be an interesting future direction.

Original:  R O PAWS: R OBUST PAWS FOR UNCURATED DATA
Revised:  R O PAWS: R OBUST PAWS FOR UNCURATED DATA

Original: We propose RoPAWS, a robust semi-supervised learning method for uncurated data.
Revised: We propose RoPAWS, a robust semi-supervised learning method for uncurated data.

Original: In the following subsections, we first explain the problem of PAWS (Assran et al., 2021) on uncurated data, then introduce a novel probabilistic interpretation of PAWS.
Revised: In the following subsections, we first explain the problem of PAWS (Assran et al., 2021) on uncurated data, then introduce a novel probabilistic interpretation of PAWS.

Original: Finally, we describe our proposed RoPAWS, which calibrates the prediction of targets from the probabilistic view for robust training.
Revised: Finally, we describe our proposed RoPAWS, which calibrates the prediction of targets from the probabilistic view for robust training.

Original: Figurevisualizes the overall framework of RoPAWS.
Revised: Figurevisualizes the overall framework of RoPAWS.

Original: Detailed derivations are provided in Appendix A. 𝑓 !
Revised: Detailed derivations are provided in Appendix A. 𝑓 !

Original: 𝑥 ",$% 𝑥 ",&% 𝑧 ’( 𝑧 ",$% 𝑧 ",&% view(output) view(target)
Revised: 𝑥 ", [Figure] % 𝑥 ",&% 𝑧 ’( 𝑧 ",$% 𝑧 ",&% view(output) view(target)

Original: labeleddata 𝑝(𝑦|𝑥 ",$% ) 𝑝(𝑦|𝑥 ",&% )
Revised: labeleddata 𝑝(𝑦|𝑥 ",$% ) 𝑝(𝑦|𝑥 ",&% )

Original: CE loss = 𝑦 ’( 𝑥 ’( 𝑦 ’( { !"
Revised: CE loss = 𝑦 ’( 𝑥 ’( 𝑦 ’( { !"

Original: #$ 𝑥 ",&) 𝑧 ",&) unlabeleddata{
Revised: #$ [Figure] [Figure] unlabeleddata

Original: [Figure]
Revised: [Figure]

Original: Same as PAWS
Revised: Same as PAWS

Original: New in RoPAWS } }
Revised: New in RoPAWS } }

Original: 3. P ROBLEM SETUP AND PRELIMINARIES
Revised: 3. P ROBLEM SETUP AND PRELIMINARIES

Original: Problem setup.
Revised: Problem setup.

Original: Semi-supervised learning aims to learn a classifier from a large unlabeled dataset
Revised: Semi-supervised learning aims to learn a classifier from a large unlabeled dataset

Original: D u := { x u } and a small labeled dataset D l := { x l , y l } for { x l } ⊂ D u .
Revised: D u := { x u } and a small labeled dataset D l := { x l , y l } for { x l } ⊂ D u .

Original: Most prior works assume that D l and D u are drawn from the same distribution.
Revised: Most prior works assume that D l and D u are drawn from the same distribution.

Original: However, the unlabeled data are often uncurated , i.e., containing uncertain (far from labels, often distribution-shifted) or out-of-class data.
Revised: However, the unlabeled data are often uncurated , i.e., containing uncertain (far from labels, often distribution-shifted) or out-of-class data.

Original: We aim to extend the applicability of semi-supervised learning for these uncurated scenarios.
Revised: We aim to extend the applicability of semi-supervised learning for these uncurated scenarios.

Original: Preliminaries.
Revised: Preliminaries.

Original: In this paper, we focus on semi-supervised for image classification.
Revised: In this paper, we focus on semi-supervised for image classification.

Original: We start with introducing PAWS (Assran et al., 2021), which is the state-of-the-art method.
Revised: We start with introducing PAWS (Assran et al., 2021), which is the state-of-the-art method.

Original: PAWS leverages the power of self-supervised learning, particularly the siamese network approach (Caron et al., 2020), to learn useful representation from unlabeled data.
Revised: PAWS leverages the power of self-supervised learning, particularly the siamese network approach (Caron et al., 2020), to learn useful representation from unlabeled data.

Original: Specifically, PAWS enforces the predictions of two views from the same image to be invariant.
Revised: Specifically, PAWS enforces the predictions of two views from the same image to be invariant.

Original: Unlike self-supervised learning, PAWS defines the prediction using labeled samples.
Revised: Unlike self-supervised learning, PAWS defines the prediction using labeled samples.

Original: Formally, let B l = { x il , y il } Ni =1 and B u
Revised: Formally, let B l = { x il , y il } Ni =1 and B u

Original: = { x iu } Mi =1 be the labeled and unlabeled batches of size N and M , sampled from D l and D u , respectively.
Revised: = { x iu } Mi =1 be the labeled and unlabeled batches of size N and M , sampled from D l and D u , respectively.

Original: Here, the labeled batch B l is class-balanced, i.e., |B yl | =
Revised: Here, the labeled batch B l is class-balanced, i.e., |B yl | =

Original: NC for B yl := { ( x il , y il )
Revised: NC for B yl := { ( x il , y il )

Original: ∈ B l | y il = y } for all the classes y ∈
Revised: ∈ B l | y il = y } for all the classes y ∈

Original: [ C ] := { 1 , .., C } .
Revised: [ C ] := { 1 , .., C } .

Original: PAWS samples a subset of classes for each mini-batch.
Revised: PAWS samples a subset of classes for each mini-batch.

Original: Given the batches, PAWS first computes normalized embedding z = f θ ( x ) /
Revised: Given the batches, PAWS first computes normalized embedding z = f θ ( x ) /

Original: ∥ f θ ( x ) ∥ , i.e.,
Revised: ∥ f θ ( x ) ∥ , i.e.,

Original: ∥ z ∥ =using an encoder f θ ( · ) .
Revised: ∥ z ∥ =using an encoder f θ ( · ) .

Original: For simplicity, we omit the encoder f θ ( · ) in the future equations and denote the normalized embedding by substituting x with z from the original data x , i.e., f θ ( x supersub ) =
Revised: For simplicity, we omit the encoder f θ ( · ) in the future equations and denote the normalized embedding by substituting x with z from the original data x , i.e., f θ ( x supersub ) =

Original: z supersub .
Revised: z supersub .

Original: We assume that an embedding z is a 1 × d vector, and z l and z u are N × d and M ×
Revised: We assume that an embedding z is a 1 × d vector, and z l and z u are N × d and M ×

Original: d matrices represent the embeddings of labeled and unlabeled data.
Revised: d matrices represent the embeddings of labeled and unlabeled data.

Original: Here, PAWS predicts the label p ( y | x ) of data x by finding the nearest labeled sample x l and softly assigning the label y l .
Revised: Here, PAWS predicts the label p ( y | x ) of data x by finding the nearest labeled sample x l and softly assigning the label y l .

Original: Note that we compute the similarity on the representation space and p ( y | x ) is a function of z .
Revised: Note that we compute the similarity on the representation space and p ( y | x ) is a function of z .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: where p y | x l ∈ R N × C are the corresponding one-hot (or soft) labels with a row-wise sum of 1 , and σ τ ( · ) is the softmax function with temperature τ .
Revised: where p y | x l ∈ R N × C are the corresponding one-hot (or soft) labels with a row-wise sum of 1 , and σ τ ( · ) is the softmax function with temperature τ .

Original: PAWS then enforces the prediction of two views p i 1 :
Revised: PAWS then enforces the prediction of two views p i 1 :

Original: = p ( y | t 1 ( x i )) and p i 2 := p ( y | t 2 ( x i )) from the same unlabeled data x i with augmentations t 1 , t 2 ∼ T to be identical.
Revised: = p ( y | t 1 ( x i )) and p i 2 := p ( y | t 2 ( x i )) from the same unlabeled data x i with augmentations t 1 , t 2 ∼ T to be identical.

Original: Additionally, the sharpening function ρ ( · ) is applied to targets which are detached from backpropagation.
Revised: Additionally, the sharpening function ρ ( · ) is applied to targets which are detached from backpropagation.

Original: PAWS further regularizes the average sharpened prediction ¯ p :=
Revised: PAWS further regularizes the average sharpened prediction ¯ p :=

Original: 12 M (cid:80) i ( ρ ( p i 1 )
Revised: 12 M (cid:80) i ( ρ ( p i 1 )

Original: + ρ ( p i 2 )) to be uniform.
Revised: + ρ ( p i 2 )) to be uniform.

Original: To sum up, the training objective of PAWS is:
Revised: To sum up, the training objective of PAWS is:

Original: [Equation]
Revised: [Equation]

Original: where H ( · , · ) is cross-entropy and and H ( · ) is entropy.
Revised: where H ( · , · ) is cross-entropy and and H ( · ) is entropy.

Original: Recall that the prediction is computed from the labeled data, and thus the objective is a function of both B l and B u .
Revised: Recall that the prediction is computed from the labeled data, and thus the objective is a function of both B l and B u .

Original: Problem with PAWS.
Revised: Problem with PAWS.

Original: The prediction of PAWS is overconfident to given [ C ] classes since it assigns the label of the nearby labeled data.
Revised: The prediction of PAWS is overconfident to given [ C ] classes since it assigns the label of the nearby labeled data.

Original: However, it can be problematic for uncurated data containing out-of-class data.
Revised: However, it can be problematic for uncurated data containing out-of-class data.

Original: Specifically, PAWS equally assigns high confidence for both in- and out-of-class data, unlike RoPAWS assigns high confidence for in-class and low confidence for out-of-class data (see Appendix G).
Revised: Specifically, PAWS equally assigns high confidence for both in- and out-of-class data, unlike RoPAWS assigns high confidence for in-class and low confidence for out-of-class data (see Appendix J).

Original: Thus, we aim to calibrate the prediction to make PAWS robust.
Revised: Thus, we aim to calibrate the prediction to make PAWS robust.

Original: 3. P ROBABILISTIC VIEW OF PAWS
Revised: 3. P ROBABILISTIC VIEW OF PAWS

Original: Our key observation is that PAWS can be interpreted as a generative classifier, which estimates the densities over the representation space.
Revised: Our key observation is that PAWS can be interpreted as a generative classifier, which estimates the densities over the representation space.

Original: To this end, we consider the kernel density estimation (KDE) (Rosenblatt, 1956) using the kernel as the similarity on the representation space:
Revised: To this end, we consider the kernel density estimation (KDE) (Rosenblatt, 1956) using the kernel as the similarity on the representation space:

Original: [Equation]
Revised: [Equation]

Original: where Z is a normalizing constant to make 1 Z (cid:82)
Revised: where Z is a normalizing constant to make 1 Z (cid:82)

Original: z ′ k
Revised: z ′ k

Original: ( z, z ′ )
Revised: ( z, z ′ )

Original: = 1 .
Revised: = 1 .

Original: From the definition, we can estimate the conditional p ( x | y ) and marginal p ( x ) densities using the labeled batch of each class B yl and the entire labeled batch B l , respectively.
Revised: From the definition, we can estimate the conditional p ( x | y ) and marginal p ( x ) densities using the labeled batch of each class B yl and the entire labeled batch B l , respectively.

Original: Formally, the densities of PAWS are:
Revised: Formally, the densities of PAWS are:

Original: [Equation]
Revised: [Equation]

Original:  Applying them to the Bayes’ rule, we can compute the prediction p ( y | x ) :
Revised: p [Equation] Applying them to the Bayes’ rule, we can compute the prediction p ( y | x ) :

Original: [Equation]
Revised: [Equation]

Original: where p yl are one-hot labels.
Revised: where p yl are one-hot labels.

Original: Remark that it recovers the original prediction of PAWS in Eq.
Revised: Remark that it recovers the original prediction of PAWS in Eq.

Original: We further extend the densities to consider soft labels, e.g., PAWS applies label smoothing (M ¨ uller et al., 2019), using the weighted kernel k y[Equation] z [Equation]
Revised: We further extend the densities to consider soft labels, e.g., PAWS applies label smoothing (M ¨ uller et al., 2019), using the weighted kernel k y

Original: z

Original: [Equation]

Original: [Equation]

Original: K p

Original: [Equation]

Original: y

Original: [Equation]

Original: x

Original: [Equation]
Revised: [Equation]

Original: k

Original: [Equation]

Original: z [Equation] [Equation] assuming that the normalizing constant K is identical for all classes (class-balanced).
Revised: assuming that the normalizing constant K is identical for all classes (class-balanced).

Original: Then, this definition also recovers the original prediction of PAWS considering soft labels:
Revised: Then, this definition also recovers the original prediction of PAWS considering soft labels:

Original: [Equation]
Revised: [Equation]

Original: To summarize, PAWS is a generative classifier, and one can estimate the density of a sample on the representation space learned by PAWS.
Revised: To summarize, PAWS is a generative classifier, and one can estimate the density of a sample on the representation space learned by PAWS.

Original: This probabilistic view gives us a principled way to extend
Revised: This probabilistic view gives us a principled way to extend

Original: PAWS for various scenarios, e.g., robust training on uncurated data, as stated below.
Revised: PAWS for various scenarios, e.g., robust training on uncurated data, as stated below.

Original: 3. R OBUST PAWS VIA CALIBRATED PREDICTION
Revised: 3. R OBUST PAWS VIA CALIBRATED PREDICTION

Original: Recall that the prediction of PAWS is overconfident to the known classes [ C ] since it propagates labels from nearby labeled data.
Revised: Recall that the prediction of PAWS is overconfident to the known classes [ C ] since it propagates labels from nearby labeled data.

Original: Assuming that we also know the labels of unlabeled data, which have less confident values for uncertain (or out-of-class) data, we can calibrate the prediction by considering both labels of nearby labeled and unlabeled samples.
Revised: Assuming that we also know the labels of unlabeled data, which have less confident values for uncertain (or out-of-class) data, we can calibrate the prediction by considering both labels of nearby labeled and unlabeled samples.

Original: However, we do not know the labels of unlabeled data, and using them for prediction raises a chicken-and-egg problem.
Revised: However, we do not know the labels of unlabeled data, and using them for prediction raises a chicken-and-egg problem.

Original: Interestingly, the probabilistic view gives the solution: from the Bayes’ rule, the prediction p u ( y | x ) of unlabeled data is a function of densities p u ( x | y ) and p u ( x ) , which are again a function of labels p l ( y | x ) of labeled data and predictions p u ( y | x ) of unlabeled data.
Revised: Interestingly, the probabilistic view gives the solution: from the Bayes’ rule, the prediction p u ( y | x ) of unlabeled data is a function of densities p u ( x | y ) and p u ( x ) , which are again a function of labels p l ( y | x ) of labeled data and predictions p u ( y | x ) of unlabeled data.

Original: Solving the equation, we can get a closed-form solution of p u ( y | x ) as a function of labels p l ( y | x ) and distances of embeddings z l , z u .
Revised: Solving the equation, we can get a closed-form solution of p u ( y | x ) as a function of labels p l ( y | x ) and distances of embeddings z l , z u .

Original: To this end, we first extend the densities in Eq.
Revised: To this end, we first extend the densities in Eq.

Original: (6) to consider both labeled and unlabeled data:p
Revised: (6) to consider both labeled and unlabeled data:p

Original: [Equation]
Revised: [Equation]

Original: Recall that the labeled is

Original: -balanced, hence

Original: B

Original: = N ,

Original: B yl | =

Original: NC , and p y )

Original: = for all

Original: y We balance the scale of labeled and unlabeled data with a ratio r : 1 when the unlabeled batch is large.
Revised: We balance the scale of labeled and unlabeled data with a ratio r : 1 when the unlabeled batch is large.

Original: where K ′ =
Revised: where K ′ =

Original: ( N + M ) K is the normalizing constant.
Revised: ( N + M ) K is the normalizing constant.

Original: Applying them to the Bayes’ rule:
Revised: Applying them to the Bayes’ rule:

Original: [Equation]
Revised: [Equation]

Original: where [ s l | s u ] := σ τ ( z · [ z ⊺ l | z ⊺ u ]) ∈ R 1 × ( N + M ) are similarities of a sample x to labeled and unlabeled data.
Revised: where [ s l | s u ] := σ τ ( z · [ z ⊺ l | z ⊺ u ]) ∈ R 1 × ( N + M ) are similarities of a sample x to labeled and unlabeled data.

Original: Aggregating the predictions of unlabeled data, we can formulate the equation in a matrix form:
Revised: Aggregating the predictions of unlabeled data, we can formulate the equation in a matrix form:

Original: [Equation]
Revised: [Equation]

Original: where s l , s u are matrix versions of s l , s u , p yu is a matrix version of p ( y ) , and ⊙ is an element-wise product.
Revised: where s l , s u are matrix versions of s l , s u , p yu is a matrix version of p ( y ) , and ⊙ is an element-wise product.

Original: By organizing the formula, we get the prediction q ( y | x ) of RoPAWS:
Revised: By organizing the formula, we get the prediction q ( y | x ) of RoPAWS:

Original: [Equation]
Revised: [Equation]

Original: This formula can also be interpreted as label propagation (Zhou et al., 2003; Bengio et al., 2006;
Revised: This formula can also be interpreted as label propagation (Zhou et al., 2003; Bengio et al., 2006;

Original: Iscen et al., 2019) of labels p y
Revised: Iscen et al., 2019) of labels p y

Original: | x l and priors p yu of unlabeled data to get the posterior p y | x u .
Revised: | x l and priors p yu of unlabeled data to get the posterior p y | x u .

Original: In-domain prior.
Revised: In-domain prior.

Original: Considering both nearby labeled and unlabeled samples gives a better calibrated prediction but has a threat of amplifying the wrong prediction of unlabeled data.
Revised: Considering both nearby labeled and unlabeled samples gives a better calibrated prediction but has a threat of amplifying the wrong prediction of unlabeled data.

Original: Indeed, propagating the labels from (certain) labels and (uncertain) predictions of unlabeled data equally can be too risky.
Revised: Indeed, propagating the labels from (certain) labels and (uncertain) predictions of unlabeled data equally can be too risky.

Original: Thus, we further improve the prediction formula by putting more weights on certain samples.
Revised: Thus, we further improve the prediction formula by putting more weights on certain samples.

Original: This modification can be naturally done under the proposed framework by controlling the prior p ( y ) to consider out-of-class data.
Revised: This modification can be naturally done under the proposed framework by controlling the prior p ( y ) to consider out-of-class data.

Original: Specifically, we give a data-dependent prior of being in-domain (i.e., close to labeled data) for unlabeled data.
Revised: Specifically, we give a data-dependent prior of being in-domain (i.e., close to labeled data) for unlabeled data.

Original: Let p in u ∈ R M × 1 be the in-domain prior for each sample.
Revised: Let p in u ∈ R M × 1 be the in-domain prior for each sample.

Original: Then, we set the prior belief p ( y ) of in-class as p in u /C
Revised: Then, we set the prior belief p ( y ) of in-class as p in u /C

Original: (class-balanced).
Revised: (class-balanced).

Original: Here, we only adopt the prior belief of unlabeled data, and the densities p ( x | y ) and p ( x ) remains the same as in Eq.
Revised: Here, we only adopt the prior belief of unlabeled data, and the densities p ( x | y ) and p ( x ) remains the same as in Eq.

Original: Thus, applying the adopted belief of the prior gives the regularized prediction formula:
Revised: Thus, applying the adopted belief of the prior gives the regularized prediction formula:

Original: [Equation]
Revised: [Equation]

Original: The final remaining step is to estimate the in-domain prior p in u .
Revised: The final remaining step is to estimate the in-domain prior p in u .

Original: Here, we assume that a data is likely to be in-domain if it is close to some labeled data.
Revised: Here, we assume that a data is likely to be in-domain if it is close to some labeled data.

Original: To this end, we measure the similarity between data x and labeled data { x il } using the kernel defined in Eq.
Revised: To this end, we measure the similarity between data x and labeled data { x il } using the kernel defined in Eq.

Original: However, we normalize the kernel to make the similarity between x il and x il itself to be 1 (i.e., in-domain prior of x il is 1) and use the different temperature τ prior which controls the overall confidence of in-domain prior; use lower τ prior for when data are more uncertain.
Revised: However, we normalize the kernel to make the similarity between x il and x il itself to be 1 (i.e., in-domain prior of x il is 1) and use the different temperature τ prior which controls the overall confidence of in-domain prior; use lower τ prior for when data are more uncertain.

Original: Putting it all together, the in-domain prior is given by:
Revised: Putting it all together, the in-domain prior is given by:

Original: [Equation]
Revised: [Equation]

Original: where p in u is a matrix version of p ( in | x ) .
Revised: where p in u is a matrix version of p ( in | x ) .

Original: Note that p ( in | x il ) = exp(0) = 1 .
Revised: Note that p ( in | x il ) = exp(0) = 1 .

Original: We remark that setting p ( in | x ) < 1 implicitly assigns the remaining probability to the background class.
Revised: We remark that setting p ( in | x ) < 1 implicitly assigns the remaining probability to the background class.

Original: However, unlike the background class enforcing the uncertain samples to be collapsed into a few regions (Dhamija et al., 2018), RoPAWS implicitly considers them and all computations are done in the label space of [ C ] .
Revised: However, unlike the background class enforcing the uncertain samples to be collapsed into a few regions (Dhamija et al., 2018), RoPAWS implicitly considers them and all computations are done in the label space of [ C ] .

Original: Here, the prediction q ( y | x ) ∈ R C from Eq.
Revised: Here, the prediction q ( y | x ) ∈ R C from Eq.

Original: (12) has the sum < 1 , where this sum gives a posterior probability of being in-domain q ( in | x ) .
Revised: (12) has the sum < 1 , where this sum gives a posterior probability of being in-domain q ( in | x ) .

Original: We convert this prediction q ( y | x ) to the normalized C -class probability by adding a uniform multiplied by 1 − q ( in | x ) , i.e., uncertain samples are close to uniform (Lee et al., 2018a), gives the final prediction values.
Revised: We convert this prediction q ( y | x ) to the normalized C -class probability by adding a uniform multiplied by 1 − q ( in | x ) , i.e., uncertain samples are close to uniform (Lee et al., 2018a), gives the final prediction values.

Original: Training objective.
Revised: Training objective.

Original: Recall that the training objective of PAWS is to enforce the prediction of view(output) to be identical to the prediction of view 2 (target).
Revised: Recall that the training objective of PAWS is to enforce the prediction of view(output) to be identical to the prediction of view 2 (target).

Original: Here, we apply the calibrated prediction in Eq.
Revised: Here, we apply the calibrated prediction in Eq.

Original: (12) only to the target (denoted as q i ) and keep the original prediction in Eq.
Revised: (12) only to the target (denoted as q i ) and keep the original prediction in Eq.

Original: (1) for the output (denoted as p i ).
Revised: (1) for the output (denoted as p i ).

Original: We use the same inference (soft nearest-neighbor or fine-tuning) with PAWS, meaning that we keep the output the same but only calibrate the target.
Revised: We use the same inference (soft nearest-neighbor or fine-tuning) with PAWS, meaning that we keep the output the same but only calibrate the target.

Original: RoPAWS regularizes the prediction of uncertain (or out-of-class) data to be uniform, which may make the prediction too unconfident.
Revised: RoPAWS regularizes the prediction of uncertain (or out-of-class) data to be uniform, which may make the prediction too unconfident.

Original: To avoid this, we give less weight to the objectives of uncertain
Revised: To avoid this, we give less weight to the objectives of uncertain

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: samples (Ren et al., 2020).
Revised: samples (Ren et al., 2020).

Original: We simply set the weight w i as the average in-domain posterior of two views w i :=
Revised: We simply set the weight w i as the average in-domain posterior of two views w i :=

Original: ( 12 ( q ( y | x i 1 ) + q ( y | x i 2 )))
Revised: ( 12 ( q ( y | x i 1 ) + q ( y | x i 2 )))

Original: k , where k controls the power of reweighting.
Revised: k , where k controls the power of reweighting.

Original: To sum up, the objective of RoPAWS is (differences from PAWS in Eq.
Revised: To sum up, the objective of RoPAWS is (differences from PAWS in Eq.

Original: (2) are highlighted in red):
Revised: (2) are highlighted in red):

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Setup.
Revised: Setup.

Original: We follow the default setup of PAWS: ImageNet (Deng et al., 2009) setup on ResNet50 (He et al., 2016) for large-scale experiments, and CIFAR-10 (Krizhevsky et al., 2009) setup on
Revised: We follow the default setup of PAWS: ImageNet (Deng et al., 2009) setup on ResNet50 (He et al., 2016) for large-scale experiments, and CIFAR-10 (Krizhevsky et al., 2009) setup on

Original: WRN-28-2 (Zagoruyko & Komodakis, 2016) for small-scale experiments.
Revised: WRN-28-2 (Zagoruyko & Komodakis, 2016) for small-scale experiments.

Original: RoPAWS has three hyperparameters: the scale ratio of labeled and unlabeled batch r , the temperature for in-domain prior τ prior , and the power of reweighted loss k .
Revised: RoPAWS has three hyperparameters: the scale ratio of labeled and unlabeled batch r , the temperature for in-domain prior τ prior , and the power of reweighted loss k .

Original: We set r = 5 for all experiments, and set τ prior = 3 . 0 for
Revised: We set r = 5 for all experiments, and set τ prior = 3 . 0 for

Original: ResNet-50 and τ prior = 0 .
Revised: ResNet-50 and τ prior = 0 .

Original: 1 for WRN-28-2.
Revised: 1 for WRN-28-2.

Original: We set k = 1 for all experiments except k = 3 for finetuning from ImageNet pre-trained models.
Revised: We set k = 1 for all experiments except k = 3 for finetuning from ImageNet pre-trained models.

Original: To evaluate, we use the soft nearest-neighbor classifier for Semi-iNat and CIFAR-10 and accuracy after fine-tuning for ImageNet.
Revised: To evaluate, we use the soft nearest-neighbor classifier for Semi-iNat and CIFAR-10 and accuracy after fine-tuning for ImageNet.

Original: All the reported results use the same architecture, ResNet-50 for large-scale and WRN-28-2 for small-scale experiments.
Revised: All the reported results use the same architecture, ResNet-50 for large-scale and WRN-28-2 for small-scale experiments.

Original: See Appendix C for additional details and discussions on the choice of hyperparameters.
Revised: See Appendix C for additional details and discussions on the choice of hyperparameters.

Original: 4. L ARGE - SCALE EXPERIMENTS
Revised: 4. L ARGE - SCALE EXPERIMENTS

Original: Semi-iNat.
Revised: Semi-iNat.

Original: Semi-iNat (Su & Maji, 2021b) is a realistic large-scale benchmark for semi-supervised learning, containing a labeled dataset of 9,721 images with 810 classes, along with two unlabeled datasets: (a) a small curated one of 91,336 images with the same 810 classes and (b) a large uncurated one of 313,248 images with 2,439 classes.
Revised: Semi-iNat (Su & Maji, 2021b) is a realistic large-scale benchmark for semi-supervised learning, containing a labeled dataset of 9,721 images with 810 classes, along with two unlabeled datasets: (a) a small curated one of 91,336 images with the same 810 classes and (b) a large uncurated one of 313,248 images with 2,439 classes.

Original: We compare RoPAWS (and PAWS) with various baselines:
Revised: We compare RoPAWS (and PAWS) with various baselines:

Original: Pseudo-label (Lee et al., 2013), FixMatch (Sohn et al., 2020), Self-training (Hinton et al.,  2014), and CCSSL (Yang et al., 2022), trained from scratch or pre-trained MoCo-v2 (Chen et al., 2020c).{+MoCo-v
+}
Revised: Pseudo-label (Lee et al., 2013), FixMatch (Sohn et al., 2020), Self-training (Hinton et al., [Table] 2014), and CCSSL (Yang et al., 2022), trained from scratch or pre-trained {+MoCo-v
+}

Revised: (Chen et al., 2020c).

Original: We include the results reported in CCSSL and Semi-iNat (Su & Maji, 2021a).
Revised: We include the results reported in CCSSL and Semi-iNat (Su & Maji, 2021a)

Original: [Table]
Revised: [Table]

Original: Table 1 presents the results on Semi-iNat.
Revised: Table 1 presents the results on Semi-iNat.

Original: RoPAWS achieves 43.0% of accuracy for the uncurated, from scratch setup, giving +5.3% of gain over PAWS and improving the state-of-the-art by +1.7%.
Revised: RoPAWS achieves 43.0% of accuracy for the uncurated, from scratch setup, giving +5.3% of gain over PAWS and improving the state-of-the-art by +1.7%.

Original: Note that the prior art, MoCo-v2+FixMatch+CCSSL requires a pre-training of MoCo-v2 and takes a longer training time.
Revised: Note that the prior art, CCSSL requires a pre-training of MoCo-v2 and takes a longer training time.

Original: Also, remark that RoPAWS trained on uncurated data from scratch exceeds the one trained on curated data, verifying the benefit of representation learned from large uncurated data.
Revised: Also, remark that RoPAWS trained on uncurated data from scratch exceeds the one trained on curated data, verifying the benefit of representation learned from large uncurated data.

Original: In addition, RoPAWS improves the curated setups since it regularizes the prediction of uncertain inclass samples.
Revised: In addition, RoPAWS improves the curated setups since it regularizes the prediction of uncertain in-class samples.

Original: 7 Finally, RoPAWS further boosts the accuracy when using an ImageNet pre-trained model.
Revised: 9 Finally, RoPAWS further boosts the accuracy when using an ImageNet pre-trained model.

Original: Note that learning representations for the downstream task is still important when fine-tuning a model (Reed et al., 2022), and RoPAWS gives an effective solution for this practical scenario.
Revised: Note that learning representations for the downstream task is still important when fine-tuning a model (Reed et al., 2022), and RoPAWS gives an effective solution for this practical scenario.

Original: ImageNet.
Revised: ImageNet.

Original: Following prior works, we provide (curated) ImageNet (Deng et al., 2009)
Revised: Following prior works, we provide (curated) ImageNet (Deng et al., 2009)

Original: experiments using the full ImageNet as unlabeled data and its 1% or 10% subset as labeled data.
Revised: experiments using the full ImageNet as unlabeled data and its 1% or 10% subset as labeled data.

Original: We compare
Revised: We compare

Original: RoPAWS and PAWS (state-of-the-art) with various baselines: (a) prediction-based semi-supervised learning (Semi-SL) such as UDA (Xie et al., 2020a), FixMatch (Sohn et al., 2020), and MPL (Pham et al., 2021), and (b) pre-training with self-supervised learning (Self-SL) and fine-tuning such as
Revised: RoPAWS and PAWS (state-of-the-art) with various baselines: (a) prediction-based semi-supervised learning (Semi-SL) such as UDA (Xie et al., 2020a), FixMatch (Sohn et al., 2020), and MPL (Pham et al., 2021), and (b) pre-training with self-supervised learning (Self-SL) and fine-tuning such as

Original: BYOL (Grill et al., 2020), SwAV (Caron et al., 2020), and SimCLRv2 (Chen et al., 2020b).
Revised: BYOL (Grill et al., 2020), SwAV (Caron et al., 2020), and SimCLRv2 (Chen et al., 2020b).

Original: PAWS and RoPAWS fall into the representation-based Semi-SL.
Revised: PAWS and RoPAWS fall into the representation-based Semi-SL.

Original: Table 2 presents RoPAWS outperforms
Revised: Table 2 presents RoPAWS outperforms

Original: PAWS and achieves state-of-the-art, e.g., +0.4% for 1% labels.
Revised: PAWS and achieves state-of-the-art, e.g., +0.4% for 1% labels.

Original: It verifies that the calibrated prediction of RoPAWS is also effective for curated datasets.

Original: 4. S MALL - SCALE EXPERIMENTS
Revised: 4. S MALL - SCALE EXPERIMENTS

Original: CIFAR-10.
Revised: CIFAR-10.

Original: We consider three settings for CIFAR-10: using (a) no extra data (curated setting), (b)
Revised: We consider three settings for CIFAR-10: using (a) no extra data (curated setting), (b)

Original: CIFAR-100 (hard OOD), and (c) Tiny-ImageNet (easy OOD), as extra unlabeled data.
Revised: CIFAR-100 (hard OOD), and (c) Tiny-ImageNet (easy OOD), as extra unlabeled data.

Original: We compare RoPAWS with PAWS and FixMatch (Sohn et al., 2020), together with state-of-the-art robust semisupervised learning (OOD filtering) methods: UASD (Chen et al., 2020d), DS 3 L (Guo et al., 2020), OpenMatch (Saito et al., 2021), and OpenCos (Park et al., 2021).
Revised: We compare RoPAWS with PAWS and FixMatch (Sohn et al., 2020), together with state-of-the-art robust Semi-SL (OOD filtering) methods: UASD (Chen et al., 2020d), DS 3 L (Guo et al., 2020), OpenMatch (Saito et al., 2021), and OpenCos (Park et al., 2021).

Original: We use the reported results from the FixMatch paper for setup (a) and the OpenCos paper for setup (c).
Revised: We use the reported results from the FixMatch paper for setup (a) and the OpenCos paper for setup (c).

Original: For other setups, we reproduce
Revised: For other setups, we reproduce

Original: FixMatch and OpenMatch using the default configurations.
Revised: FixMatch and OpenMatch using the default configurations.

Original: We use 25 and 400 labels for each class following an existing practice.
Revised: We use 25 and 400 labels for each class.

Original: We report the mean and standard deviation for 5 runs for (a) and (b) and 3 runs for (c), following prior works.
Revised: We report the mean and standard deviation for 5 runs for (a) and (b) and 3 runs for (c).

Original: Table 3 presents the results on CIFAR-10.
Revised: Table 3 presents the results on CIFAR-10.

Original: RoPAWS improves PAWS for all scenarios, especially when using extra uncurated data.
Revised: RoPAWS improves PAWS for all scenarios, especially when using extra uncurated data.

Original: For example, RoPAWS improves PAWS by +22.7% for the easy
Revised: For example, RoPAWS improves PAWS by +22.7% for the easy

Original: OOD (+ Tiny-ImageNet) and +6.9% for the hard OOD (+ CIFAR-100) setups using 25 labels. 
Revised: OOD (+ Tiny-ImageNet) and +6.9% for the hard OOD (+ CIFAR-100) setups using 25 labels. [Table] [Figure] [Table]

Revised: [Figure]

Original: RoPAWS also exceeds the prior state-of-the-arts: OpenMatch and OpenCos.
Revised: RoPAWS also exceeds the prior state-of-the-arts: OpenMatch and OpenCos.

Original: These OOD filtering methods are effective for easy OOD setups.
Revised: These OOD filtering methods are effective for easy OOD setups.

Original: However, RoPAWS even exceeds OpenCos pre-trained [Table] [Figure] on SimCLR for (+ Tiny-ImageNet) without such pre-training.
Revised: However, RoPAWS even exceeds OpenCos pretrained on SimCLR for (+ Tiny-ImageNet) without such pre-training.

Original: Also, note that the OOD filtering methods are less effective for hard OOD setups (see Appendix I for additional discussion).
Revised: Also, note that the OOD filtering methods are less effective for hard OOD setups (see Appendix L for additional discussion).

Original: RoPAWS is still effective for (+ CIFAR-100) since it does not filter samples but learns from uncurated data.
Revised: RoPAWS is still effective for (+ CIFAR-100) since it does not filter samples but learns from uncurated data.

Revised: We provide additional comparison with robust Semi-SL methods in Appendix G.

Original: 4.  A BLATION STUDY AND ANALYSIS
Revised:  A BLATION STUDY AND ANALYSIS

Original: Ablation study.
Revised: Ablation study.

Original: Table 4 presents the ablation study of the components of RoPAWS, trained on SemiiNat (uncurated).
Revised: Table 4 presents the ablation study of the components of RoPAWS, trained on SemiiNat (uncurated).

Original: Semi-supervised density estimation in Eq.
Revised: Semi-supervised density estimation in Eq.

Original: (11) highly improves the accuracy, and in-domain prior in Eq.
Revised: (11) highly improves the accuracy, and in-domain prior in Eq.

Original: (13) (combined with a reweighted loss in Eq.
Revised: (13) (combined with a reweighted loss in Eq.

Original: (14)) enhances further.
Revised: (14)) enhances further.

Original: We provide additional ablation studies in Appendix D, showing that RoPAWS gives reasonable results over varying hyperparameters and that all the proposed components are essential.
Revised: We provide additional ablation studies in Appendix D, showing that RoPAWS gives reasonable results over varying hyperparameters and that all the proposed components are essential.

Original: Analysis.
Revised: Analysis.

Original: We analyze the learned representations of RoPAWS.
Revised: We analyze the learned representations of RoPAWS.

Original: Figure 4 shows the t-SNE (Van der
Revised: Figure 4 shows the t-SNE (Van der

Original: Maaten& Hinton, 2008) visualization of embeddings, trained on CIFAR-10 (+ CIFAR-100),labels. As PAWS confidently predicts unlabeled data as in-class, OOD data (black points) are concentrated in the in-class clusters (colored points). In contrast, RoPAWS pushes out-of-class data far from all the in-class clusters. We provide additional analyses in Appendix G showing that RoPAWS increases the confidence of in-domain and decreases one of OOD data.
Revised: Maaten

Original: Nearest neighbors.
Revised: Nearest neighbors.

Original: We visualize the nearest labeled data of out-of-class data inferred by PAWS and RoPAWS in Appendix F.
Revised: We visualize the nearest labeled data of out-of-class data inferred by PAWS and RoPAWS in Appendix F.

Original: Even though the classes are different, both models can find visually similar in-class data with slight variation.
Revised: Even though the classes are different, both models can find visually similar in-class data with slight variation.

Original: However, PAWS has high similarity scores to the wrong in-class neighbors, unlike RoPAWS has lower similarities, giving uncertain predictions.
Revised: However, PAWS has high similarity scores to the wrong in-class neighbors, unlike RoPAWS has lower similarities, giving uncertain predictions.

Original: Computation time.
Revised: Computation time.

Original: The inference time of RoPAWS is identical to PAWS.
Revised: The inference time of RoPAWS is identical to PAWS.

Original: The training time is only slightly increased ( < 5% for each mini-batch) due to the (detached) targets in Eq.
Revised: The training time is only slightly increased ( < 5% for each mini-batch) due to the (detached) targets in Eq.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We proposed RoPAWS, a novel robust semi-supervised learning method for uncurated data, which calibrates prediction via density modeling on representation space.
Revised: We proposed RoPAWS, a novel robust semi-supervised learning method for uncurated data, which calibrates prediction via density modeling on representation space.

Original: We verified the effectiveness of
Revised: We verified the effectiveness of

Original: RoPAWS under various scenarios, achieving new state-of-the-art results on realistic benchmarks.
Revised: RoPAWS under various scenarios, achieving new state-of-the-art results.

Original: In addition, the probabilistic view of RoPAWS can be extended to various problems, e.g., handling class imbalance (Kim et al., 2020; Wei et al., 2021) by adjusting the prior of labels or open-world (Cao et al., 2022; Vaze et al., 2022) by modeling densities of novel classes.
Revised: In addition, the probabilistic view of RoPAWS can be extended to various problems, as discussed in Appendix M. A CKNOWLEDGMENTS This work was supported by Institute of Information & communications Technology Planning & evaluation (IITP) grant funded by the Korea government (MSIT) (No.2019-0-00075, Artificial Intelligence Graduate School Program (KAIST) and No.2021-0-02068, Artificial Intelligence Innovation Hub).

Revised: SW thanks to Ser-Nam Lim for supporting technical issues for using fairaws cluster.

Original: E THICS STATEMENT
Revised: E THICS STATEMENT

Original: Learning representations from large and diverse data makes the model more robust and fair (Goyal et al., 2022).
Revised: Learning representations from large and diverse data makes the model more robust and fair (Goyal et al., 2022).

Original: However, the uncurated data may inherit the natural bias of data distribution.
Revised: However, the uncurated data may inherit the natural bias of data distribution.

Original: Thus, the user should not treat the model as a black-box output from arbitrary data but should carefully look at the data’s statistics and be aware of the potential biases.
Revised: Thus, the user should not treat the model as a black-box output from arbitrary data but should carefully look at the data’s statistics and be aware of the potential biases.

Original: R EPRODUCIBILITY STATEMENT
Revised: R EPRODUCIBILITY STATEMENT

Original: We provide implementation details in Appendix C. We provide our code in the supplementary material, which will be publicly released after the paper acceptance.
Revised: We provide implementation details in Appendix C. We provide our code in the supplementary material, which will be publicly released after the paper acceptance

Original: A D ETAILED DERIVATIONS
Revised: A D ETAILED DERIVATIONS

Original: We provide detailed derivations for the prediction of probabilistic PAWS and RoPAWS.
Revised: We provide detailed derivations for the prediction of probabilistic PAWS and RoPAWS.

Original: PAWS (one-hot).
Revised: PAWS (one-hot).

Original: We defined the conditional p ( x | y ) and marginal p ( x ) densities as follows:
Revised: We defined the conditional p ( x | y ) and marginal p ( x ) densities as follows:

Original: [Equation]
Revised: [Equation]

Original: where k ( z, z ′ ) :
Revised: where k ( z, z ′ ) :

Original: = 1 Z · exp( z · z ′ /τ ) .
Revised: = 1 Z · exp( z · z ′ /τ ) .

Original: Recall that |B l | = N , |B yl | =
Revised: Recall that |B l | = N , |B yl | =

Original: NC , and p ( y ) = 1 C .
Revised: NC , and p ( y ) = 1 C .

Original: Note that this definition satisfies the marginalization:
Revised: Note that this definition satisfies the marginalization:

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Applying the definitions to the Bayes’ rule,
Revised: Applying the definitions to the Bayes’ rule,

Original: [Equation]
Revised: [Equation]

Original: where σ τ ( z · z ⊺ l ) ∈ R 1 × N is a similarity vector,
Revised: where σ τ ( z · z ⊺ l ) ∈ R 1 × N is a similarity vector,

Original: 1 x ′ ∈
Revised: 1 x ′ ∈

Original: R N × 1 is an index vector where the index of x ′ is 1 otherwise 0, 1 is an indicator function, and p y | x l ∈ R N × 1 is a (one-hot) label vector.
Revised: R N × 1 is an index vector where the index of x ′ is 1 otherwise 0, 1 is an indicator function, and p y | x l ∈ R N × 1 is a (one-hot) label vector.

Original: PAWS (soft).
Revised: PAWS (soft).

Original: Applying the weighted kernel k y ( z, z ′ )
Revised: Applying the weighted kernel k y ( z, z ′ )

Original: = 1 K p ( y | x ′ ) k
Revised: = 1 K p ( y | x ′ ) k

Original: ( z, z ′ ) , the extended definition of conditional density p ( x | y ) is as follows:
Revised: ( z, z ′ ) , the extended definition of conditional density p ( x | y ) is as follows:

Original: [Equation]
Revised: [Equation]

Original: Then, the marginal density p ( x ) is given by:
Revised: Then, the marginal density p ( x ) is given by:

Original: [Equation]
Revised: [Equation]

Original: Applying the definitions to the Bayes’ rule
Revised: Applying the definitions to the Bayes’ rule

Original: [Equation]
Revised: [Equation]

Original: where p y | x l ∈ R N × 1 is a (soft) label vector.
Revised: where p y | x l ∈ R N × 1 is a (soft) label vector.

Original: RoPAWS.
Revised: RoPAWS.

Original: We extend the densities to consider both labeled and unlabeled data:
Revised: We extend the densities to consider both labeled and unlabeled data:

Original: [Equation]
Revised: [Equation]

Original: Here, we balance the effect of labeled and unlabeled data with ratio r : 1 .
Revised: Here, we balance the effect of labeled and unlabeled data with ratio r : 1 .

Original: Specifically, we use
Revised: Specifically, we use

Original: [Equation]
Revised: [Equation]

Original: where r ′ :
Revised: where r ′ :

Original: = r · |B u | / |B l | balances the overall effect of labeled batch.
Revised: = r · |B u | / |B l | balances the overall effect of labeled batch.

Original: Similar to PAWS soft labels, it satisfies the marginalization and gives a similar prediction formula
Revised: Similar to PAWS soft labels, it satisfies the marginalization and gives a similar prediction formula

Original: [Equation]
Revised: [Equation]

Original: We set 
Revised: We set ˜ r

Original: [Equation]
Revised: [Equation]

Original: x [Equation] := [Equation]
Revised: x ′ ) :

Original: [Equation] x [Equation] l + 1 x ′ ∈B u , and simply we can write:
Revised: = r ′ · 1 x ′ ∈B l + 1 x ′ ∈B u , and simply we can write:

Original: [Equation]
Revised: [Equation]

Original: Applying the definitions to the Bayes’ rule
Revised: Applying the definitions to the Bayes’ rule,

Original: [Equation]
Revised: [Equation]

Original: Recall that we do not assume p ( y ) to be uniform nor cancel out with C to control the in-domain prior.
Revised: Recall that we do not assume p ( y ) to be uniform nor cancel out with C to control the in-domain prior.

Original: This formula can be simplified as a softmax (as above) by adding a constant vector r .
Revised: This formula can be simplified as a softmax (as above) by adding a constant vector r .

Original: Specifically, let r ∈ R 1 × N + M be a vector where the index of x ′
Revised: Specifically, let r ∈ R 1 × N + M be a vector where the index of x ′

Original: ∈ B l is τ · log( r ′ ) and otherwise 0.
Revised: ∈ B l is τ · log( r ′ ) and otherwise 0.

Original: Then,
Revised: Then,

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: and splitting the concatenated vectors z and p y | x (and omitting r ) into labeled and unlabeled terms gives the final formula:
Revised: and splitting the concatenated vectors z and p y | x (and omitting r ) into labeled and unlabeled terms gives the final formula:

Original: [Equation]
Revised: [Equation]

Original: To sum up, the prediction of unlabeled data is computed from the predictions of N labeled and M unlabeled data.
Revised: To sum up, the prediction of unlabeled data is computed from the predictions of N labeled and M unlabeled data.

Original: We stack this for M unlabeled data and then get the matrix formula of prediction for unlabeled data q ∈ R M × C and labeled data p ∈ R N × C with weights a ∈ R M × N and b ∈ R M × M given by the similarity matrix.
Revised: We stack this for M unlabeled data and then get the matrix formula of prediction for unlabeled data q ∈ R M × C and labeled data p ∈ R N × C with weights a ∈ R M × N and b ∈ R M × M given by the similarity matrix.

Original: Namely, the simplified version of Eq.
Revised: Namely, the simplified version of Eq.

Original: (10) looks like:
Revised: (10) looks like:

Original: [Equation]
Revised: [Equation]

Original: and organizing the formula gives the closed-form solution:
Revised: and organizing the formula gives the closed-form solution:

Original: [Equation]
Revised: [Equation]

Original: and thus, q
Revised: and thus, q

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: 51) which gives the final prediction formula in Eq.
Revised: which gives the final prediction formula in Eq.

Original: Note that ( I − b ) is invertible since b is the part of the similarity matrix, i.e., the row-wise sum of b < 1 .
Revised: Note that ( I − b ) is invertible since b is the part of the similarity matrix, i.e., the row-wise sum of b < 1 .

Original: We remark that this formula can be interpreted as the label propagation, i.e., propagating the belief of labels p (and also the priors on q in our case) to get the posterior prediction of q .
Revised: We remark that this formula can be interpreted as the label propagation, i.e., propagating the belief of labels p (and also the priors on q in our case) to get the posterior prediction of q .

Original: Indeed, think about the iterative formula:
Revised: Indeed, think about the iterative formula:

Original: [Equation]
Revised: [Equation]

Original: for i ≥ 0 and initialized from the prior q 0 .
Revised: for i ≥ 0 and initialized from the prior q 0 .

Original: Here, we can think our closed-form solution as the fixed-point of the iteration (it converges at i → ∞ since ∥ b ∥ <
Revised: Here, we can think our closed-form solution as the fixed-point of the iteration (it converges at i → ∞ since ∥ b ∥ <

Original: I , i.e., contraction mapping).
Revised: I , i.e., contraction mapping).

Original: Finally, recall that the prediction q ( y | x ) (a row from q ) can have a row-wise sum < 1 when using the in-domain prior.
Revised: Finally, recall that the prediction q ( y | x ) (a row from q ) can have a row-wise sum < 1 when using the in-domain prior.

Original: It implies that the data x is likely to be in-domain with (posterior) probability ¯ q = (cid:80) y q
Revised: It implies that the data x is likely to be in-domain with (posterior) probability ¯ q = (cid:80) y q

Original: ( y | x ) and OOD with probability 1 − ¯ q .
Revised: ( y | x ) and OOD with probability 1 − ¯ q .

Original: We regularize the prediction to be close to uniform for uncertain (i.e., OOD-like) samples.
Revised: We regularize the prediction to be close to uniform for uncertain (i.e., OOD-like) samples.

Original: Here, the final prediction value is q ( y | x )+(1 − ¯ q ) /C , which makes the sum of the probability for C in-class to be one.
Revised: Here, the final prediction value is q ( y | x )+(1 − ¯ q ) /C , which makes the sum of the probability for C in-class to be one.

Original: B F UTURE RESEARCH DIRECTIONS
Revised: B N ON - COLLAPSING PROPERTY OF REPRESENTATION

Original: We believe the probabilistic formulation ofRoPAWS, i.e., KDE modeling view over representation space, has the potential to be extended to various scenarios in a principled manner. For example, we assume the prior class distribution p ( y ) is uniform over all classes. One can extend RoPAWS to class-imbalanced setup (Kim et al., 2020; Wei et al., 2021) by assuming non-uniform prior.
Revised: One of the main advantages of PAWS is non-collapsing representations.

Original: RoPAWScan also handle label shift, i.e., class distribution of labeled and unlabeled data are different.
Revised: RoPAWS also favors this property.

Original: Indeed, the prediction of RoPAWS only assumes the prior on unlabeled data, which can be controlled independently from the prior on labeled data.
Revised: Recall that the proof logic of Proposition 1 in the PAWS paper is as follows:

Original: On the other hand, the entropy mean entropy maximization (me-max) regularization − H (¯ p ) in the objective is identical to minimizing the KL divergence between the mean prediction and the uniform prior. This can be extended to minimizing the (posterior) mean prediction and the prior when using non-uniform prior.
Revised: When the representation collapses, the distances between representations d ( z, z i ) become identical for all z i in the labeled batch B l . 2.

Original: Also, one can extend RoPAWS to open-world setup (Cai et al., 2022; Vaze et al., 2022), which clusters novel classes in addition to predicting known classes. Here, one can model conditional densities p ( x | y ) for novel classes y ∈
Revised: Then, the prediction p is given by the average of class labels y i , which becomes a uniform distribution due to the class-balanced sampling.

Original: { C +1 , ..., C + C novel } . Then, finding the maximum likelihood assignment of cluster y for unlabeled data would cluster both labeled and unlabeled data similarly to the infinite Gaussian mixture model (Rasmussen, 1999) but under KDE cluster modeling. Finally, one may use the estimated probability, e.g., p  (in | x ) or p ( x | y )  , to choose the most uncertain sample to label for active learning (Settles, 2009).
Revised: However, the target sharpening makes the target p + not equal to the uniform distribution; hence, ∇ θ H ( p + , p ) > 0 , and the model escapes from the representation collapse.

Original: Combining semi-supervised learning and active learning under large uncurated data would be important for practical scenarios. We remark that  the principle of RoPAWS can be generalized beyond PAWS and image classification.
Revised: RoPAWS follows the same logic: the prediction p becomes a uniform distribution when representation collapses, being escaped by step 3 above. The difference from PAWS is that now the prediction is the average of true labels

Original: For example, the standard softmax classifier can be interpreted as a generative classifier modeling densities with the Gaussian mixture model (GMM) (Lee et al., 2018b). Thus, one can adapt the softmax classifier of previous algorithms, such as FixMatch (Sohn et al., 2020), as a GMM classifier to estimate the uncertainty of samples.
Revised: y i of labeled batch B l and pseudo-labels ˆ y i of unlabeled batch B u .

Original: However, GMM needs sufficient labels to estimate the variance of each class; hence, less fits for semi-supervised learning. One may extend RoPAWS for non-classification tasks such as object detection (Liu et al., 2022) by applying dense self-supervision (Wang et al., 2021) or object-aware augmentation (Mo et al., 2021). Also, one may tackle non-image domains, using different augmentations (Lee et al., 2021).
Revised: Here, the initial belief of pseudo-labels ˆ y i becomes a uniform distribution by averaging the true labels y i , and averaging this uniform distribution keeps the updated belief uniform after label propagation.

Original: C I MPLEMENTATION DETAILS C. C OMMON SETUPS
Revised: C I MPLEMENTATION DETAILS C. C OMMON SETUPS

Original: We follow the default configurations of PAWS.
Revised: We follow the default configurations of PAWS.

Original: Specifically, we follow the ImageNet configuration for large-scale (Semi-iNat, ImageNet) experiments and the CIFAR-10 configuration for small-scale (CIFAR-10) experiments.
Revised: Specifically, we follow the ImageNet configuration for large-scale (Semi-iNat, ImageNet) experiments and the CIFAR-10 configuration for small-scale (CIFAR-10) experiments.

Original: We only adopt the number of labels and training epochs for each dataset, as the dataset has a smaller size (longer epochs) or fewer labels (smaller labeled batch).
Revised: We only adopt the number of labels and training epochs for each dataset, as the dataset has a smaller size (longer epochs) or fewer labels (smaller labeled batch).

Original: Training.
Revised: Training.

Original: We sample an unlabeled and a labeled batch for each iteration.
Revised: We sample an unlabeled and a labeled batch for each iteration.

Original: For the unlabeled batch, we sample 2 views for targets and 6 additional multi-crop views (8 in total) for outputs.
Revised: For the unlabeled batch, we sample 2 views for targets and 6 additional multi-crop views (8 in total) for outputs.

Original: We compute the semi-supervised prediction of RoPAWS for each view independently, and the size of the unlabeled batch (so as computation complexity) does not increase over the number of views.
Revised: We compute the semi-supervised prediction of RoPAWS for each view independently, and the size of the unlabeled batch (so as computation complexity) does not increase over the number of views.

Original: We use the softmax temperature τ = 0 .
Revised: We use the softmax temperature τ = 0 .

Original: 1 and sharpen the target with temperature T = 0 .
Revised: 1 and sharpen the target with temperature T = 0 .

Original: 25 , i.e.,
Revised: 25 , i.e.,

Original: [Equation]
Revised: [Equation]

Original: where p i and ρ ( p ) i denotes the i -th element of probability p, ρ ( p ) ∈ R 1 × C .
Revised: where p i and ρ ( p ) i denotes the i -th element of probability p, ρ ( p ) ∈ R 1 × C .

Original: We apply the mean entropy maximization (me-max) regularization − H (¯ p ) for all experiments.
Revised: We apply the mean entropy maximization (me-max) regularization − H (¯ p ) for all experiments.

Original: For the labeled batch, we sample C classes (from overall C all > C classes) for each iteration, where
Revised: For the labeled batch, we sample C classes (from overall C all > C classes) for each iteration, where

Original: C depends on datasets, e.g., C = 960 for ImageNet with C all = 1000 .
Revised: C depends on datasets, e.g., C = 960 for ImageNet with C all = 1000 .

Original: We sample the same number of data for each class (i.e., class balanced) with 1 or 2 augmentation views.
Revised: We sample the same number of data for each class (i.e., class balanced) with 1 or 2 augmentation views.

Original: Thus, the overall support size is the number of classes × samples per class × number of views.
Revised: Thus, the overall support size is the number of classes × samples per class × number of views.

Original: We apply the same data augmentation with PAWS, SimAug (Chen et al., 2020a), for all experiments.
Revised: We apply the same data augmentation with PAWS, SimAug (Chen et al., 2020a), for all experiments.

Original: The same augmentation is applied for both unlabeled and labeled data.
Revised: The same augmentation is applied for both unlabeled and labeled data.

Original: For large-scale experiments, we train a ResNet-50 network using an unlabeled batch of size 4096 on 64 GPUs.
Revised: For large-scale experiments, we train a ResNet-50 network using an unlabeled batch of size 4096 on 64 GPUs.

Original: We use a 3-layer multi-layer perceptron (MLP) projection head with 2048 hidden dimensions and a 2-layer MLP prediction head with 512 hidden dimensions.
Revised: We use a 3-layer multi-layer perceptron (MLP) projection head with 2048 hidden dimensions and a 2-layer MLP prediction head with 512 hidden dimensions.

Original: This projection/prediction head is widely applied for self-supervised learning (Grill et al., 2020).
Revised: This projection/prediction head is widely applied for self-supervised learning (Grill et al., 2020).

Original: We use the LARS (You et al., 2017) optimizer with momentum value 0.9 and weight decay 10 − 6 .
Revised: We use the LARS (You et al., 2017) optimizer with momentum value 0.9 and weight decay 10 − 6 .

Original: We linearly warm up the learning rate from 0.3 to 6.4 during the first 10 epochs and apply the cosine schedule afterward.
Revised: We linearly warm up the learning rate from 0.3 to 6.4 during the first 10 epochs and apply the cosine schedule afterward.

Original: For small-scale experiments, we train a WRN-28-2 network using an unlabeled batch of sizeon 1 GPU.
Revised: For small-scale experiments, we train a WRN-28-2 network using an unlabeled batch of sizeon 1 GPU.

Original: We use a 3-layer MLP projection head with 128 hidden dimensions and do not use a prediction head.
Revised: We use a 3-layer MLP projection head with 128 hidden dimensions and do not use a prediction head.

Original: We use the LARS optimizer with the same configuration of the large-scale setup but linearly warm up the learning rate from 0.8 to 3.2 and then apply the cosine schedule.
Revised: We use the LARS optimizer with the same configuration of the large-scale setup but linearly warm up the learning rate from 0.8 to 3.2 and then apply the cosine schedule.

Original: Evaluation.
Revised: Evaluation.

Original: We use a soft-nearest neighbor classifier with softmax temperature τ = 0 .
Revised: We use a soft-nearest neighbor classifier with softmax temperature τ = 0 .

Original: 1 (same to the training) for Semi-iNat and CIFAR-10.
Revised: 1 (same to the training) for Semi-iNat and CIFAR-10.

Original: We fine-tune the classifier upon the learned representation for ImageNet since it gives a further gain.
Revised: We fine-tune the classifier upon the learned representation for ImageNet since it gives a further gain.

Original: We follow the fine-tuning configuration of PAWS, training a linear classifier on top of the first layer of the projection head initialized from zero weights.
Revised: We follow the fine-tuning configuration of PAWS, training a linear classifier on top of the first layer of the projection head initialized from zero weights.

Original: We fine-tune the backbone and linear classifier simultaneously, optimizing a supervised cross-entropy loss on the small set of labeled data.
Revised: We fine-tune the backbone and linear classifier simultaneously, optimizing a supervised cross-entropy loss on the small set of labeled data.

Original: We apply the basic data augmentations (random crop and horizontal flip) and sweep the learning rate from { 0 . 01 , 0 . 02 , 0 . 05 , 0 . 1 , 0 . 2 } and epochs from { 30 , 50 } , choosing the hyperparameter based on a 12,000 images subset from the ImageNet training set.
Revised: We apply the basic data augmentations (random crop and horizontal flip) and sweep the learning rate from { 0 . 01 , 0 . 02 , 0 . 05 , 0 . 1 , 0 . 2 } and epochs from { 30 , 50 } , choosing the hyperparameter based on a 12,000 images subset from the ImageNet training set.

Original: We use SGD with Nesterov momentum (with value of 0.9) and a batch size of 1024.
Revised: We use SGD with Nesterov momentum (with value of 0.9) and a batch size of 1024.

Original: We apply a single center-crop augmentation for test samples and report the top-1 accuracy.
Revised: We apply a single center-crop augmentation for test samples and report the top-1 accuracy.

Original: C. D ATASET - SPECIFIC SETUPS
Revised: C. D ATASET - SPECIFIC SETUPS

Original: Semi-iNat.
Revised: Semi-iNat.

Original: We train from scratch models for 800 epochs, and ImageNet pre-trained models forepochs.
Revised: We train from scratch models for 800 epochs, and ImageNet pre-trained models forepochs.

Original: We use 12 classes per batch (thus, 12 × 64 = 768 < 810 classes are used for each iteration in total) and 5 images for each class (Semi-iNat is long-tailed, and the tail classes only have 5 labels) with 1 view.
Revised: We use 12 classes per batch (thus, 12 × 64 = 768 < 810 classes are used for each iteration in total) and 5 images for each class (Semi-iNat is long-tailed, and the tail classes only havelabels) with 1 view.

Original: We use the official SwAV weights 8 for the ImageNet pre-trained experiments, where we also compare them with the supervised weights 9 used in the prior work.
Revised: We use the official SwAV weights 11 for the ImageNet pre-trained experiments, where we also compare them with the supervised weights 12 used in the prior work.

Original: We found that SwAV gives a better initialization for PAWS and RoPAWS as the objective (and thus, learned representations) of SwAV is similar to PAWS, and used SwAV for our main experiments.
Revised: We found that SwAV gives a better initialization for PAWS and RoPAWS as the objective (and thus, learned representations) of SwAV is similar to PAWS, and used SwAV for our main experiments.

Original: ImageNet.
Revised: ImageNet.

Original: We train models for 300 epochs, following the default PAWS configuration.
Revised: We train models for 300 epochs, following the default PAWS configuration.

Original: We useclasses per batch (thus, 15 × 64 = 960 < 1 , 000 classes are used for each iteration in total) andimages for each class with 1 view.
Revised: We useclasses per batch (thus, 15 × 64 = 960 < 1 , 000 classes are used for each iteration in total) andimages for each class with 1 view.

Original: We use the same subset split of PAWS.
Revised: We use the same subset split of PAWS.

Original: CIFAR-10.
Revised: CIFAR-10.

Original: We train models for 600 epochs, following the default PAWS configuration.
Revised: We train models for 600 epochs, following the default PAWS configuration.

Original: We use the full 10 classes and 64 images for each class with 2 views for 400 labels setup.
Revised: We use the full 10 classes and 64 images for each class with 2 views for 400 labels setup.

Original: For 25 labels andlabels setups, we decrease the images for each class and increase the number of views to keep the total size of labeled support.
Revised: For 25 labels andlabels setups, we decrease the images for each class and increase the number of views to keep the total size of labeled support.

Original: Specifically, we use 16 images for each class with 8 views for 25 labels setup and 4 images for each class with 32 views for 4 labels setup.
Revised: Specifically, we use 16 images for each class with 8 views for 25 labels setup and 4 images for each class with 32 views for 4 labels setup.

Original: C. H YPERPARAMETERS OF R O PAWS
Revised: C. H YPERPARAMETERS OF R O PAWS

Original: RoPAWS has three hyperparameters: the ratio of labeled and unlabeled batch r , the temperature for in-domain prior τ prior , and the power of reweighted loss k .
Revised: RoPAWS has three hyperparameters: the ratio of labeled and unlabeled batch r , the temperature for in-domain prior τ prior , and the power of reweighted loss k .

Original: Choice of r .
Revised: Choice of r .

Original: We use r = 5 for all experiments following the default batch size ratio in CIFAR-10.
Revised: We use r = 5 for all experiments following the default batch size ratio in CIFAR-10.

Original: CIFAR-10 has a labeled batch size of 10 classes × 64 samples per class × 2 views = 1280, and an unlabeled batch size of 256.
Revised: CIFAR-10 has a labeled batch size of 10 classes × 64 samples per class × 2 views = 1280, and an unlabeled batch size of 256.

Original: Thus, the labeled and unlabeled batch ratio is 1280 / 256 = 5 .
Revised: Thus, the labeled and unlabeled batch ratio is 1280 / 256 = 5 .

Original: We observed that using r > 1 is helpful since the prediction on the unlabeled data is uncertain; giving more weights for certain labels is a safer choice.
Revised: We observed that using r > 1 is helpful since the prediction on the unlabeled data is uncertain; giving more weights for certain labels is a safer choice.

Original: Recall that the original batch ratio of Semi-iNat is 768 × 5 × 1 / 4096 ≈ 0 . 94 , giving more weights for unlabeled than labeled data.
Revised: Recall that the original batch ratio of Semi-iNat is 768 × 5 × 1 / 4096 ≈ 0 . 94 , giving more weights for unlabeled than labeled data.

Original: Choice of τ prior .
Revised: Choice of τ prior .

Original: We set τ prior = 3 . 0 for ResNet-50 and τ prior = 0 .
Revised: We set τ prior = 3 . 0 for ResNet-50 and τ prior = 0 .

Original: 1 for WRN-28-2.
Revised: 1 for WRN-28-2.

Original: Since ResNethas a larger latent dimension (2048) than WRN-28-2 (128), the embeddings are more likely to differ from the other embeddings, and cosine similarity decreases faster.
Revised: Since ResNethas a larger latent dimension (2048) than WRN-28-2 (128), the embeddings are more likely to differ from the other embeddings, and cosine similarity decreases faster.

Original: Thus, we use a larger temperature τ prior for ResNet-50 to not decay the in-domain prior too rapidly.
Revised: Thus, we use a larger temperature τ prior for ResNet-50 to not decay the in-domain prior too rapidly.

Original: Choice of k .
Revised: Choice of k .

Original: We set k = 1 for all experiments except k = 3 for fine-tuning experiments on SemiiNat, using ImageNet pre-trained models.
Revised: We set k = 1 for all experiments except k = 3 for fine-tuning experiments on SemiiNat, using ImageNet pre-trained models.

Original: Note that higher k makes the model more conservative; focus on confident samples and ignore uncertain samples.
Revised: Note that higher k makes the model more conservative; focus on confident samples and ignore uncertain samples.

Original: Models from scratch need to learn representation from uncurated data, thus setting lower k .
Revised: Models from scratch need to learn representation from uncurated data, thus setting lower k .

Original: In contrast, ImageNet pre-trained models have good representation and correct labels more matter, thus setting higher k .
Revised: In contrast, ImageNet pre-trained models have good representation and correct labels more matter, thus setting higher k .

Original: C. O THER BASELINES
Revised: C. O THER BASELINES

Original: We run FixMatch and OpenMatch for CIFAR-10 experiments.
Revised: We run FixMatch and OpenMatch for CIFAR-10 experiments.

Original: Here, we follow the default setup for CIFAR-10 for both models, using the unofficial (but matching the official performance)
Revised: Here, we follow the default setup for CIFAR-10 for both models, using the unofficial (but matching the official performance)

Original: PyTorch implementation of FixMatch 10 and the official PyTorch implementation of OpenMatch 11 .
Revised: PyTorch implementation of FixMatch 13 and the official PyTorch implementation of OpenMatch 14 .

Original: We did not tune hyperparameters specialized to each setup.
Revised: We did not tune hyperparameters specialized to each setup.

Original: However, FixMatch was effective for both curated (CIFAR-10) and hard OOD (CIFAR-10 + CIFAR-100) setups, implying that the default choices are already well-tuned.
Revised: However, FixMatch was effective for both curated (CIFAR-10) and hard OOD (CIFAR-10 + CIFAR-100) setups, implying that the default choices are already well-tuned.

Original: OpenMatch was effective for easy OOD (400 labels) but did not work well for hard OOD (25 labels) setups on CIFAR-10 (+ CIFAR-100).
Revised: OpenMatch was effective for easy OOD (400 labels) but did not work well for hard OOD (25 labels) setups on CIFAR-10 (+ CIFAR-100).

Original: It can be improved with a better choice of hyperparameters, yet, we remark that the OOD filtering approach (especially the ones discarding samples explicitly) is sensitive to the threshold and is not robust in the OOD setup.10 https https:// /OP_Match
Revised: It can be improved with a better choice of hyperparameters, yet, we remark that the OOD filtering approach (especially the ones discarding samples explicitly) is sensitive to the threshold and is not robust in the OOD setup.

Original: D A DDITIONAL ABLATION STUDIES
Revised: D A DDITIONAL ABLATION STUDIES

Original: We provide additional ablation studies for RoPAWS.
Revised: We provide additional ablation studies for RoPAWS.

Original: Table 5 presents the test accuracy of RoPAWS trained on Semi-iNat (uncurated) varying hyperparameters.
Revised: Table 5 presents the test accuracy of RoPAWS trained on Semi-iNat (uncurated) varying hyperparameters.

Original: Specifically, (a) shows the results varying the in-domain τ prior while fixing the reweight power k = 1 . 0 , and (b) shows the results varying the reweight power k while fixing the in-domain prior τ prior = 3 . 0 .
Revised: Specifically, (a) shows the results varying the in-domain τ prior while fixing the reweight power k = 1 . 0 , and (b) shows the results varying the reweight power k while fixing the in-domain prior τ prior = 3 . 0 .

Original: The results are stable under a reasonable range of hyperparameters, yet tuning may give a further boost.
Revised: The results are stable under a reasonable range of hyperparameters, yet tuning may give a further boost.

Original: Table 6 presents an extra ablation study for the components of RoPAWS.
Revised: Table 6 presents an extra ablation study for the components of RoPAWS.

Original: We applied reweighted loss when using in-domain prior in Table 4.
Revised: We applied reweighted loss when using in-domain prior in Table 4.

Original: Indeed, using in-domain prior without reweighted loss harms the accuracy.
Revised: Indeed, using in-domain prior without reweighted loss harms the accuracy.

Original: In addition, only using in-domain prior (and reweighted loss) without semi-supervised density harms the accuracy.
Revised: In addition, only using in-domain prior (and reweighted loss) without semi-supervised density harms the accuracy.

Original: It verifies that all the proposed components are essential
Revised: It verifies that all the proposed components are essential

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: E E FFECT OF PRE - TRAINING METHOD
Revised: E E FFECT OF PRE - TRAINING METHOD

Original: Table 7 presents the test accuracy of PAWS and RoPAWS, initialized from ImageNet supervised or
Revised: Table 7 presents the test accuracy of PAWS and RoPAWS, initialized from ImageNet supervised or

Original: SwAV (Caron et al., 2020) pre-trained models.
Revised: SwAV (Caron et al., 2020) pre-trained models.

Original: We used SwAV for our main table since it gives a better initialization.
Revised: We used SwAV for our main table since it gives a better initialization.

Original: Nevertheless, RoPAWS initialized from an ImageNet supervised model is already better than prior arts using the same pre-trained model.
Revised: Nevertheless, RoPAWS initialized from an ImageNet supervised model is already better than prior arts using the same pre-trained model.

Original: Intuitively, the training objective of
Revised: Intuitively, the training objective of

Original: PAWS (and RoPAWS) more resembles SwAV and thus has less representation shifts.
Revised: PAWS (and RoPAWS) more resembles SwAV and thus has less representation shifts.

Original: [Table]
Revised: [Table]

Original: F V ISUALIZATION OF NEAREST NEIGHBORS
Revised: F V ISUALIZATION OF NEAREST NEIGHBORS

Original: Figure 5 visualizes the nearest labeled data and its cosine similarity to queries inferred from PAWS and RoPAWS trained on Semi-iNat (uncurated).
Revised: Figure 5 visualizes the nearest labeled data and its cosine similarity to queries inferred from PAWS and RoPAWS trained on Semi-iNat (uncurated).

Original: Even for the out-of-class data, both models can find visually similar in-class data with slight variation.
Revised: Even for the out-of-class data, both models can find visually similar in-class data with slight variation.

Original: However, PAWS generates predictions with high confidence (high similarity score) while RoPAWS has lower confidence.
Revised: However, PAWS generates predictions with high confidence (high similarity score) while RoPAWS has lower confidence.

Original: For the in-class data, both models generate high confidence predictions and can find the correct labels.
Revised: For the in-class data, both models generate high confidence predictions and can find the correct labels.

Original: [Figure]
Revised: [Figure]

Original: N e a r e s t l a b e l e d d a t a ( P A
Revised: N [Figure] a r e s t l a b e l e d d a t a ( P A

Original: W S )
Revised: W S )

Original: N e a r e s t l a b e l e d d a t a ( R o P
Revised: N e a r e s t l a b e l e d d a t a ( R o P

Original: A W S )
Revised: A W S )

Original: [Figure]
Revised: [Figure]

Original: G A DDITIONAL ANALYSES FOR R O PAWS
Revised: G A DDITIONAL COMPARISON WITH ROBUST S EMI -SL METHODS

Original: Table 8 shows the prediction confidence and AUROC for PAWS and RoPAWS, trained on CIFAR- (+ CIFAR-100) using 25 or 400 labels.
Revised: We conduct additional experiments comparing with robust SSL algorithms, using different in-class datasets: (CIFAR-10, CIFAR-100) and out-of-class datasets (SVHN, Tiny-IN).

Original: AUROC is a popular metric for OOD detection, measuring the area under the receiver operating characteristic (ROC) curve (Bradley, 1997). Precisely, the ROC curve gives the trade-off between true/false positive/negative varying the threshold for OOD detection, and AUROC computes the overall OOD discriminability over thresholds.
Revised: We follow the same configuration of CIFAR-10 experiments in Table 3, except increasing the classes per batch toand unlabeled batch size to 1024 when the in-class dataset is CIFAR-100.

Original: We use thesoft nearest-neighbor classifier for prediction p ( y | x ) and maximum representation similarity to labeled data s ( x ) : = max x l ∈B l z · z l for OOD detection (low s ( x ) = OOD). PAWS predicts
Revised: We used 25 labels per class, set the labeled batch per class as 16, and reported the average and standard deviation over three runs.

Original: CIFAR-10 (in) and CIFAR-100 (out) with a similar confidence level. In contrast, RoPAWS predicts CIFAR-10 with high confidence while assigning low confidence for OOD data. Table
Revised: The baseline results are from the OpenCos (Park et al., 2021) paper.

Original: [Table] H E FFECT OF LABEL PROPAGATION We compare the original soft-NN classifier using only labeled data (Eq. and
Revised: Table 8 shows that RoPAWS consistently outperforms the SOTA robust SSL algorithms under different settings.

Original: the semi-supervised (label propagation) version using both labeled and unlabeled data (Eq.

Original: (52); becomes Eq. (12) at Iter. ∞ ). Table 9 shows the effect of label propagation during inference, using the RoPAWS model trained on CIFAR-10 (+ CIFAR-100), 400 labels.
Revised: In particular, RoPAWS improves the SOTA accuracy with a large margin (+5.8%) in the most challenging scenario of CIFAR-100 (+ Tiny-IN).

Original: We sample a batch of labeled and unlabeled data and evaluate the accuracy of unlabeled (in) data, along with the prediction confidence of unlabeled (in) and unlabeled (out) data. The semi-supervised soft-NN calibrates the prediction and improves the accuracy as epochs go, outperforming the original soft-NN. It converges after 3 iterations, so one may apply the iterative formula instead of the matrix inversion for RoPAWS prediction [Table]
Revised: Semi-SL methods under various scenarios.

Original: [Table] I O PEN M ATCH UNDER SMALL DISTRIBUTION SHIFTS OOD filtering approaches are less effective when in- vs. out-of-domain samples are not discriminative. Since they cannot identify which data is OOD, they conservatively select training samples hence the model is optimal.
Revised: We bold the best results.

Original: Consequently, OpenMatch performs well for CIFAR-10 (+ CIFAR-100) using 400 labels but underperforms when using 25 labels. Table 10 shows that OpenMatch fails to detect OOD samples for 25 labels setup, leading to a meager selected ratio. Also, the OOD filtering approaches underperform on curated datasets since it filters uncertain in-class samples [Table]
Revised: RoPAWS achieves state-of-the-art results in all considered scenarios

Original: [Table]
Revised: [Table]

Original: J C OMPARISON WITH SSKDE
Revised: H C OMPARISON WITH SSKDE

Original: SSKDE (Wang et al., 2009) also compute the posterior probability of labels based on the KDE over the feature space.
Revised: SSKDE (Wang et al., 2009) also compute the posterior probability of labels based on the KDE over the feature space.

Original: However, SSKDE has several critical limitations:
Revised: However, SSKDE has several critical limitations:

Original: • SSKDE requires an ad-hoc feature extractor.
Revised: • SSKDE requires an ad-hoc feature extractor.

Original: SSKDE applies KDE to the precomputed features, unlike RoPAWS jointly learning the representation and the KDE generative classifier.
Revised: SSKDE applies KDE to the precomputed features, unlike RoPAWS jointly learning the representation and the KDE generative classifier.

Original: The ability to learn good representations is  the core difference between RoPAWS and pre-deep-learning approaches like SSKDE.
Revised: The ability to learn good representations is one of the main differences between RoPAWS and pre-deeplearning approaches like SSKDE.

Original: • SSKDE is infeasible for large-scale data.
Revised: • SSKDE is infeasible for large-scale data.

Original: Since SSKDE is a nonparametric method, it computes the kernel matrix between all training data.
Revised: Since SSKDE is a nonparametric method, it computes the kernel matrix between all training data.

Original: Thus, the training cost of SSKDE is O ( N 3 ) (or O ( N 2 ) with approximation), where N is the number of training data.
Revised: Thus, the training cost of SSKDE is O ( N 3 ) , where N is the number of training data.

Original: In contrast, RoPAWS only computes the kernel matrix within the mini-batches, reducing the training cost to O ( N ∗ M 3 ) , where M is the size of the mini-batch and M ≪ N .
Revised: In contrast, RoPAWS only computes the kernel matrix within the mini-batches, reducing the training cost to O ( N ∗ M 3 ) , where M is the size of the mini-batch and M ≪ N .

Original: This enables the training under arbitrarily large data.
Revised: This enables the training under arbitrarily large data.

Original: Due to the limitations listed above, SSKDE cannot be applied to real-world large-scale datasets without significantmethod modification.
Revised: Due to the limitations listed above, SSKDE cannot be applied to real-world large-scale datasets without significant modification.

Original: In addition, RoPAWS has multiple technical novelties over SSKDE:
Revised: In addition, RoPAWS has multiple technical novelties over SSKDE:

Original: • The target problem is different.
Revised: • The target problem is different.

Original: The main goal of RoPAWS is robust semi-supervised learning under uncurated data.
Revised: The main goal of RoPAWS is robust semi-supervised learning under uncurated data.

Original: Thus, we propose in-domain prior to handling out-of-distribution (OOD) samples.
Revised: Thus, we propose in-domain prior to handling out-of-distribution (OOD) samples.

Original: In contrast, SSKDE aims for the curated setup and does not address the overconfidence issue of OOD samples.
Revised: In contrast, SSKDE aims for the curated setup and does not address the overconfidence issue of OOD samples.

Original: In addition, SSKDE mainly targets the video annotation problem and only conducts small-scale experiments, while RoPAWS verifies its effectiveness on realistic large-scale image benchmarks.
Revised: In addition, SSKDE mainly targets the video annotation problem and only conducts small-scale experiments, while RoPAWS verifies its effectiveness on realistic large-scale image benchmarks.

Original: • The definition of kernel is different.
Revised: • The definition of kernel is different.

Original: RoPAWS discovers that PAWS is implicitly modeling KDE with cosine similarity kernels, where the kernel definition comes from the design of recent selfsupervised learning methods.
Revised: RoPAWS discovers that PAWS is implicitly modeling KDE with cosine similarity kernels, where the kernel definition comes from the design of recent selfsupervised learning methods.

Original: In contrast, SSKDE uses Gaussian or Exponential kernels, which may not fit with deep learning techniques.
Revised: In contrast, SSKDE uses Gaussian or Exponential kernels, which may not fit with deep learning techniques.

Original: • The formulation and required assumption are different.
Revised: • The formulation and required assumption are different.

Original: RoPAWS leverages the class-balanced sampling of PAWS to simplify the formula, canceling out the denominator of p ( x | y ) in the Bayes rule.
Revised: RoPAWS leverages the class-balanced sampling of PAWS, canceling out the denominator of p ( x | y ) in the Bayes rule.

Original: SSKDE considers a different assumption that p ( y | x ) isgiven by a linear combination of ˆ p ( y | x ) and ground-truth labels (if exists), where ˆ p ( y | x ) is again a function of p ( y | x ) and kernels.
Revised: SSKDE considers a different assumption that p ( y | x ) is a linear combination of ˆ p ( y | x ) and ground-truth labels (if exists), where ˆ p ( y | x ) is again a function of p ( y | x ) and kernels.

Original: As a result, the final prediction formula of SSKDE in Eq.
Revised: As a result, the final prediction formula of SSKDE in Eq.

Original: (14) of their paper and RoPAWS in Eq.
Revised: (14) of their paper and RoPAWS in Eq.

Original: (11) are different.
Revised: (11) are different.

Original: Despite the fundamental limitations of SSKDE, we provide anadditional experiment comparing RoPAWS and SSKDE for the readers’ interest.
Revised: Despite the fundamental limitations of SSKDE, we provide an experiment comparing RoPAWS and SSKDE.

Original: Since SSKDE requires an ad-hoc feature extractor, we run SSKDE upon PAWS and RoPAWS backbones, in addition to a randomly initialized linear layer of dimension 3x32x32 → 128.
Revised: Since SSKDE requires an ad-hoc feature extractor, we run SSKDE upon PAWS and RoPAWS backbones, in addition to a randomly initialized linear layer of dimension 3x32x32 → 128.

Original: We follow the default configuration of SSKDE: use the Gaussian kernel with gamma =
Revised: We follow the default configuration of SSKDE: use the Gaussian kernel with gamma =

Original: 1/ { # of features }12 , weight for the linear interpolation t = 0 .
Revised: 1/ { # of features } , weight for the linear interpolation t = 0 .

Original: 9 , and number of iterations M = 40 .
Revised: 9 , and number of iterations M = 40 .

Original: Table 11 shows the test accuracy models on CIFAR-10 using 25 labels per class.
Revised: Table 9 shows the test accuracy models on CIFAR-10 using 25 labels per class.

Original: First, one can see that the quality of the feature extractor mostly decides the final accuracy.
Revised: First, one can see that the quality of the feature extractor mostly decides the final accuracy.

Original: We remark that RoPAWS learns a good representation, which is a clear advantage over SSKDE.
Revised: We remark that RoPAWS learns a good representation, which is a clear advantage over SSKDE.

Original: Second, the Soft-NN classifier (default of PAWS/RoPAWS) performs better than the SSKDE classifier.
Revised: Second, the Soft-NN classifier (default of PAWS/RoPAWS) performs better than the SSKDE classifier.

Original: Indeed, the representation of PAWS/RoPAWS is optimized for the Soft-NN classifier.
Revised: Indeed, the representation of PAWS/RoPAWS is optimized for the Soft-NN classifier.

Original: The results confirm that RoPAWS is empirically better than SSKDE.
Revised: The results confirm that RoPAWS is empirically better than SSKDE.

Original: However, exploring different designs of generative classifiers (e.g., extending SSKDE to learn representations jointly) would be an interesting future research direction.
Revised: However, exploring different designs of generative classifiers (e.g., extending SSKDE to learn representations jointly) would be an interesting future research direction.

Original: [Table]
Revised: [Table]

Original: K I MPROVEMENTS ON CURATED DATA
Revised: I I MPROVEMENTS ON CURATED DATA

Original: RoPAWS improves PAWS for curated data since it refines the pseudo-labels via label propagation.
Revised: RoPAWS improves PAWS for curated data since it refines the pseudo-labels via label propagation.

Original: This benefit is more significant when the original pseudo-label of PAWS is inaccurate, e.g., only a few labels are available.
Revised: This benefit is more significant when the original pseudo-label of PAWS is inaccurate, e.g., only a few labels are available.

Original: As a result, RoPAWS gives a large gain (+2.3%) for CIFAR-10 usinglabels.
Revised: As a result, RoPAWS gives a large gain (+2.3%) for CIFAR-10 usinglabels.

Original: However, the gap saturates when using more labels, e.g., 250 labels for CIFAR-10 or 10K (1%) or 100K (10%) labels for ImageNet.
Revised: However, the gap saturates when using more labels, e.g., 250 labels for CIFAR-10 or 10K (1%) or 100K (10%) labels for ImageNet.

Original: Table 12 shows the test accuracy of models trained on
Revised: Table 10 shows the test accuracy of models trained on

Original: CIFAR 10 using 
Revised: CIFAR-10 using 4 labels.

Original: We report the average and standard deviation over five runs.
Revised: We report the average and standard deviation over five runs.

Original: RoPAWS highly improves (+3.0%) PAWS in this more challenging scenario.
Revised: RoPAWS highly improves (+3.0%) PAWS in this more challenging scenario.

Original: [Table]
Revised: [Table]

Original: We provide additional analyses verifying the benefit of RoPAWS.
Revised: We provide additional analyses verifying the benefit of RoPAWS.

Original: Specifically, RoPAWS reduces confirmation bias (Arazo et al., 2020) and improves calibration (Loh et al., 2022) of pseudo-labels.
Revised: Specifically, RoPAWS reduces confirmation bias (Arazo et al., 2020) and improves calibration (Loh et al., 2022) of pseudo-labels.

Original: Recall that lowering wrong predictions (i.e., less confirmation bias) and understanding right confidence (i.e., better calibration) play a crucial role in label propagation.
Revised: Recall that lowering wrong predictions (i.e., less confirmation bias) and understanding right confidence (i.e., better calibration) play a crucial role in label propagation.

Original: First, we verify that label propagation refines the pseudo-labels and reduces confirmation bias (i.e., improves correctness).
Revised: First, we verify that label propagation refines the pseudo-labels and reduces confirmation bias (i.e., improves correctness).

Original: To this end, we present the accuracy of pseudo-labels over label propagation, following the same setup of Appendix H. Table 13 shows the test accuracy of RoPAWS trained on curated CIFAR-10 using 25 labels, using different epochs of checkpoints (total: 600) and different soft nearest-neighbor (Soft-NN) classifiers.
Revised: To this end, we present the accuracy of pseudo-labels over label propagation, following the same setup of Appendix K. Table 11 shows the test accuracy of RoPAWS trained on curated CIFAR-10 using 25 labels, using different epochs of checkpoints (total: 600) and different soft nearest-neighbor (Soft-NN) classifiers.

Original: Iter. 0 means the Soft-NN classifier only uses labeled data, and Iter.
Revised: Iter. 0 means the Soft-NN classifier only uses labeled data, and Iter.

Original: k ≥ 1 means the iterations of label propagation (using both labeled and unlabeled data) where k ≥ ∞ is the fixed-point solution.
Revised: k ≥ 1 means the iterations of label propagation (using both labeled and unlabeled data) where k = ∞ is the fixed-point solution.

Original: The pseudo-label becomes more accurate as iteration goes on.
Revised: The pseudo-label becomes more accurate as iteration goes on.

Original: Also, note that the effect of label refinement is more impactful in early epochs, where the pseudo-labels are inaccurate.
Revised: Also, note that the effect of label refinement is more impactful in early epochs, where the pseudo-labels are inaccurate.

Original: This correction makes RoPAWS converge to better optima.
Revised: This correction makes RoPAWS converge to better optima.

Original: [Table]
Revised: [Table]

Original: Second, we compare the calibration of PAWS and RoPAWS classifiers.
Revised: Second, we compare the calibration of PAWS and RoPAWS classifiers.

Original: To this end, we measure expected calibration error (ECE) (Loh et al., 2022).
Revised: To this end, we measure expected calibration error (ECE) (Loh et al., 2022).

Original: We apply the TorchMetric implementationand use the default 15 bins.
Revised: We apply the TorchMetric implementationand use the default 15 bins.

Original: Table 14 shows the ECE (lower is better) of models trained on CIFAR10 using 25 labels.
Revised: Table 12 shows the ECE (lower is better) of models trained on CIFAR10 using 25 labels.

Original: RoPAWS gives better calibration than PAWS.
Revised: RoPAWS gives better calibration than PAWS.

Original: [Table]
Revised: [Table]

Revised: J A DDITIONAL ANALYSES FOR R O PAWS

Revised: Table 13 shows the prediction confidence and AUROC for PAWS and RoPAWS, trained on CIFAR10 (+ CIFAR-100) using 25 or 400 labels.

Revised: AUROC is a popular metric for OOD detection, measuring the area under the receiver operating characteristic (ROC) curve (Bradley, 1997).

Revised: Precisely, the

Original: L N ON - COLLAPSING REPRESENTATIONS
Revised: ROC curve gives the trade-off between true/false positive/negative varying the threshold for OOD detection, and AUROC computes the overall OOD discriminability over thresholds. We use the soft nearest-neighbor classifier for prediction p ( y | x ) and maximum representation similarity to labeled data s ( x ) : = max x l ∈B l z · z l for OOD detection (low s ( x ) = OOD).

Original:  RoPAWS also favors the non-collapsing property of representations.
Revised: PAWS predicts CIFAR-10 (in) and CIFAR-100 (out) with a similar confidence level. In contrast, RoPAWS predicts CIFAR-10 with high confidence while assigning low confidence for OOD data.

Original: Recall that the proof logic of
Revised: [Table] K E FFECT OF LABEL PROPAGATION We compare the original soft-NN classifier using only labeled data (Eq. and

Revised: the semi-supervised (label propagation) version using both labeled and unlabeled data (Eq.

Original: Proposition 1 in the PAWS paper is as follows:
Revised: (52); becomes Eq. (12) at Iter. ∞ ). Table 14 shows the effect of label propagation during inference, using the RoPAWS model trained on CIFAR-10 (+ CIFAR-100), 400 labels.

Original: When the representation collapses, the distances between representations d ( z, z i ) become identical for all z i in the labeled batch B l . 2.
Revised: We sample a batch of labeled and unlabeled data and evaluate the accuracy of unlabeled (in) data, along with the prediction confidence of unlabeled (in) and unlabeled (out) data. The semi-supervised soft-NN calibrates the prediction and improves the accuracy as epochs go, outperforming the original soft-NN. It converges after 3 iterations, so one may apply the iterative formula instead of the matrix inversion for RoPAWS prediction. [Table]

Original: Then, the prediction p is given by the average of class labels y i , which becomes a uniform distribution due to the class-balanced sampling.
Revised: OOD filtering approaches are less effective when in- vs. out-of-domain samples are not discriminative. Since they cannot identify which data is OOD, they conservatively select training samples hence the model is optimal.

Original: However, the target sharpening makes the target p + not equal to  the uniform distribution; hence, ∇ θ H ( p + , p ) > 0 , and the model escapes from the representation collapse.
Revised: Consequently, OpenMatch performs well for CIFAR-10 (+ CIFAR-100) using 400 labels but underperforms when using 25 labels. Table 15 shows that OpenMatch fails to detect OOD samples for 25 labels setup, leading to a meager selected ratio. Also, the OOD filtering approaches underperform on curated datasets since it filters uncertain in-class samples. [Table]

Original: RoPAWS follows the same logic: the prediction p becomes a uniform distribution when representation collapses, being escaped by step 3 above.
Revised: We believe the probabilistic formulation of RoPAWS, i.e., KDE modeling view over representation space, has the potential to be extended to various scenarios in a principled manner. For example, we assume the prior class distribution p ( y ) is uniform over all classes. One can extend

Original: The difference from PAWS is that now the  prediction is the average of true labels
Revised: RoPAWS to class-imbalanced setup (Kim et al., 2020; Wei et al., 2021) by assuming non-uniform prior. RoPAWS can also handle label shift, i.e., class distribution of labeled and unlabeled data are different. Indeed, the prediction of RoPAWS only assumes the prior on unlabeled data, which can be controlled independently from the prior on labeled data. On the other hand, the entropy mean entropy maximization (me-max) regularization − H (¯ p ) in the objective is identical to minimizing the KL divergence between the mean prediction and the uniform prior.

Revised: This can be extended to minimizing the (posterior) mean prediction and the prior when using non-uniform prior.

Revised: Also, one can extend RoPAWS to open-world setup (Cai et al., 2022; Vaze et al., 2022), which clusters novel classes in addition to predicting known classes.

Original:  y i of  labeledbatch B l andpseudo-labels ˆ y i of unlabeled batch B u .
Revised: Here, one can model conditional densities p ( x | y ) for novel classes y ∈ { C +1 , ..., C + C novel } . Then, finding the maximum likelihood assignment of cluster y for unlabeled data would cluster both labeled and unlabeled data similarly to the infinite Gaussian mixture model (Rasmussen, 1999) but under KDE cluster modeling.

Revised: Finally, one may use the estimated probability, e.g., p ( in | x ) or p ( x | y ) , to choose the most uncertain sample to label for active learning (Settles, 2009).

Revised: Combining semi-supervised learning and active learning under large uncurated data would be important for practical scenarios.

Original: Here, the initial belief of pseudo-labels ˆ y i becomes a uniform distribution by averaging the true labels y i , and averaging this uniform distribution keeps the updated belief uniform after label propagation.
Revised: We remark that the principle of RoPAWS can be generalized beyond PAWS and image classification. For example, the standard softmax classifier can be interpreted as a generative classifier modeling densities with the Gaussian mixture model (GMM) (Lee et al., 2018b). Thus, one can adapt the softmax classifier of previous algorithms, such as FixMatch (Sohn et al., 2020), as a GMM classifier to estimate the uncertainty of samples. However, GMM needs sufficient labels to estimate the variance of each class; hence, less fits for semi-supervised learning.

Revised: One may extend RoPAWS for non-classification tasks such as object detection (Liu et al., 2022) by applying dense self-supervision (Wang et al., 2021) or object-aware augmentation (Mo et al., 2021).

Revised: Also, one may tackle non-image domains, using different augmentations (Lee et al., 2021).