Original: Improving Barely Supervised Learning by Discriminating Unlabeled Samples with Super-ClassAnonymous Author(s)
Revised: Improving Barely Supervised Learning by Discriminating Unlabeled Data with Super-Class

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: In semi-supervised learning (SSL), a common practice is to learn consistent information from unlabeled data and discriminative information from labeled datato ensure both the immutability and the separability of the classification model.
Revised: In semi-supervised learning (SSL), a common practice is to learn consistent information from unlabeled data and discriminative information from labeled data to ensure both the immutability and the separability of the classification model.

Original: Existing SSL methods suffer from failures in barely-supervised learning (BSL),where only one or two labels per class are available, as the insufficient labels causethe discriminative information to be difficult or even infeasible to learn.
Revised: Existing SSL methods suffer from failures in barely-supervised learning (BSL), where only one or two labels per class are available, as the insufficient labels cause the discriminative information to be difficult or even infeasible to learn.

Original: To bridgethis gap, we investigate a simple yet effective way to leverage unlabeled samples fordiscriminative learning, and propose a novel discriminative information learningmodule to benefit model training.
Revised: To bridge this gap, we investigate a simple yet effective way to leverage unlabeled data for discriminative learning, and propose a novel discriminative information learning module to benefit model training.

Original: Specifically, we formulate the learning objectiveof discriminative information at the super-class level and dynamically assign different classes into different super-classes based on model performance improvement.
Revised: Specifically, we formulate the learning objective of discriminative information at the super-class level and dynamically assign different categories into different super-classes based on model performance improvement.

Original: On top of this on-the-fly process, we further propose a distribution-based loss tolearn discriminative information by utilizing the similarity between samples andsuper-classes.
Revised: On top of this on-the-fly process, we further propose a distribution-based loss to learn discriminative information by utilizing the similarity between samples and super-classes.

Original: It encourages the unlabeled samples to stay closer to the distributionof their corresponding super-class than those of others.
Revised: It encourages the unlabeled data to stay closer to the distribution of their corresponding super-class than those of others.

Original: Such a constraint is softerthan the direct assignment of pseudo labels, while the latter could be very noisy in BSL.
Revised: Such a constraint is softer than the direct assignment of pseudo labels, while the latter could be very noisy in BSL.

Original: We compare our method with state-of-the-art SSL and BSL methods throughextensive experiments on standard SSL benchmarks.
Revised: We compare our method with state-of-the-art SSL and BSL methods through extensive experiments on standard SSL benchmarks.

Original: Our method can achievesuperior results, e.g. , an average accuracy of 76.76% on CIFAR-10 with merely 1label per class. 
Revised: Our method can achieve superior results, e.g. , an average accuracy of 76.76% on CIFAR-10 with merely 1 label per class. The code is available at https://github .com/GuanGui-nju/SCMatch .

Original:  Introduction
Revised:  Introduction

Original: As a paradigm to reduce the dependency on a large amount of labeled data, semi-supervised learning(SSL) has been widely concerned and utilized [1, 2].
Revised: As a paradigm to reduce the dependency on a large amount of labeled data, semi-supervised learning (SSL) has been widely concerned and utilized [1, 2].

Original: Although existing advanced SSL methods couldachieve outstanding performance even with less than 1% labels on several datasets ( e.g. , CIFAR-10),the labeling process could still be lengthy, especially when there is a large number of object categories,which may preclude the deployment of SSL model in those applications.
Revised: Although existing advanced SSL methods [3, 4, 5, 6] could achieve outstanding performance even with less than 1% labels on several datasets ( e.g. , CIFAR-10), the labeling process could still be lengthy, especially when there is a large number of object categories, which may preclude the deployment of SSL model in those applications.

Original: To tackle this challenge,barely-supervised learning (BSL), a novel paradigm with rising interest [3, 4], has been proposedrecently to explore whether the model can be trained with the extremely scarce label, e.g. , only onelabel per class.
Revised: To tackle this challenge, barely-supervised learning (BSL), a novel paradigm with rising interest [3, 7], has been proposed recently to explore whether the model can be trained with the extremely scarce label, e.g. , only one label per class.

Original: Unfortunately, current state-of-the-art SSL methods cannot well address the label-scarce challengein BSL.

Original: As shown in Figure 1(a), many recent SSL methods can achieve auspicious performancewhen sufficient labeled samples are provided, e.g. , higher than 90% with more than 40 labels on

Original: CIFAR10.

Original: However, these methods will suffer severe performance degeneration when the labelamount is reduced.

Original: For example, when only 10 labels are available on CIFAR10, the test accuracy of

Original: FixMatch will drop sharply by more than 45% compared to that of 40 labels.

Original: In order to explore thereasons for such performance dropping, we then track the predicted class distribution of FixMatch

Original: 4000 label amount      acc
Revised: 4000 label amount      acc

Original: u r ac y ( % ) 
Revised: u r ac y ( % ) model performance

Original: ReMixMatch FixMatch
Revised: ReMixMatch FixMatch

Original: CoMatch LESS our method
Revised: CoMatch LESS our method

Original: [Figure]
Revised: [Figure]

Original: 1 2 3 4 5 6 7 8 9 10 class index 0.
Revised: 1 2 3 4 5 6 7 8 9 10 class index 0.

Original: after 10 epochs
Revised: after 10 epochs

Original: FixMatch our method 1 2 3 4 5 6 7 8 9 10 class index 0.
Revised: FixMatch our method 1 2 3 4 5 6 7 8 9 10 class index 0.

Original: after 100 epochs
Revised: after 100 epochs

Original: FixMatch our method
Revised: FixMatch our method

Original: [Figure]
Revised: [Figure]

Original:  distributions  and  for epochs respectively
Revised: The predicted class distributions of FixMatch and our method after training the model for 10 andepochs, respectively.

Original: Due to the lack of guidance  discriminative information, FixMatch tends to predict all  as the same class while our  alleviates this problem through our  discriminative information module during the training process.
Revised: Due to the lack of guidance from discriminative information, FixMatch tends to predict all samples as the same class, while our method alleviates this problem through our proposed discriminative information learning module.

Revised: Unfortunately, current state-of-the-art SSL methods cannot well address the label-scarce challenge in BSL.

Revised: As shown in Figure 1(a), many recent SSL methods can achieve auspicious performance when sufficient labeled data are provided, e.g. , higher than 90% accuracy with more than 40 labels on CIFAR-10.

Revised: However, these methods will suffer severe performance degeneration when the label amount is reduced.

Original:  As shown in Figure 1(b), FixMatch will end with the model collapseafter training for 100 epochs, i.e. , the model completely cannot distinguish different samples, and allsamples are predicted as a single class.
Revised: For example, when only 10 labels are available on CIFAR-10, the test accuracy of FixMatch will drop sharply by more than 45% compared to that of 40 labels. In order to explore the reasons for such performance dropping, we then track the predicted class distribution of FixMatch during the training process. As shown in Figure 1(b), FixMatch will end with the model collapse after training for 100 epochs, i.e. , the model completely cannot distinguish different samples, and all samples are predicted as a single class.

Original: [Figure]
Revised: [Figure]

Original: To analyze the reasons for above phenomenon, weprefer to investigate SSL models in term of separability and immutability .
Revised: To analyze the reasons for above phenomenon, we prefer to investigate classification models in term of separability and immutability .

Original: Here immutability refers tothe capacity of the model to be robust to perturbations [5, 6].
Revised: Here immutability refers to the capacity of the model to be robust to perturbations.

Original: It can be mathematically expressedas arg max p m ( y | u i ) =
Revised: It can be mathematically expressed as arg max p m ( y | u i ) =

Original: arg max p m ( y | α ( u i )) ,where u i is a sample, y id the model prediction,and α ( · ) is a random perturbation.
Revised: arg max p m ( y | α ( u i )) , where u i is a sample, y is the model prediction, and α ( · ) is a random perturbation.

Original: Separability,on the other hand, refers to the capacity of themodel to differentiate two different  samples, i.e., 
Revised: Separability, on the other hand, refers to the capacity of the model to differentiate two different categories of samples, i.e., arg

Original: [Equation]
Revised: [Equation]

Revised: where u i , u j are sampled from different categories.

Original: Figure 2shows a graphic explanation of these two properties.
Revised: Figure 2 shows a graphic explanation of these two properties.

Original: For SSL  models, in common practice, the immutability is often achieved by learning the consistentinformation of the unlabeled samples, and the separability is achieved by learning the discriminativeinformation of the labeled samples.
Revised: For SSL classification models, in common practice, the immutability is often achieved by learning the consistent information of the unlabeled data, and the separability is achieved by learning the discriminative information of the labeled data.

Original: To achieve good performance, SSL models need to well balancetheir immutability and separability.
Revised: To achieve good performance, SSL models need to well balance their immutability and separability.

Original: However, such a balance is destroyed in BSL.
Revised: However, such a balance is destroyed in BSL.

Original: The insufficientsupervision information from the extremely scarce labels significantly damages the learning forseparability, so that the model performance is dominated by the learning towards immutability.
Revised: The insufficient supervision information from the extremely scarce labels significantly damages the learning for separability, so that the model performance is dominated by the learning towards immutability.

Original: That’swhy the model collapse could be observed when all samples are predicted as one same class.
Revised: That’s why the model collapse could be observed when all samples are predicted as one same class.

Original: Motivated by these observations, in this paper, we aim to enhance the discriminative learning forthe model ’s separability under BSL.
Revised: Motivated by these observations, in this paper, we aim to enhance the discriminative learning for the model’s separability under BSL.

Original: Since the labeled samples are very limited, we explore howto leverage unlabeled samples for additional discriminative supervision.
Revised: Since the labeled data are very limited, we explore how to mine additional discriminative supervision from unlabeled data.

Original: Although without labelinformation, the unlabeled samples could still provide some “latent guidance" to complement theprocess of learning only from labeled samples.
Revised: Although without label information, the unlabeled data could still provide some “latent guidance" to complement the process of learning only from labeled data.

Original: We hereby propose a novel module to dynamicallyform super-classes to “roughly categorize" unlabeled samples, then the discriminative informationis learned by measuring the similarity between samples and super-classes, which is realized by ournewly proposed loss function on the distribution level.
Revised: We hereby propose a novel module to dynamically form super-classes to “roughly categorize" unlabeled samples, then the discriminative information is learned by measuring the similarity between samples and super-classes, which is realized by our newly proposed loss function on the distribution level.

Original: Furthermore, with the improvement of themodel, we gradually form more  super-classes for finer categorization of unlabeled samples, aiming toprovide more fine-grained discriminative information to guide the model training.
Revised: Furthermore, with the improvement of the model, we gradually form more [Figure] super-classes for finer categorization of unlabeled samples, aiming to provide more fine-grained discriminative information to guide the model training.

Original: In a nutshell, our proposed method is a simple yet effective way to mine discriminative informationfrom unlabeled data.
Revised: In a nutshell, our proposed method is a simple yet effective way to mine discriminative information from unlabeled data.

Original: Compared to directly assigning pseudo labels to each sample, which could bevery noisy in BSL, learning the similarity between super-classes and samples is  softer guidance, thusreducing the error risk of pseudo labels.
Revised: Compared to directly assigning pseudo labels to each sample [3, 4], which could be very noisy in BSL, learning the similarity between super-classes and samples is the softer guidance, thus reducing the error risk of pseudo labels.

Original: We evaluate our method on CIFAR-10,CIFAR-100, and
Revised: We evaluate our method on CIFAR-10,

Original:  STL-10, showing that our method outperforms all other SSL and BSL methods by a large margin.
Revised: CIFAR-100, and STL-10, showing that our method outperforms all other SSL and BSL methods by a large margin.

Original: For example, only using one label per class on CIFAR-10, our method successfully avoided theoccurrence of model collapse and achieved an accuracy of 76.76% with a variance of 6.78%.[Figure]
Revised: For example, only using one label per class on CIFAR-10, our method successfully avoided the occurrence of model collapse and achieved an accuracy of 76.76% with a variance of 6.78%.

Original:  Method
Revised:  Method

Original: Similar to the setting of SSL, a labeled set X and an unlabeled set U are also given in BSL.
Revised: Similar to the setting of SSL, a labeled set X and an unlabeled set U are also given in BSL.

Original: X =
Revised: X =

Original: { ( x 1 , y 1 ) , ( x 2 , y 2 ) , . . .
Revised: { ( x 1 , y 1 ) , ( x 2 , y 2 ) , . . .

Original: , ( x n , y n ) } , where y i denotes the label of the i -th labeled sample x i .
Revised: , ( x n , y n ) } , where y i denotes the label of the i -th labeled sample x i .

Original: Eachsample is classified into one of n k classes denoted as { c 1 , c 2 , . . .
Revised: Each sample is classified into one of n k classes denoted as { c 1 , c 2 , . . .

Original: , c n k } .
Revised: , c n k } .

Original: U =
Revised: U =

Original: { u 1 , u 2 , . . .
Revised: { u 1 , u 2 , . . .

Original: , u n } ,where u i denotes i -th unlabeled sample, and typically |X| ≪ |U| .
Revised: , u n } , where u i denotes i -th unlabeled sample, and typically |X| ≪ |U| .

Original: In BSL, a more challenging settingis considered, |X| < 4 n k , where only few labeled samples are available.
Revised: In BSL, a more challenging setting is considered, |X| < 4 n k , where only few labeled data are available.

Original: In the implementation, thesamples are provided on a per batch basis, with a batch of labeled samples B x and unlabeled samples B u .
Revised: In the implementation, the samples are provided on a per batch basis, with a batch of labeled data B x and unlabeled data B u .

Original: Thus, the key to BSL lies in training a robust and stable model by efficiently leveraging theunlabeled samples together with such scarce labeled samples.
Revised: As discussed before, the key to BSL lies in training a robust and stable model by efficiently leveraging the unlabeled data together with such scarce labeled data.

Original: Unlike recent state-of-the-art SSL methods that only encourage consistency regularization on unlabeled data, our method aims to mine consistent and discriminative information from the unlabeleddata simultaneously.
Revised: Unlike recent state-of-the-art SSL methods that only encourage consistency regularization on unlabeled data, our method aims to learn consistent and discriminative information from the unlabeled data simultaneously.

Original: As shown in Figure 3, we construct two modules to leverage unlabeled dataaccordingly, i.e., the consistent information learning module and the discriminative informationlearning module.
Revised: As shown in Figure 3, we construct two modules to leverage unlabeled data accordingly, i.e., the consistent information learning module and the discriminative information learning module.

Revised: In the consistent information learning module, we learn the information from the samples and their corresponding augmented versions, like [3].

Original: In addition, to enforce prediction consistency on unlabeled samples, our methoddevelops the super-class distributions by clustering unlabeled samples within a mini-batch and thenuses them to minimize a novel distribution loss on unlabeled samples.
Revised: While in the discriminative information learning module, we develops the super-class distributions by clustering unlabeled samples within a mini-batch and then uses them to minimize a novel distribution loss on unlabeled samples.

Original: 2.1 Consistent Information Learning Module
Revised: 2.1 Consistent information learning module

Original: Like most consistency-based SSL methods, we encourage the model to output the same predictionson two differently-augmented versions of the same sample.
Revised: Like most consistency-based SSL methods, we encourage the model to output the same predictions on two differently-augmented versions of the same sample.

Original: Specifically, we produce pseudo labelson weakly-augmented samples and use them as training targets for their corresponding stronglyaugmented variants.
Revised: Specifically, we produce pseudo labels on weakly-augmented samples and use them as training targets for their corresponding stronglyaugmented variants.

Original: Of them, the weak augmentation α ( · ) includes standard flip and shift operations,while the strong augmentation strategy A ( · ) consists of RandAugment [7] and CutOut [8].
Revised: Of them, the weak augmentation α ( · ) includes standard flip and shift operations, while the strong augmentation strategy A ( · ) consists of RandAugment [8] and CutOut [9].

Original: Formally, this consistency-based unsupervised loss L con is defined as,
Revised: Formally, this consistency-based unsupervised loss L con is defined as,

Original: [Equation]
Revised: [Equation]

Original: con [Equation] [Equation] where H ( p 1 , p 2 ) denotes the standard cross entropy between p 1 and p 2 , and τ 1 is a pre-definedthreshold to retain only high-confidence pseudo labels.
Revised: where H ( p 1 , p 2 ) denotes the standard cross entropy between p 1 and p 2 , and τ 1 is a pre-defined threshold to retain only high-confidence pseudo labels.

Original: As discussed in [3], τ 1 is commonly set as ahigh value to alleviate the confirmation bias in SSL.
Revised: As discussed in [3], τ 1 is commonly set as a high value to alleviate the confirmation bias in SSL.

Original: 2.2 Discriminative Information Learning Module
Revised: 2.2 Discriminative information learning module

Original: In addition to relying on labeled samples to learn discriminative information, we propose a novelmodule, an on-the-fly learning process to first form super-classes and then exploit the similaritybetween super-classes and samples to improve the model’s separability.
Revised: In addition to relying on labeled data to learn discriminative information, we propose a novel module, an on-the-fly learning process to first form super-classes and then exploit the similarity between super-classes and samples to improve the model’s separability.

Original: One of the most intuitive ways to explore discriminative information is to generate class informationfor unlabeled samples by clustering in the feature space.
Revised: One of the most intuitive ways to explore discriminative information is to generate class information for unlabeled data by clustering in the feature space.

Original: Ideally, samples with similar class attributeswill form a separate cluster so that the model can discriminate the samples from all other n k − 1clusters of samples.
Revised: Ideally, samples of the same category will form a separate cluster so that the model can discriminate the samples from all other n k − 1 clusters of samples.

Original: However, forming such fine-grained clusters carries a considerable risk of errors,especially for tasks with a large number of object categories.
Revised: However, forming such fine-grained clusters carries a considerable risk of errors, especially for tasks with a large number of object categories.

Original: What’s worse, in the early training stage,due to the weak feature extraction ability, the model inevitably produces wrong discriminative information, resulting in severe accumulated errors.
Revised: What’s worse, in the early training stage, due to the weak feature extraction ability, the model inevitably produces wrong discriminative information, resulting in severe accumulated errors.

Original: To properly explore the discriminative informationfor unlabeled data, we propose the following designs,
Revised: To properly explore the discriminative information for unlabeled data, we propose the following designs,

Original: • First, instead of fine-grained clusters, we simplify the clustering task by allowing a cluster to contain multiple classes, i.e., a super-class cluster.
Revised: • First, instead of fine-grained clusters, we simplify the clustering task by allowing a cluster to contain multiple categories, i.e., a super-class cluster.

Original: In this way, the discriminative information is relatively weakened but more robust to clustering errors.
Revised: In this way, the discriminative information is relatively weakened but more robust to clustering errors.

Original: • Second, it can still be noisy to adopt the super-class label as training targets for unlabeled data.
Revised: Second, it can still be noisy to adopt the super-class label as training targets for unlabeled data.

Original: Therefore, we tend to utilize the similarity between each sample and the super- classes rather than explicitly assign the training targets for unlabeled data.
Revised: Therefore, we tend to utilize the similarity between each sample and the super-classes rather than explicitly assign the training targets for unlabeled data.

Original: Concretely

Original: , our method encourages theclass prediction of each unlabeled sample to stay closer to the  class  distribution of its
Revised: Concretely, our method encourages the unlabeled samples to stay closer to the predicted class probability distribution of their corresponding super-class than those of others.

Original: super-classcompared to the

Original: distributions of other super-classes

Original: Such

Original: a smoothing

Original: way

Original: can better

Original: tolerate

Original:  the inaccurate prediction of a single image as well as potential clustering errors
Revised: Such a smoothing way can better tolerate the inaccurate prediction of a single sample as well as potential clustering errors.

Original: • Third, althoughusing a small value of K is beneficial in the early training stages, the discriminated information carried by super-classes becomes insufficient when the model’s separability is improved.
Revised: • Third, although the discriminative information provided by the coarse-grained clusters is robust, it will be insufficient when the model’s separability is improved.

Original: Thus we proposed the progressive construction of super-classes to gradually increase the clusteringnumber so that our discriminative information learning module can adapt to the model evolutionduring the training course.
Revised: Thus we propose the progressive construction of super-classes to gradually increase the clustering number so that our discriminative information learning module can adapt to the model evolution during the training process.

Original: When the cluster number is small, each super-class provides moremoderate discriminative information, called a high-level super-class.
Revised: When the cluster number is small, each super-class provides more moderate discriminative information, called a low-level super-class.

Original: In contrast, a large clusternumber can enforce each super-class to abstract more concrete information, and we call it alow-level super-class.
Revised: In contrast, a large cluster number can enforce each super-class to abstract more concrete information, and we call it a high-level super-class.

Original: Super-Class Representation
Revised: Super-class representation

Original: As shown in Figure 3, we employ standard K-Means on these features z wi of weakly-augmentedsamples within a mini-batch.
Revised: As shown in Figure 3, we employ standard K-Means on these features z wi of weakly-augmented samples within a mini-batch.

Original: With a given target number K of super-classes, these features aregathered into K clusters, and each cluster is denoted by C k , k = 1 , 2 , . . .
Revised: With a given target number K of super-classes, these features are gathered into K clusters, and each cluster is denoted by C k , k = 1 , 2 , . . .

Original: , K .
Revised: , K .

Original: Each super-class canthen be represented by the mean distribution of all the samples it contains.
Revised: Each super-class can then be represented by the mean distribution of all the samples it contains.

Original: Given unlabeled sampleu i , and its predicted class probability distribution p m ( y | α ( u i )) and p m ( y |A
Revised: Given unlabeled sample u i , and its predicted class probability distribution p m ( y | α ( u i )) and p m ( y |A

Original: ( u i )) , the super-classdistribution q k for each super-class C k can be calculated by, 
Revised: ( u i )) , the super-class distribution q k for each super-class C k can be calculated by, [Equation] with u

Original: [Equation]
Revised: [Equation]

Original: In this way, the super-class distribution can represent the distribution characteristics of the classesit contains so that it can be well discriminated from other super-classes.
Revised: In this way, the super-class distribution can represent the distribution characteristics of the categories it contains so that it can be well discriminated from other super-classes.

Original: As shown in the lowerhalf of Figure 3, in the automobile-and-airplanesuper-class super-class, it is possibly not easy todetermine the exact class for a single sample.
Revised: As shown in the lower half of Figure 3, in the automobile-and-airplane super-class, it is possibly not easy to determine the exact category for a single sample.

Original: However, we can find that the sample in this super-classshould be closer to the super-class distribution of the automobile-and-airplane super-class compared to those of other super-classes.
Revised: However, we can find that the sample in this super-class should be closer to the super-class distribution of the automobile-and-airplane super-class compared to those of other super-classes.

Original: Additionally, the super-class is more robust to the noisy samples.
Revised: Additionally, the super-class is more robust to the noisy samples.

Original: Forsamples likely to be misclassified ( e.g. , samples inside the dashed box), their negative impact on thesuper-class distribution is well suppressed by other correctly classified samples.
Revised: For samples likely to be misclassified ( e.g. , samples inside the dashed box), their negative impact on the super-class distribution is well suppressed by other correctly classified samples.

Original: Discriminative distribution Loss
Revised: Discriminative distribution loss

Original: Let q ik denote the super-class distribution of the cluster that contains the unlabeled image u i withu i ∈ C k .

Original: To distinguish the image from other super-classes, this image is supposed to be moresimilar to its corresponding super-class  distribution.
Revised: To distinguish the sample from other super-classes, this sample is supposed to be more similar to its corresponding super-class on distribution.

Original: Inspired by [9, 10], we design a contrastivelike distribution loss to distinguish the sample from other super-classes.
Revised: Inspired by [10, 11], we design a contrastivelike distribution loss to distinguish the sample from other super-classes.

Original: Formally, this auxiliarydistribution loss is,
Revised: Formally, this auxiliary distribution loss is,

Original: [Equation]
Revised: [Equation]

Original: where T is a common temperature parameter.
Revised: where T is a common temperature parameter, and u i is assigned to super-class C k .

Original: Like in Equation 1, we adopt a parameter τ 2 tocontrol the learned unlabeled samples.
Revised: Like in Equation 1, we adopt a parameter τ 2 to control the learned unlabeled data.

Original: As mentioned before, the similarity between samples andsuper-classes is a weak constraint, so we are conditioned to use a lower threshold to learn moresamples.
Revised: As mentioned before, the similarity between samples and super-classes is a weak constraint, so we are conditioned to use a lower threshold to learn more samples.

Original: We provide an empirical value via extensive ablation studies.
Revised: We provide an empirical value via extensive ablation studies.

Original: Notice that we computegradients only on strongly augmented samples.
Revised: Notice that we compute gradients only on strongly-augmented samples.

Original: Progressive Super-Class Construction
Revised: Progressive super-class construction

Original: Although small K reduces the clustering error, it comes at the cost that the learned discriminantinformation is limited.
Revised: Although small K reduces the clustering error, it comes at the cost that the learned discriminant information is limited.

Original: Assuming the extreme case, when K = 1 , the amount of information is 0because all samples belong to one super-class, the model will not discriminate against any samples.
Revised: Assuming the extreme case, when K = 1 , the amount of information isbecause all samples belong to one super-class, the model will not discriminate against any samples.

Original: With this point of view, we propose the progressive construction of super-class to adapt to the modelevolution during training.
Revised: With this point of view, we propose the progressive construction of super-class to adapt to the model evolution during training.

Original: That is, when the model is not well trained at the beginning, we use asmall K to form the coarser super-classes to ease the clustering task and thus attain relatively reliablediscriminative guidance.
Revised: That is, when the model is not well trained at the beginning, we use a small K to form the coarser super-classes to ease the clustering task and thus attain relatively reliable discriminative guidance.

Original: When the model is better trained, to avoid the training of the model beingstagnant due to the limitation of discriminative information, we gradually increase K to provideenhanced discriminative guidance.
Revised: When the model is better trained, to avoid the training of the model being stagnant due to the limitation of discriminative information, we gradually increase K to provide enhanced discriminative guidance.

Original: In practice, a daunting challenge is that we do not know the most appropriate number of super-classfor the training samplesset without prior knowledge.
Revised: In practice, a daunting challenge is that we do not know the most appropriate number of super-class for the training samples without prior knowledge.

Original: To this end, we design a dichotomous methodand set the value range of K by:
Revised: To this end, we design a dichotomous method and set the value range of K by:

Original: [Equation]
Revised: [Equation]

Original: The above formula restricts the value of K based on the principle of dichotomy so that frequentchanges of K can be avoided.
Revised: The above formula restricts the value of K based on the principle of dichotomy so that frequent changes of K can be avoided.

Original: Especially when there are many classes in the sample set ( e.g. ,100classes on CIFAR-100), it would be tedious and pointless to learn all K values.
Revised: Especially when there are many classes in the sample set ( e.g. , classes on CIFAR-100), it would be tedious and pointless to learn all K values.

Original: Furthermore, toensure that the clustering task with different K can be performed for a certain period, we adopt alinear-step growth strategy to adjust K dynamically:
Revised: Furthermore, to ensure that the clustering task with different K can be performed for a certain period, we adopt a linear-step growth strategy to adjust K dynamically:

Original: [Equation]
Revised: [Equation]

Original: where t and t s denote the value of the current iteration and the total number of iterations, respectively.α ∈ (0 , 1) and it controls the growth rate of K .
Revised: where t and t s denote the value of the current iteration and the total number of iterations, respectively. α ∈ (0 , 1) and it controls the growth rate of K .

Original: With this clustering task, K super-classes aredynamically formed at each iteration.
Revised: With this clustering task, K super-classes are dynamically formed at each iteration.

Original: 2.3 Total Loss

Original: Similar to most SSL methods, the supervised loss for a batch of labeled samples B x is obtained by astandard cross-entropy loss,

Original: [Equation]

Original: In summary, the total loss in our method is,

Original: [Equation]

Original: (7) where λ con and λ dis are the weights of L con and L dis , respectively.

Original: The full algorithm is provided in

Original: Algorithm 1.

Original: Input : Labeled batch B x = { ( x i , y i ) } , unlabeled batch B u = { u i } , weak augmentation strategy α ( · ) , strong augmentation strategy A ( · )
Revised: Input : Labeled batch B x = { ( x i , y i ) } , unlabeled batch B u = { u i } , weak augmentation strategy α ( · ) , strong augmentation strategy A ( · )

Original: Parameter : threshold τ 1 , τ 2 , temperature T , loss weight λ con , λ dis 1: compute L sup = 1 | B x | (cid:80) | B x | i
Revised: Parameter : threshold τ 1 , τ 2 , temperature T , loss weight λ con , λ dis 1: compute L sup = 1 | B x | (cid:80) | B x | i

Original: =
Revised: =

Original: 1 H ( p m ( y | α ( u i )) , y i ) : 2: for t ← 1 to t s do 3: for u
Revised: 1 H ( p m ( y | α ( u i )) , y i ) : 2: for t ← 1 to t s do 3: for u

Original: i ∈
Revised: i ∈

Original: B u do 4: z wi =
Revised: B u do 4: z wi =

Original: Encoder( α ( u i ) ) // record features of weakly augmented samples.
Revised: Encoder( α ( u i ) ) // record features of weakly augmented samples.

Original: 5: p m ( y | α ( u i )) , p m ( y |A
Revised: 5: p m ( y | α ( u i )) , p m ( y |A

Original: ( u i )) // compute prediction of α ( u i ) and A ( u i ) .
Revised: ( u i )) // compute prediction of α ( u i ) and A ( u i ) .

Original: 6: end for 7:
Revised: 6: end for 7:

Original: L con = 1 | B u | (cid:80) | B u | i =1 

Revised: L con = 1 | B u | (cid:80) | B u | i =1

Original: max( p m ( y | α ( u i ))) ≥ τ 1 ) H ( p m ( y | α
Revised: max( p m ( y | α ( u i ))) ≥ τ 1 ) H ( p m ( y | α

Original: ( u i )) ,
Revised: ( u i )) ,

Original: p m ( y |A
Revised: p m ( y |A

Original: ( u i ))) 8: update K .
Revised: ( u i ))) 8: update K .

Original: 9: form super-classes by K-Means( K, z wi ).
Revised: 9: form super-classes by K-Means( K, z wi ).

Original: 10: q k = 1 / | C k | (cid:80) | C k | i =1 p m ( y | α ( u i )) , ∀ u i ∈ C k 11:
Revised: 10: q k = 1 / | C k | (cid:80) | C k | i =1 p m ( y | α ( u i )) , ∀ u i ∈ C k 11:

Original: L dis = − 1 | B u | (cid:80) | B u | i =1 1 (max( p m ( y | α ( u i ))) ≥ τ 2 ) log exp( p m ( y |A
Revised: L dis = − 1 | B u | (cid:80) | B u | i =1 1 (max( p m ( y | α ( u i ))) ≥ τ 2 ) log exp( p m ( y |A

Original: ( u i )) · q k /T )
Revised: ( u i )) · q k /T )

Original: (cid:80)
Revised: (cid:80)

Original: Kj =1 exp( p m ( y |A
Revised: Kj =1 exp( p m ( y |A

Original: ( u i )) · q j /T ) 12: minimizing the total loss L =
Revised: ( u i )) · q j /T ) 12: minimizing the total loss L =

Original: L sup + λ con L con + λ dis L dis .
Revised: L sup + λ con L con + λ dis L dis .

Original: 13: end for
Revised: 13: end for

Revised: 2.3 Total Loss

Revised: Similar to most SSL methods, the supervised loss for a batch of labeled data B x is obtained by a standard cross-entropy loss,

Original: Experiments
Revised: [Equation] In summary, the total loss in our method is, [Equation] where λ con and λ dis are the weights of L con and L dis , respectively.

Original:  In this section, we validate the effectiveness of our proposedour method by conducting experimentson widely-used SSL benchmark datasets: CIFAR-10, CIFAR-100 [11], and STL-10 [12].
Revised: The full algorithm is provided in Algorithm 1.  Experiments In this section, we validate the effectiveness of our proposed method by conducting experiments on widely-used SSL benchmark datasets: CIFAR-10, CIFAR-100 [12], and STL-10 [13].

Original: 3.1 Implementation Details
Revised: 3.1 Implementation details

Original: To face the challenge of BSL, we randomly sample 1 or 2 labels for each class on these data sets.
Revised: To face the challenge of BSL, we randomly sample 1 or 2 labels for each class on these data sets.

Original: We adopt "WideResNet-28-2" and "WideResNet-28-8" [13] as the backbone for CIFAR-10 and
Revised: We adopt "WideResNet-28-2" and "WideResNet-28-8" [14] as the backbone for CIFAR-10 and

Original: CIFAR-100, respectively, while using "ResNet18" [14] for STL-10.
Revised: CIFAR-100, respectively, while using "ResNet18" [15] for STL-10.

Original: For the consistent informationlearned model, we follow the same setting with [3], where τ 1 = 0 .
Revised: For the consistent information learned module, we follow the same setting with [3], where τ 1 = 0 .

Original: | B x | = 64 , | B u | = 7 | B x | .
Revised: | B x | = 64 , | B u | = 7 | B x | .

Original: And for the discriminative information learned model, we set T = 1 , τ 2 = 0 .
Revised: And for the discriminative information learned module, we set T = 1 , τ 2 = 0 .

Original: In addition, Sincethe essence of the three losses of L sup , L con , L dis is in the form of cross entropy, it’s prefer to set L con =
Revised: In addition, Since the essence of the three losses of L sup , L con , L dis is in the form of cross entropy, it’s prefer to set λ con =

Original: L dis = 1 to further reduce of hyperparameters.
Revised: λ dis = 1 to further reduce of hyperparameters.

Original: Therefore, our proposed our method only has K as one hyperparameter.

Original: For CIFAR-10 and STL-10task, we set the K ∈ {⌊ n k / 3 ⌋ , n k / 2 , n k } = { 3 , 5 , 10 } .
Revised: For CIFAR-10 and STL-10 task, we set the K ∈ {⌊ n k / 3 ⌋ , n k / 2 , n k } = { 3 , 5 , 10 } .

Original: For CIFAR-100, considering that the samplesof each cluster should be sufficient, we set the K ∈ { n k / 20 , n k / 10 , n k / 5 } = { 5 , 10 , 20 } .
Revised: For CIFAR-100, considering that the samples of each cluster should be sufficient, we set the K ∈ { n k / 20 , n k / 10 , n k / 5 } = { 5 , 10 , 20 } .

Original: In fact,the specific numerical setting of K has little effect on the model performance, and more analysis andexperiments about K will be discussed in the later ablation experiments.
Revised: In fact, the specific numerical setting of K has little effect on the model performance, and more analysis and experiments about K will be discussed in the later ablation experiments.

Revised: [Table]

Original: The model is trained with a total of 2 20 iterations, and the K increased in the first 30% iterations.
Revised: The model is trained with a total of 2 20 iterations, and the K increased in the first 30% iterations.

Original: Weuse an exponential moving average with a decay rate of 0.999 to test our model and repeat the sameexperiment for five runs with different seeds to report the mean accuracy.
Revised: We use an exponential moving average with a decay rate of 0.999 to test our model and repeat the same experiment for five runs with different seeds to report the mean accuracy.

Original: 3.2 Baseline Methods
Revised: 3.2 Baseline methods

Original: First, FixMatch [3], Dash[15], CoMatch [16],  FlexMatch [17] are the advanced semi-supervisedmodels in recent years, and we compare these methods under the challenge of barely-supervisedlearning.
Revised: First, FixMatch [3], Dash [16], CoMatch [5], FlexMatch [4] are the advanced semi-supervised models in recent years, and we compare these methods under the challenge of barely-supervised learning.

Original: We also use FixMatch with the distribution alignment (DA).
Revised: We also use FixMatch with the distribution alignment (DA).

Original: SLA [18] and LESS [4] arethe latest models on BSL, and we also use them as our comparison method.
Revised: SLA [6] and LESS [7] are the latest models on BSL, and we also use them as our comparison method.

Original: In addition to this, wealso select some classical semi-supervised methods such as MeanTeacher [19], MixMatch [20] and ReMixMatch [21] for comparison.[Table]
Revised: In addition to this, we also select some classical semi-supervised methods such as MeanTeacher [17], MixMatch [18] and ReMixMatch [19] for comparison.

Original: 3.3 Experimental Results
Revised: 3.3 Experimental results

Original: Performance comparisons .
Revised: Performance comparisons .

Original: In Table 1, we compare the test accuracy of our proposed methodagainst recent SSL and BSL methods.
Revised: In Table 1, we compare the test accuracy of our proposed method against recent SSL and BSL methods.

Original: It can be seen that our results are state-of-the-art in all settings.
Revised: It can be seen that our results are state-of-the-art in all settings.

Original: Especially when there is only one label per class, our method compensates for the shortage of labeledsamples by mining latent discriminative information from unlabeled samples, thus showing enormoussuperiority.
Revised: Especially when there is only one label per class, our method compensates for the shortage of labeled data by mining latent discriminative information from unlabeled data, thus showing enormous superiority.

Original: LESS [4], recent work on BSL, since it generates predictions for samples with lowconfidence and then learns more consistent information, still ignores the learning of discriminativeinformation, it cannot solve the challenges in BSL.
Revised: LESS [7], recent work on BSL, since it generates predictions for samples with low confidence and then learns more consistent information, still ignores the learning of discriminative information, it cannot solve the challenges in BSL.

Original: On CIFAR-10 task with 10 labels, our method achieves the mean accuracy of 76.76%, whichoutperforms other methods by 10%.
Revised: On CIFAR-10 task with 10 labels, our method achieves the mean accuracy of 76.76%, which outperforms other methods by 10%.

Original: On STL-10 task with 10 labels, the recent BSL methods LESSand SLA achieve the accuracy of 64.40% and 65.87%, respectively, while our method achievesthe mean accuracy of 52.52%, which improved nearly by 20%.
Revised: On STL-10 task with 10 labels, the recent BSL methods LESS and SLA achieve the accuracy of 34.25% and 32.38%, respectively, while our method achieves the mean accuracy of 52.51%, which improved nearly by 20%.

Original: For larger dataset CIFAR-100, ourmethod also outperforms other methods by at least 6% when there is 1 label per class.
Revised: For larger dataset CIFAR-100, our method also outperforms other methods by at least 6% when there is 1 label per class.

Original: Besides, wecan see that, regardless of the dataset, the performance of our method when using only 1 label perclass is close to or even exceeds the performance of other methods when using 2 labels per class.
Revised: Besides, we can see that, regardless of the dataset, the performance of our method when using only 1 label per class is close to or even exceeds the performance of other methods when using 2 labels per class.

Original: On
Revised: On

Original: CIFAR-100 task, LESS and SLA achieve the mean accuracy of 39.50% and 38.65% with 100 labels,while our method achieves the mean accuracy of 37.50% with half of the labels they use.
Revised: CIFAR-100 task, LESS and SLA achieve the mean accuracy of 42.50% and 38.65% with 100 labels, while our method achieves the mean accuracy of 37.50% with half of the labels they use.

Original: As mentioned by [3], the quality of very few labeled samples will significantly affect the performanceof the model.
Revised: As mentioned by [3], the quality of very few labeled data will significantly affect the performance of the model.

Original: Taking the CIFAR-10 task with 10 labels as an example, the variance of advanced SSLmethods and BSL methods are all more than 10%, while the variance of our method is only 6.78%.
Revised: Taking the CIFAR-10 task with 10 labels as an example, the variance of advanced SSL methods and BSL methods are all more than 10%, while the variance of our method is only 6.78%.

Original: These results further illustrate that our method can alleviate the dependence on labeled samples bylearning discriminative information from unlabeled samples.
Revised: These results further illustrate that our method can alleviate the dependence on labeled data by learning discriminative information from unlabeled data.

Original: We also find that the technique of distribution alignment, which forces the alignment of probabilitydistributions is still an effective technique under BSL.
Revised: We also find that the technique of distribution alignment, which forces the alignment of probability distributions is still an effective technique under BSL.

Original: Through the result of FixMatch (w/DA), wecan see that DA successfully helped FixMatch improve its performance significantly.
Revised: Through the result of FixMatch (w/DA), we can see that DA successfully helped FixMatch improve its performance significantly.

Original: However, DAis a technique that relies on prior information, and our method does not rely on any prior informationand can achieve better performance than it. 
Revised: However, DA is a technique that relies on prior information, and our method does not rely on any prior information and can achieve better performance than it. acc u r ac y ( % ) 70.23 72.15 68. (a) K 0.

Revised: acc

Revised: u r ac y ( % ) linear-mode exp-mode step-mode (b) α 0.

Original:  [Table]
Revised: acc u r ac y ( % ) (c) τ [Table]

Original: Stability of the model .
Revised: Stability of the model .

Original: First of all, we discuss the phenomenon of model collapse under the BSL challenge.
Revised: First of all, we discuss the phenomenon of model collapse under the BSL challenge.

Original: For a fair comparison, we use the same random seed in eachtrial for FixMatch and our method.
Revised: For a fair comparison, we use the same random seed in each trial for FixMatch and our method.

Original: Sinceonly 1 label per class is available, this SSLmethod that depends on labeled samples tolearn discriminative information would bevolatile.
Revised: Since only 1 label per class is available, this SSL method that depends on labeled data to learn discriminative information would be volatile.

Original: As shown in table 2, we can seethat the performance of FixMatch is extremely unstable, where it can achieve very high accuracy of85 .
Revised: As shown in table 2, we can see that the performance of FixMatch is extremely unstable, where it can achieve very high accuracy of 85 .

Original: 11% when seed = 3 but obtain an extremely low accuracy of 17 . 09% when seed = 4 .
Revised: 11% when seed = 3 but obtain an extremely low accuracy of 17 . 09% when seed = 4 .

Original: Differently,integrating the proposed super-class distribution to provide more discriminative information, our

Original: acc

Original: u r ac y ( % ) 70.23 72.15 68.

Original: (a) K 0.

Original: acc u r ac y ( % ) linear-mode exp-mode step-mode (b) α 0. acc u r ac y ( % ) (c) τ method can successfully alleviate the model collapse: the accuracy exceeded 70% in all experimentsand also exceeded 80% sometimes.Method 40 labels 250 labels
Revised: Differently, integrating the proposed super-class distribution to provide more discriminative information, our method can successfully alleviate the model collapse: the accuracy exceeded 70% in all experiments and also exceeded 80% sometimes.

Original: [Table]
Revised: [Table]

Original: ReMixMatch

Original: [Table]

Original: FixMatch

Original: [Table]

Original: LESS

Original: [Table]

Original: our

Original: [Table]

Original: Performance under SSL settings.
Revised: Performance under SSL settings.

Original: We also analyze our method in standard SSL settings where sufficientlabeled samples are provided.
Revised: We also analyze our method in standard SSL settings where sufficient labeled data are provided.

Original: Asshown in Table 3, we test our methodon CIFAR-10 with 40, 250, and 4000labels.
Revised: As shown in Table [Table] we test our method on CIFAR-10 with 40, 250, andlabels.

Original: It can be seen that whenthe number of labels increases, ourmethod is not SOTA, but the gap withother methods is within 1%.
Revised: It can be seen that when the number of labels increases, our method is not SOTA, but the gap with other methods is within 1%.

Original: It canbe interpreted as these methods using other advanced techniques in thelearning consistent information process.
Revised: It can be interpreted as these methods using other advanced techniques in the learning consistent information process.

Original: For example, FlexMatch [17] and SLA [18] both leverage prior knowledge of class proportions,and CoMatch [16] leverages graph-based contrastive learning, etc.
Revised: For example, FlexMatch [4] and SLA [6] both leverage prior knowledge of class proportions, and CoMatch [5] leverages graph-based contrastive learning, etc.

Original: While we study an independentmodule for solving BSL, so when the labels are enough, our method does not prevail.
Revised: While we study an independent module for solving BSL, so when the labels are enough, our method does not prevail.

Original: However,though our implementation is based on FixMatch [3], the results show that our method can stillimprove slightly when the number of labels is large.
Revised: However, though our implementation is based on FixMatch [3], the results show that our method can still improve slightly when the number of labels is large.

Original: 3.4 Ablation Study
Revised: 3.4 Ablation study

Original: Performance under different strategies of K .
Revised: Performance under different strategies of K .

Original: We explore the effect of K for different strategieson the model: (1) fix-mode, the number of super-class remains constant during model training.
Revised: We explore the effect of K for different strategies on the model: (1) fix-mode, the number of super-class remains constant during model training.

Original: (2)linear-mode, the number of super-class increases linearly during model training.
Revised: (2) linear-mode, the number of super-class increases linearly during model training.

Original: (3)exp-mode, thenumber of super-class increases exponentially.
Revised: (3) exp-mode, the number of super-class increases exponentially.

Original: (4) step-mode, a step-by-step jump growth based onlinear-mode.
Revised: (4) step-mode, a step-by-step jump growth based on linear-mode.

Original: We fixed different K values for experiments in terms of the fixed strategy.
Revised: We fixed different K values for experiments in terms of the fixed strategy.

Original: As shown in Figure 4(a),when K is small, the model tends to outperform the larger K .
Revised: As shown in Figure 4(a), when K is small, the model tends to outperform the larger K .

Original: When K is large, the model in theearly stage does not provide high-quality features to perform the formation of high-level superclasses, so the discriminative information learned has an extremely high risk of error, leading to themodel’s failure.
Revised: When K is large, the model in the early stage does not provide high-quality features to perform the formation of high-level superclasses, so the discriminative information learned has an extremely high risk of error, leading to the model’s failure.

Original: On the other hand, when K is small, our method can learn effective discriminativeinformation from these low-level super-classes.
Revised: On the other hand, when K is small, our method can learn effective discriminative information from these low-level super-classes.

Original: However, as the model performance improves, thislimited discriminative information provided by low-level super-classes can no longer help the modellearn continuously, so the model’s performance will stagnate.
Revised: However, as the model performance improves, this limited discriminative information provided by low-level super-classes can no longer help the model learn continuously, so the model’s performance will stagnate.

Original: It is worth noting that even if we adoptthe fix-mode with K , the model can learn a certain degree of discriminative from the super-class toface the challenge of BSL, and its performance also exceeds other methods.
Revised: It is worth noting that even if we adopt the fix-mode with K , the model can learn a certain degree of discriminative from the super-class to face the challenge of BSL, and its performance also exceeds other methods.

Original: Linear-mode, exp-mode, step-mode can all work well to solve the problem in the fix-mode above,while there is an additional hyperparameter to control the rate of K growth in these modes.
Revised: Linear-mode, exp-mode, step-mode can all work well to solve the problem in the fix-mode above, while there is an additional hyperparameter to control the rate of K growth in these modes.

Original: As shown in Figure 4(b), we conduct experiments for different growth rates, and it turns out that the growthrate of K does not affect the performance of the model too much.
Revised: As shown in Figure 4(b), we conduct experiments for different growth rates, and it turns out that the growth rate of K does not affect the performance of the model too much.

Original: In addition, since this mode canexplore different levels of discriminative information, the performance is significantly better thanthat in the fix-mode.
Revised: In addition, since this mode can explore different levels of discriminative information, the performance is significantly better than that in the fix-mode.

Original: Although the performance of these modes is exceptionally close, we preferstep-mode as it can be more suitable for large data sets, e.g. , it is impractical to increase K from 3 to100 sequentially when we test on CIFAR-100.
Revised: Although the performance of these modes is exceptionally close, we prefer step-mode as it can be more suitable for large data sets, e.g. , it is impractical to increase K from 3 to 100 sequentially when we test on CIFAR-100.

Original: Performance under different τ 2 .
Revised: Performance under different τ 2 .

Original: We investigate 5 different τ 2 values on CIFAR-10 datasets with10 labels.
Revised: We investigate 5 different τ 2 values on CIFAR-10 datasets with 10 labels.

Original: As shown in Figure 4(c), the test performance achieve the best when τ 2  .
Revised: As shown in Figure 4(c), the test performance achieve the best when τ 2 = 0 .

Original: It shows thatappropriately lowering the threshold can learn discriminative information from more samples, therebyhelping model training.
Revised: It shows that appropriately lowering the threshold can learn discriminative information from more samples, thereby helping model training.

Original: However, if the threshold τ 2 is too low, the noise of the sample will increase,which is not conducive to the training of the model.
Revised: However, if the threshold τ 2 is too low, the noise of the sample will increase, which is not conducive to the training of the model.

Original:  Related Work
Revised:  Related Work

Original: Recent popular semi-supervised learning studies can be classified into entropy minimization (ER)based methods and consistency regularization (CR) based methods.
Revised: Recent popular semi-supervised learning studies can be classified into entropy minimization (ER) based methods and consistency regularization (CR) based methods.

Original: Self training is the typicalrepresentative of ER-based methods.
Revised: Self training is the typical representative of ER-based methods.

Original: In these methods [1], the model is first trained on the providedlabeled samples and then used to generate pseudo-labels for unlabeled samples.
Revised: In these methods [1], the model is first trained on the provided labeled data and then used to generate pseudo-labels for unlabeled data.

Original: After that, suchmethods add these unlabeled samples with high-confidence predictions into the labeled set to retrainthe model, repeating this process until all unlabeled samples are involved [22].
Revised: After that, such methods add these unlabeled data with high-confidence predictions into the labeled set to retrain the model, repeating this process until all unlabeled data are involved [20].

Original: Recent studies tend toinvolve more advanced techniques in this framework to enhance the SSL performance.
Revised: Recent studies tend to involve more advanced techniques in this framework to enhance the SSL performance.

Original: [23] introducesmultiple views to provide more robust pseudo-labels.
Revised: [21] introduces multiple views to provide more robust pseudo-labels.

Original: LaSSL [24] and Curriculum Labeling [25]integrate the contrastive learning and curriculum learning techniques to improve the accuracy ofpseudo-labels further.
Revised: LaSSL [22] and Curriculum Labeling [23] integrate the contrastive learning and curriculum learning techniques to improve the accuracy of pseudo-labels further.

Original: As the most widely-used and successful technique in recent SSL methods, CR is the semantics ofa sample should be consistent after data perturbations [26, 19, 20, 27, 28].
Revised: As the most widely-used and successful technique in recent SSL methods, CR is the semantics of a sample should be consistent after data perturbations [24, 17, 18, 25, 26].

Original: FixMatch [3] combinesstrong augmentation technology [7, 8] and the labels of weakly augmented samples with high confidence are used to guide the learning of strong augmented samples.
Revised: FixMatch [3] combines strong augmentation technology [8, 9] and the labels of weakly augmented samples with high confidence are used to guide the learning of strong augmented samples.

Original: Although a major breakthroughhas been made in conventional semi-supervised learning, it still cannot avoid model collapse.
Revised: Although a major breakthrough has been made in conventional semi-supervised learning, it still cannot avoid model collapse.

Original: [15, 17]further dynamically adjusts the confidence threshold based on FixMatch.
Revised: [16, 4] further dynamically adjusts the confidence threshold based on FixMatch.

Original: Although it can learn morelow-confidence samples to improve the performance of the model, it cannot cope with the challengeof lack of discriminative information under BSL.
Revised: Although it can learn more low-confidence samples to improve the performance of the model, it cannot cope with the challenge of lack of discriminative information under BSL.

Original: In the literature, there have been few works on barely-supervised learning.
Revised: In the literature, there have been few works on barely-supervised learning.

Original: FixMatch [3] initiallycame up with the concept of BSL and emphasized that the quality of the labeled samples played acrucial role in the test performance.
Revised: FixMatch [3] initially came up with the concept of BSL and emphasized that the quality of the labeled data played a crucial role in the test performance.

Original: Our experiment results also demonstrate that its testing resultshave a very high variance under the BSL settings.
Revised: Our experiment results also demonstrate that its testing results have a very high variance under the BSL settings.

Original: Recent SLA [18] also achieved better performancein BSL by formulating an optimal transportation problem between samples and labels.
Revised: Recent SLA [6] also achieved better performance in BSL by formulating an optimal transportation problem between samples and labels.

Original: It introducedmany extra hyper-parameters and adopted the Sinkhorn-Knopp algorithm to solve the optimizationproblem approximately.
Revised: It introduced many extra hyper-parameters and adopted the Sinkhorn-Knopp algorithm to solve the optimization problem approximately.

Original: Differently, our method gets rid of complicated operations but can effectivelyimprove the BSL performance.
Revised: Differently, our method gets rid of complicated operations but can effectively improve the BSL performance.

Original: Another work [4] argues that the dilemma in BSL is that there areno pseudo-labels that can be predicted with high confidence, so online deep clustering is used tosupplement the pseudo-labels predicted by the model.
Revised: Another work [7] argues that the dilemma in BSL is that there are no pseudo-labels that can be predicted with high confidence, so online deep clustering is used to supplement the pseudo-labels predicted by the model.

Original: Although more pseudo-labels can be used,it still learns consistent information.
Revised: Although more pseudo-labels can be used, it still learns consistent information.

Original: [29] is a very recent work that uses the coarse-grained classlabels to guide the SSL model.
Revised: [27] is a very recent work that uses the coarse-grained class labels to guide the SSL model.

Original: However, it requires strong prior knowledge about the class hierarchystructure in advance; while we are faced with BSL scenarios without any prior knowledge, even thehierarchy structure may not exist.
Revised: However, it requires strong prior knowledge about the class hierarchy structure in advance; while we are faced with BSL scenarios without any prior knowledge, even the hierarchy structure may not exist.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we analyze the failure of SSL methods in the face of BSL as insufficient discriminativeinformation learning.
Revised: In this paper we analyze the failure of SSL methods in the face of BSL as insufficient discriminative information learning.

Original: To tackle this problem, we design a discriminative learning module to leverageunlabeled samples for additional discriminative supervision.
Revised: To tackle this problem we design a discriminative learning module to leverage unlabeled data for additional discriminative supervision.

Original: In this module, super-classes aredynamically reformed with the model training, and then the discriminative information is learned bymeasuring the similarity between samples and super-classes.
Revised: In this module, super-classes are dynamically reformed with the model training and then the discriminative information is learned by measuring the similarity between samples and super-classes.

Original: We conduct our methods on several SSL benchmarks, and it shows that our method outperforms other methods in 
Revised: We conduct our methods on several SSL benchmarks, and it shows that our method outperforms other methods in BSL.  Acknowledgment

Original: BSL.
Revised: This work was supported by the Science and Technology Innovation 2030 New Generation Artificial Intelligence Major Project (2021ZD0113303), the NSFC Program (62222604, 62206052, 62192783), CAAI-Huawei MindSpore Project (CAAIXSJLJJ-2021-042A), China Postdoctoral Science Foundation Project (2021M690609), Jiangsu Natural Science Foundation Project (BK20210224) and CCF-Lenovo Bule Ocean Research Fund

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contribu- tions and scope?

Original: [Yes](b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes](c) Did you discuss any potential negative societal impacts of your work?
Revised: [Yes] (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes](d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes]2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes](b) Did you include complete proofs of all theoretical results?
Revised: [Yes] (b) Did you include complete proofs of all theoretical results?

Original: [Yes]3.
Revised: [Yes] 3.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimentalresults [Table] [Table] [Table]
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Table] [Table]
Revised: [Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Table] [Table] A Appendixs Proof 1.
Revised: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: Given the number K of super-class, then probability that u i belongs to each super-class is 1 /K . Thinking of each sample u i as a signal source, then the amount of information generated is:
Revised: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Equation]
Revised: [Yes] 4.

Original: then, the amount of information generated by D is: [Equation] To better analyze the reasons for the above dilemma, we prefer to understand the SSL model from the separability 458 and the immutability, which are properties that a healthy model should have. The immutability refers to themodel’s prediction of an sample that is not affected by perturbations [5, 6, 30], and it can be learned from the 460 consistent information.
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: Property 1. immutability.
Revised: [Yes] (b) Did you mention the license of the assets?

Original: We assume D denotes the data set, α denotes a random perturbation ( e.g. , flip, shift, 462 Gauss noise, etc. ), p m ( y | x ) denotes the model’s prediction, then the model is supposed to be satisfied: [Equation]
Revised: [Yes] (c) Did you include any new assets either in the supplemental material or as a URL? [No] (d) Did you discuss whether

Original: However, when the model generates the same output for all data( e.g. , predicting all samples to 0), which is 464 well-satisfied with the immutability, the model is invalid. Therefore, the model also needs to learn discriminativeinformation to be separable to avoid this dilemma. The separability refers to the model that can distinguish 466 samples from different classes [31, 32], and an intuitive way of learning discriminative information is byoptimizing the cross-entropy loss on labeled data [33].
Revised: and how consent was obtained from people whose data you’re us- ing/curating?

Original: Property 2. separability. We assume C x , C y denote the set of samples of different classes, then the model is 469 supposed to be satisfied:
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable informa- tion or offensive content?

Original: [Equation]
Revised: [N/A] 5.

Original: While in the challenge of BSL, there are barely labeled samples to provide explicit discriminative information 471 directly.
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: That is why these CR-based models are prone to collapse under the dominance of consistent information.
Revised: [N/A]