Original: U NIFYING D ISTRIBUTION A LIGNMENT AS A L OSS FOR I MBALANCED S EMI - SUPERVISED L EARNING
Revised: U NIFYING D ISTRIBUTION A LIGNMENT AS A L OSS FOR I MBALANCED S EMI - SUPERVISED L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: While remarkable progress in imbalanced supervised learning has been made recently, less attention has been given to the setting of imbalanced semi-supervised learning (SSL) where not only is a few labeled data provided, but the underlying data distribution can be severely imbalanced.
Revised: While remarkable progress in imbalanced supervised learning has been made recently, less attention has been given to the setting of imbalanced semi-supervised learning (SSL) where not only is a few labeled data provided, but the underlying data distribution can be severely imbalanced.

Original: Recent works require both complicated sampling-based strategies of pseudo-labeled data and distribution alignment of the pseudo-label distribution to accommodate this imbalance.
Revised: Recent works require both complicated sampling-based strategies of pseudo-labeled data and distribution alignment of the pseudo-label distribution to accommodate this imbalance.

Original: We present a novel approach that relies only on a form of a distribution alignment but no sampling strategy where rather than aligning the pseudo-labels during inference, we move the distribution alignment component into the respective cross entropy loss computations for both the supervised and unsupervised losses.
Revised: We present a novel approach that relies only on a form of a distribution alignment but no sampling strategy where rather than aligning the pseudo-labels during inference, we move the distribution alignment component into the respective cross entropy loss computations for both the supervised and unsupervised losses.

Original: This alignment compensates for both imbalance in the data as well as the eventual distributional shift present during evaluation.
Revised: This alignment compensates for both imbalance in the data as well as the eventual distributional shift present during evaluation.

Original: Altogether, this provides a single, uniﬁed strategy that offers both signiﬁcantly reduced training requirements and improved performance across both low and richly labeled regimes and over varying degrees of imbalance.
Revised: Altogether, this provides a single, uniﬁed strategy that offers both signiﬁcantly reduced training requirements and improved performance across both low and richly labeled regimes and over varying degrees of imbalance.

Original: In experiments, we validate the efﬁcacy of our method on SSL variants of CIFAR10-LT, CIFAR100-LT, and ImageNet-127.
Revised: In experiments, we validate the efﬁcacy of our method on SSL variants of CIFAR10-LT, CIFAR100-LT, and ImageNet-127.

Original: On ImageNet-127, our method shows 1.6% accuracy improvement over the previous best method with 80% training time reduction.
Revised: On ImageNet-127, our method shows 1.6% accuracy improvement over the previous best method with 80% training time reduction.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Semi-supervised learning (SSL) uses a large pool of unlabeled data to learn a classiﬁer despite having access to only a small amount of labeled data.
Revised: Semi-supervised learning (SSL) uses a large pool of unlabeled data to learn a classiﬁer despite having access to only a small amount of labeled data.

Original: Recently, techniques have been introduced (Berthelot et al., 2019; 2020; Sohn et al., 2020) which simplify the process while at the same time pushing performance to new levels.
Revised: Recently, techniques have been introduced (Berthelot et al., 2019; 2020; Sohn et al., 2020) which simplify the process while at the same time pushing performance to new levels.

Original: However, these approaches have focused on the cases where the class distributions are balanced for both the labeled and unlabeled data.
Revised: However, these approaches have focused on the cases where the class distributions are balanced for both the labeled and unlabeled data.

Original: At the same time, work within the supervised learning community has shown renewed focus on imbalanced or sometimes long-tailed learning — owing to the fact that most data in the real world is not well-balanced.
Revised: At the same time, work within the supervised learning community has shown renewed focus on imbalanced or sometimes long-tailed learning — owing to the fact that most data in the real world is not well-balanced.

Original: A variety of methodologies for this setting have been introduced (Kang et al., 2020; Menon et al., 2021;
Revised: A variety of methodologies for this setting have been introduced (Kang et al., 2020; Menon et al., 2021;

Original: Ren et al., 2020; Hong et al., 2021).
Revised: Ren et al., 2020; Hong et al., 2021).

Original: For instance, many have observed the bias that ordinary supervised learning techniques suffer from — favoring head classes over the less numerous tail classes .
Revised: For instance, many have observed the bias that ordinary supervised learning techniques suffer from — favoring head classes over the less numerous tail classes .

Original: Kang et al.
Revised: Kang et al.

Original: (2020) show that softmax-based classiﬁers often produce classiﬁcation weights which correlate with class frequency and thus reduces the class-balanced performance of the model.
Revised: (2020) show that softmax-based classiﬁers often produce classiﬁcation weights which correlate with class frequency and thus reduces the class-balanced performance of the model.

Original: Resampling strategies (Chawla et al., 2002; He & Garcia, 2009; Buda et al., 2018; Byrd & Lipton, 2019) (which sample from the pool based on the desired distribution) are often effective as well.
Revised: Resampling strategies (Chawla et al., 2002; He & Garcia, 2009; Buda et al., 2018; Byrd & Lipton, 2019) (which sample from the pool based on the desired distribution) are often effective as well.

Original: By shifting from sampling the data distribution for the majority of training to a more class-balanced regime at the end, one can learn a good representation while mitigating the aforementioned bias at the ﬁnal classiﬁcation layer.
Revised: By shifting from sampling the data distribution for the majority of training to a more class-balanced regime at the end, one can learn a good representation while mitigating the aforementioned bias at the ﬁnal classiﬁcation layer.

Original: In addition, a distributional shift (Hong et al., 2021) can be observed within existing protocols where training occurs on an imbalanced dataset yet evaluation is done with respect to a balanced one.
Revised: In addition, a distributional shift (Hong et al., 2021) can be observed within existing protocols where training occurs on an imbalanced dataset yet evaluation is done with respect to a balanced one.

Original: In this work, we study the combined setting of semi-supervised and imbalanced learning.
Revised: In this work, we study the combined setting of semi-supervised and imbalanced learning.

Original: In particular, we consider both FixMatch (Sohn et al., 2020) and MixMatch (Berthelot et al., 2019) as base semi-supervised learners.
Revised: In particular, we consider both FixMatch (Sohn et al., 2020) and MixMatch (Berthelot et al., 2019) as base semi-supervised learners.

Original: Both employ two losses: a cross entropy loss on the labeled data, and an unsupervised loss that relies on consistency between the classiﬁer outputs among augmented versions of unlabeled examples.
Revised: Both employ two losses: a cross entropy loss on the labeled data, and an unsupervised loss that relies on consistency between the classiﬁer outputs among augmented versions of unlabeled examples.

Original: [Figure]
Revised: [Figure]

Original: available .
Revised: available .

Original: The proposed UDAL trained for only 1 / 6 of the epochs of CReST+ (Wei et al., 2021) further closes the gap in accuracy to a fully supervised baseline and heavily improves on a lower bound supervised baseline trained on 10% of the labels.
Revised: The proposed UDAL trained for only 1 / 6 of the epochs of CReST+ (Wei et al., 2021) further closes the gap in accuracy to a fully supervised baseline and heavily improves on a lower bound supervised baseline trained on 10% of the labels.

Original: Therefore, vulnerability to bias from the imbalance can happen in three ways: the supervised loss itself, the quality of pseudo-labels derived from the classiﬁer on unlabeled examples, and the pseudolabels themselves can bias the classiﬁer even if they are perfectly predicted .
Revised: Therefore, vulnerability to bias from the imbalance can happen in three ways: the supervised loss itself, the quality of pseudo-labels derived from the classiﬁer on unlabeled examples, and the pseudolabels themselves can bias the classiﬁer even if they are perfectly predicted .

Original: Furthermore, conﬁrmation bias within semi-supervised learning is already a worrisome factor even without any imbalance between the classes (Arazo et al., 2020).
Revised: Furthermore, conﬁrmation bias within semi-supervised learning is already a worrisome factor even without any imbalance between the classes (Arazo et al., 2020).

Original: Within the balanced setting, the ReMixMatch (Berthelot et al., 2020) approach to semi-supervised learning introduces strong regularization through distribution alignment (Bridle et al., 1992) (i.e. modifying the prediction by the ratio of the desired distribution to model distribution) to help mitigate this.
Revised: Within the balanced setting, the ReMixMatch (Berthelot et al., 2020) approach to semi-supervised learning introduces strong regularization through distribution alignment (Bridle et al., 1992) (i.e. modifying the prediction by the ratio of the desired distribution to model distribution) to help mitigate this.

Original: Recently CReST (Wei et al., 2021) has shown that distribution alignment also confers beneﬁts in the imbalanced setting and aims to progressively rebalance the distribution of pseudo-labels.
Revised: Recently CReST (Wei et al., 2021) has shown that distribution alignment also confers beneﬁts in the imbalanced setting and aims to progressively rebalance the distribution of pseudo-labels.

Original: Similar to resampling approaches (Kang et al., 2020), imbalanced learning considers a shift from random sampling to class-balanced sampling, CReST attempts to align the pseudo-labels themselves to a more balanced distribution as training progresses.
Revised: Similar to resampling approaches (Kang et al., 2020), imbalanced learning considers a shift from random sampling to class-balanced sampling, CReST attempts to align the pseudo-labels themselves to a more balanced distribution as training progresses.

Original: However, distribution alignment is not the only technique that CReST relies upon to achieve good performance.
Revised: However, distribution alignment is not the only technique that CReST relies upon to achieve good performance.

Original: In addition, CReST requires a generational approach to self-training which accumulates a relatively balanced subset of conﬁdent pseudo-labels to augment the labeled set with as a form of self-training.
Revised: In addition, CReST requires a generational approach to self-training which accumulates a relatively balanced subset of conﬁdent pseudo-labels to augment the labeled set with as a form of self-training.

Original: As the generations proceed, this subset is re-sampled to become more and more balanced.
Revised: As the generations proceed, this subset is re-sampled to become more and more balanced.

Original: Each generation re-initializes the classiﬁer’s network, and therefore, the only “state” retained is through these accumulated pseudo-labels (now treated as ordinary supervised labels).
Revised: Each generation re-initializes the classiﬁer’s network, and therefore, the only “state” retained is through these accumulated pseudo-labels (now treated as ordinary supervised labels).

Original: This process can be extremely costly with respect to training time, and we hypothesize that it is not optimal because it fails to directly address the imbalance in the labeled data.
Revised: This process can be extremely costly with respect to training time, and we hypothesize that it is not optimal because it fails to directly address the imbalance in the labeled data.

Original: Instead, we seek a simpler solution to imbalanced semi-supervised learning through distribution alignment alone.
Revised: Instead, we seek a simpler solution to imbalanced semi-supervised learning through distribution alignment alone.

Original: We ask the question: is this disjoint methodology truly necessary?
Revised: We ask the question: is this disjoint methodology truly necessary?

Original: Can a single, central approach be devised to address imbalance in semi-supervised learning?
Revised: Can a single, central approach be devised to address imbalance in semi-supervised learning?

Original: In this work, as our contributions , we identify an afﬁrmative answer to this question by connecting the ideas of progressive distribution alignment from Berthelot et al.
Revised: In this work, as our contributions , we identify an afﬁrmative answer to this question by connecting the ideas of progressive distribution alignment from Berthelot et al.

Original: Wei et al.
Revised: Wei et al.

Original: (2021) and the method of logit adjustment from fully supervised, imbalanced learning (Menon et al., 2021;
Revised: (2021) and the method of logit adjustment from fully supervised, imbalanced learning (Menon et al., 2021;

Original: Ren et al., 2020; Hong et al., 2021).
Revised: Ren et al., 2020; Hong et al., 2021).

Original: Furthermore, this approach can be implemented with only a few lines of code, has signiﬁcantly reduced training time requirements, and generally outperforms previous work.
Revised: Furthermore, this approach can be implemented with only a few lines of code, has signiﬁcantly reduced training time requirements, and generally outperforms previous work.

Original: Finally, it shows signiﬁcantly better performance characteristics as more labeled data becomes available and readily scales to larger datasets — achieving a 1.6% increase in accuracy compared to the best existing method on ImageNet-127.
Revised: Finally, it shows signiﬁcantly better performance characteristics as more labeled data becomes available and readily scales to larger datasets — achieving a 1.6% increase in accuracy compared to the best existing method on ImageNet-127.

Original:  P REREQUISITES
Revised:  P REREQUISITES

Original: We present background information on both problem settings.
Revised: We present background information on both problem settings.

Original: First, we formally deﬁne the problem setting of Imbalanced Semi-Supervised Learning (SSL).
Revised: First, we formally deﬁne the problem setting of Imbalanced Semi-Supervised Learning (SSL).

Original: Second, we outline the idea of distribution alignment (Berthelot et al., 2020; Wei et al., 2021) to improve pseudo-label quality within both the balanced and imbalanced settings of SSL.
Revised: Second, we outline the idea of distribution alignment (Berthelot et al., 2020; Wei et al., 2021) to improve pseudo-label quality within both the balanced and imbalanced settings of SSL.

Original: We revisit a recent method, CReST (Wei et al., 2021), which also attempts to address imbalanced semi-supervised learning.
Revised: We revisit a recent method, CReST (Wei et al., 2021), which also attempts to address imbalanced semi-supervised learning.

Original: 2. C LASS - IMBALANCED S EMI -S UPERVISED L EARNING
Revised: 2. C LASS - IMBALANCED S EMI -S UPERVISED L EARNING

Original: Semi-supervised learning relies on two sources of data: a labeled set X
Revised: Semi-supervised learning relies on two sources of data: a labeled set X

Original: = { ( x i , y i ) } Ni =1 where each x i is a training example
Revised: = { ( x i , y i ) } Ni =1 where each x i is a training example

Original: and y i is the corresponding target.
Revised: and y i is the corresponding target.

Original: Since classiﬁcation is the focus of this work, we consider y i as a class label within C = { 1 , . . .
Revised: Since classiﬁcation is the focus of this work, we consider y i as a class label within C = { 1 , . . .

Original: , C } with a total number of C classes.
Revised: , C } with a total number of C classes.

Original: In imbalanced learning, we expect varying numbers of training examples across classes.
Revised: In imbalanced learning, we expect varying numbers of training examples across classes.

Original: Therefore, we denote the number of examples in our labeled set corresponding to class c ∈ C as N c such that (cid:80) Cc =1 N c = N .
Revised: Therefore, we denote the number of examples in our labeled set corresponding to class c ∈ C as N c such that (cid:80) Cc =1 N c = N .

Original: We assume that the classes are ordered with respect to frequencies and in a descending manner i.e. N c ≥ N c
Revised: We assume that the classes are ordered with respect to frequencies and in a descending manner i.e. N c ≥ N c

Original: +1 .
Revised: +1 .

Original: It is often useful to characterize the degree of imbalance by the ratio N 1 /N C , and we refer to this as the imbalance ratio of the dataset.
Revised: It is often useful to characterize the degree of imbalance by the ratio N 1 /N C , and we refer to this as the imbalance ratio of the dataset.

Original: We use p data ( y ) and q ( y ) to denote the marginal distributions of the data and the model.
Revised: We use p data ( y ) and q ( y ) to denote the marginal distributions of the data and the model.

Original: When there is no ambiguity, we drop y as p data and q to simplify presentations.
Revised: When there is no ambiguity, we drop y as p data and q to simplify presentations.

Original: Additionally, we have an unlabeled set of examples U = { u i } Mi =1 for which we have no corresponding target.
Revised: Additionally, we have an unlabeled set of examples U = { u i } Mi =1 for which we have no corresponding target.

Original: While we expect that this set is also imbalanced, we additionally make the common assumption (Wei et al., 2021) that it follows the same class distribution and thus shares the same imbalance (ratio) as the labeled set.
Revised: While we expect that this set is also imbalanced, we additionally make the common assumption (Wei et al., 2021) that it follows the same class distribution and thus shares the same imbalance (ratio) as the labeled set.

Original: Finally, an important measure β = NN + M considers the percentage of overall examples that are labeled.
Revised: Finally, an important measure β = NN + M considers the percentage of overall examples that are labeled.

Original: 2. D ISTRIBUTION ALIGNMENT
Revised: 2. D ISTRIBUTION ALIGNMENT

Original: Distribution alignment (DA) was re-introduced in the setting of semi-supervised learning within
Revised: Distribution alignment (DA) was re-introduced in the setting of semi-supervised learning within

Original: ReMixMatch (Berthelot et al., 2020).
Revised: ReMixMatch (Berthelot et al., 2020).

Original: To mitigate the tendencies of semi-supervised learning to suffer from conﬁrmation bias, regularization can be added to the pseudo-label inference step.
Revised: To mitigate the tendencies of semi-supervised learning to suffer from conﬁrmation bias, regularization can be added to the pseudo-label inference step.

Original: In particular, if we assume the labeled and unlabeled data both come from the same distribution p data (although, we do not know particular labels from the unlabeled set), we would expect that our model should produce pseudo-labels that follow the same distribution.
Revised: In particular, if we assume the labeled and unlabeled data both come from the same distribution p data (although, we do not know particular labels from the unlabeled set), we would expect that our model should produce pseudo-labels that follow the same distribution.

Original: This marginal distribution of the model q ( y ) can be estimated by moving average, which we denote as ˆ q ( y ) or ˆ q , as in Berthelot et al.
Revised: This marginal distribution of the model q ( y ) can be estimated by moving average, which we denote as ˆ q ( y ) or ˆ q , as in Berthelot et al.

Original: If we denote our current model’s predictions on unlabeled examples as q ( y | x u ) , these predictions can be re-scaled through dividing by ˆ q and multiplying by p data .
Revised: If we denote our current model’s predictions on unlabeled examples as q ( y | x u ) , these predictions can be re-scaled through dividing by ˆ q and multiplying by p data .

Original: After normalization, we have:
Revised: After normalization, we have:

Original: [Equation]
Revised: [Equation]

Original: In equation 1, we assume element-wise operations beween q ( y | x ) , p data and ˆ q .
Revised: In equation 1, we assume element-wise operations beween q ( y | x ) , p data and ˆ q .

Original: Normalize ( p ) ensures p as a probability distribution which sums to 1.
Revised: Normalize ( p ) ensures p as a probability distribution which sums to 1.

Original: As noted in Wei et al.
Revised: As noted in Wei et al.

Original: (2021), it is not always optimal to align the predictions directly to p data when p data is imbalanced.
Revised: (2021), it is not always optimal to align the predictions directly to p data when p data is imbalanced.

Original: Rather, a smoothed form which (elementwise) exponentiates the distribution by a factor of α before normalization:
Revised: Rather, a smoothed form which (elementwise) exponentiates the distribution by a factor of α before normalization:

Original: [Equation]
Revised: [Equation]

Original: is used instead of p data and found to both regularize the predictions as well as combat bias.
Revised: is used instead of p data and found to both regularize the predictions as well as combat bias.

Original: As α → 0 , this approaches an alignment against a more uniform distribution.
Revised: As α → 0 , this approaches an alignment against a more uniform distribution.

Original: 2. L OGIT A DJUSTMENT
Revised: 2. L OGIT A DJUSTMENT

Original: While DA is clearly applicable to pseudo-label inference during training, it has no direct effect on the labeled portion.
Revised: While DA is clearly applicable to pseudo-label inference during training, it has no direct effect on the labeled portion.

Original: Since a semi-supervised approach relies on both labeled and unlabeled losses, it is critical to address the problem of imbalance at the supervised level as well.
Revised: Since a semi-supervised approach relies on both labeled and unlabeled losses, it is critical to address the problem of imbalance at the supervised level as well.

Original: For this, we examine a popular technique within supervised learning, often known as logit adjustment (Menon et al., 2021), balanced softmax (Ren et al., 2020), or LADE (Hong et al., 2021).
Revised: For this, we examine a popular technique within supervised learning, often known as logit adjustment (Menon et al., 2021), balanced softmax (Ren et al., 2020), or LADE (Hong et al., 2021).

Original: These methods modify the loss computation to compensate for the class imbalance found in the data distribution.
Revised: These methods modify the loss computation to compensate for the class imbalance found in the data distribution.

Original: Notably, when a data distribution is class-imbalanced, we attempt to minimize the classiﬁcation loss with respect to this data distribution.
Revised: Notably, when a data distribution is class-imbalanced, we attempt to minimize the classiﬁcation loss with respect to this data distribution.

Original: However , at evaluation time, we either evaluate on a class-balanced dataset or produce a class-balanced error by averaging the per-class accuracies.
Revised: However , at evaluation time, we either evaluate on a class-balanced dataset or produce a class-balanced error by averaging the per-class accuracies.

Original: This is a shift in distribution which can cause poor performance.
Revised: This is a shift in distribution which can cause poor performance.

Original: Therefore, this shift is integrated into the cross entropy loss:
Revised: Therefore, this shift is integrated into the cross entropy loss:

Original: [Equation]
Revised: [Equation]

Original: (3) where Unif ( C ) is the discrete uniform distribution over C classes, y is the true label of x , f ( x ) is the vector-valued output of the classiﬁer, and p data is the marginal class distribution of the data as a vector.
Revised: (3) where Unif ( C ) is the discrete uniform distribution over C classes, y is the true label of x , f ( x ) is the vector-valued output of the classiﬁer, and p data is the marginal class distribution of the data as a vector.

Original: As elaborated within Menon et al.
Revised: As elaborated within Menon et al.

Original: (2021), this has the effect that instead of optimizing f ( x ) directly, we optimize h ( x ) =
Revised: (2021), this has the effect that instead of optimizing f ( x ) directly, we optimize h ( x ) =

Original: f
Revised: f

Original: ( x ) + log p data which aligns the source distribution of f ( x ) correctly to the uniform class distribution seen during evaluation.
Revised: ( x ) + log p data which aligns the source distribution of f ( x ) correctly to the uniform class distribution seen during evaluation.

Original: Menon et al.
Revised: Menon et al.

Original: (2021) also discusses an inference time procedure which attempts to account for this shift without modiﬁcations to the training procedure.
Revised: (2021) also discusses an inference time procedure which attempts to account for this shift without modiﬁcations to the training procedure.

Original: Since this is “for free”, we include results combined with it in Table 1 as “LA (Inf)” for the inference time procedure.
Revised: Since this is “for free”, we include results combined with it in Table 1 as “LA (Inf)” for the inference time procedure.

Original: [Figure]
Revised: [Figure]

Original: 2. CR E ST
Revised: 2. CR E ST

Original: We brieﬂy examine CReST (Wei et al., 2021), an approach to imbalanced semi-supervised learning using FixMatch and MixMatch as a base semi-supervised learner.
Revised: We brieﬂy examine CReST (Wei et al., 2021), an approach to imbalanced semi-supervised learning using FixMatch and MixMatch as a base semi-supervised learner.

Original: They re-introduce a generational self-training approach where after each generation (usually 64 epochs), conﬁdently pseudo-labeled examples from the unlabeled set U are added to X .
Revised: They re-introduce a generational self-training approach where after each generation (usually 64 epochs), conﬁdently pseudo-labeled examples from the unlabeled set U are added to X .

Original: Ordinarily, this would exacerbate class imbalance.
Revised: Ordinarily, this would exacerbate class imbalance.

Original: Wei et al.
Revised: Wei et al.

Original: (2021) introduce a re-sampling strategy according to a more balanced form of the marginal class distribution of the data to overcome.
Revised: (2021) introduce a re-sampling strategy according to a more balanced form of the marginal class distribution of the data to overcome.

Original: However, it requires to reinitialize and train the network from scratch after each generation.
Revised: However, it requires to reinitialize and train the network from scratch after each generation.

Original: Additionally, for optimal performance, CReST requires distribution alignment to improve pseudolabel quality.
Revised: Additionally, for optimal performance, CReST requires distribution alignment to improve pseudolabel quality.

Original: Speciﬁcally, as deﬁned in Section 2.2, a schedule for α is chosen so that the strength of re-balancing can be altered over the course of training.
Revised: Speciﬁcally, as deﬁned in Section 2.2, a schedule for α is chosen so that the strength of re-balancing can be altered over the course of training.

Original: Given a hyperparameter α min which deﬁnes the minimal value α should take (corresponding to the largest re-balancing of the data distribution), they choose a linear schedule such that for generation iteration t :
Revised: Given a hyperparameter α min which deﬁnes the minimal value α should take (corresponding to the largest re-balancing of the data distribution), they choose a linear schedule such that for generation iteration t :

Original: [Equation]
Revised: [Equation]

Original: (4) where T is the number of generations over the course of training.
Revised: (4) where T is the number of generations over the course of training.

Original: This is referred to as progressive distribution alignment and is essential to strong performance.
Revised: This is referred to as progressive distribution alignment and is essential to strong performance.

Original:  P ROPOSED
Revised:  P ROPOSED

Original: [Equation]
Revised: [Equation]

Original: ETHOD
Revised: ETHOD

Original: We describe our approach, Unifying Distribution Alignment as a Loss (UDAL).
Revised: We describe our approach, Unifying Distribution Alignment as a Loss (UDAL).

Original: First, we connect
Revised: First, we connect

Original: Sections 2.2 (distribution alignment) and 2.3 (logit adjustment).
Revised: Sections 2.2 (distribution alignment) and 2.3 (logit adjustment).

Original: Then, we examine how this allows us to apply the same mitigation to both the supervised and unsupervised components of modern consistency-based SSL methods (Berthelot et al., 2019; Sohn et al., 2020) – providing a uniﬁed manner in which imbalance can be addressed with respect to labeled and unlabeled data.
Revised: Then, we examine how this allows us to apply the same mitigation to both the supervised and unsupervised components of modern consistency-based SSL methods (Berthelot et al., 2019; Sohn et al., 2020) – providing a uniﬁed manner in which imbalance can be addressed with respect to labeled and unlabeled data.

Original: Figureshows the direct comparison between the proposed UDAL and previous work.
Revised: Figureshows the direct comparison between the proposed UDAL and previous work.

Original: 3. A UNIFIED APPROACH
Revised: 3. A UNIFIED APPROACH

Original: We argue that the essence of progressive distribution alignment as seen in Sections 2.2 and 2. is sufﬁcient for strong performance in imbalanced semi-supervised learning if viewed through the lens of logit adjustment from Section 2.3.
Revised: We argue that the essence of progressive distribution alignment as seen in Sections 2.2 and 2. is sufﬁcient for strong performance in imbalanced semi-supervised learning if viewed through the lens of logit adjustment from Section 2.3.

Original: For the sake of presentation, we will use FixMatch (Sohn et al., 2020) as a base semi-supervised learner, but our method is equally applicable to other methods, including MixMatch (Berthelot et al., 2019).
Revised: For the sake of presentation, we will use FixMatch (Sohn et al., 2020) as a base semi-supervised learner, but our method is equally applicable to other methods, including MixMatch (Berthelot et al., 2019).

Original: FixMatch consists of a cross entropy loss on the labeled data and a cross entropy loss on unlabeled data with respect to inferred pseudo-labels:
Revised: FixMatch consists of a cross entropy loss on the labeled data and a cross entropy loss on unlabeled data with respect to inferred pseudo-labels:

Original: [Equation]
Revised: [Equation]

Original: where f ( x ) are the network’s outputs, p label is labeled data distribution, p unlab is the unlabeled data distribution, PL produces hard pseudo-labels for conﬁdent predictions, while u strong and u weak are strong and weakly augmented versions of the unlabeled example u , respectively.
Revised: where f ( x ) are the network’s outputs, p label is labeled data distribution, p unlab is the unlabeled data distribution, PL produces hard pseudo-labels for conﬁdent predictions, while u strong and u weak are strong and weakly augmented versions of the unlabeled example u , respectively.

Original: As described in Section 2.2 and 2.4, applying progressive distribution alignment modiﬁes predictions of unlabeled data by aligning them to a target distribution .
Revised: As described in Section 2.2 and 2.4, applying progressive distribution alignment modiﬁes predictions of unlabeled data by aligning them to a target distribution .

Original: While this aligns the predictions of the unsupervised branch, we must also align the supervised branch.
Revised: While this aligns the predictions of the unsupervised branch, we must also align the supervised branch.

Original: However, distribution alignment modiﬁes the inference process and is unable to affect the known (true) labels of the supervised branch.
Revised: However, distribution alignment modiﬁes the inference process and is unable to affect the known (true) labels of the supervised branch.

Original: Nonetheless, logit adjustment (Menon et al., 2021) demonstrates that logit adjustment can be used to align a supervised classiﬁer to the uniform distribution Unif ( C ) .
Revised: Nonetheless, logit adjustment (Menon et al., 2021) demonstrates that logit adjustment can be used to align a supervised classiﬁer to the uniform distribution Unif ( C ) .

Original: Therefore, we examine whether it can simply replace the usage of distribution alignment during pseudo-label inference.
Revised: Therefore, we examine whether it can simply replace the usage of distribution alignment during pseudo-label inference.

Original: Since we are interested in aligning to an arbitrary target distribution like in DA, we consider a generalization of Equation 3 that aligns to ˜ p
Revised: Since we are interested in aligning to an arbitrary target distribution like in DA, we consider a generalization of Equation 3 that aligns to ˜ p

Original: α t instead of Unif ( C ) :
Revised: α t instead of Unif ( C ) :

Original: [Equation]
Revised: [Equation]

Original: Deﬁning h
Revised: Deﬁning h

Original: ( x ) =
Revised: ( x ) =

Original: f
Revised: f

Original: [Equation]
Revised: [Equation]

Original: x ) + log p data ˜ p αt and solving for f ( x ) , this implies that we’re optimizing:
Revised: x ) + log p data ˜ p αt and solving for f ( x ) , this implies that we’re optimizing:

Original: [Equation]
Revised: [Equation]

Original: or rather, in the softmax space:
Revised: or rather, in the softmax space:

Original: [Equation]
Revised: [Equation]

Original: This is exactly the form of applying distribution alignment to f ( x ) using ˜ p
Revised: This is exactly the form of applying distribution alignment to f ( x ) using ˜ p

Original: α t .
Revised: α t .

Original: Therefore, we hypothesize that not only can we align the supervised branch in this manner, but that we can also apply this exact same form to the unsupervised branch:
Revised: Therefore, we hypothesize that not only can we align the supervised branch in this manner, but that we can also apply this exact same form to the unsupervised branch:

Original: [Equation]
Revised: [Equation]

Original: where we divide by the estimated model’s class distribution ˆ q through moving average, rather than p data since the unsupervised branch is learned from pseudo-labels drawn directly from the model rather than the data distribution.
Revised: where we divide by the estimated model’s class distribution ˆ q through moving average, rather than p data since the unsupervised branch is learned from pseudo-labels drawn directly from the model rather than the data distribution.

Original: This allows us to use the same, uniﬁed approach for each branch by simply replacing the distribution we are aligning from: the marginal class distribution of the data for the supervised branch and the moving average of pseudo-labels for the unsupervised branch.
Revised: This allows us to use the same, uniﬁed approach for each branch by simply replacing the distribution we are aligning from: the marginal class distribution of the data for the supervised branch and the moving average of pseudo-labels for the unsupervised branch.

Original: Altogether, this results in a loss:
Revised: Altogether, this results in a loss:

Original: [Equation]
Revised: [Equation]

Original: where α t is generalized from Equation 4 as:
Revised: where α t is generalized from Equation 4 as:

Original: [Equation]
Revised: [Equation]

Original: to allow for a rate of alignment k and where ˜ p α t is computed as in Equation 2.
Revised: to allow for a rate of alignment k and where ˜ p α t is computed as in Equation 2.

Original: We ﬁnd that sharing this exact same form between branches has performance beneﬁts as outlined in
Revised: We ﬁnd that sharing this exact same form between branches has performance beneﬁts as outlined in

Original: Section 4.5.1.
Revised: Section 4.5.1.

Original: At the same time, unlike distribution alignment, this form does not directly modify the PL inference procedure.
Revised: At the same time, unlike distribution alignment, this form does not directly modify the PL inference procedure.

Original: One might wonder whether a progressive form of alignment is even necessary for the supervised branch.
Revised: One might wonder whether a progressive form of alignment is even necessary for the supervised branch.

Original: We observe heavily degraded performance when progressive alignment is not used in Table 1 under “LA (Sup)”.
Revised: We observe heavily degraded performance when progressive alignment is not used in Table 1 under “LA (Sup)”.

Original: For MixMatch, an L 2 loss function is used between predictions and soft targets .
Revised: For MixMatch, an L 2 loss function is used between predictions and soft targets .

Original: We ﬁnd that applying the same adjustment procedure as in FixMatch but under the L 2 loss continues to work as long as α min is changed to compensate.
Revised: We ﬁnd that applying the same adjustment procedure as in FixMatch but under the L 2 loss continues to work as long as α min is changed to compensate.

Original: We provide pseudocode for this approach in Algorithm 1.
Revised: We provide pseudocode for this approach in Algorithm 1.

Original: This constitutes only a few lines of code with no alterations to the training scheme or time compared to the base FixMatch/MixMatch learner.
Revised: This constitutes only a few lines of code with no alterations to the training scheme or time compared to the base FixMatch/MixMatch learner.

Original: Additionally, no resampling is required.
Revised: Additionally, no resampling is required.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We follow the experimental settings outlined in CReST (Wei et al., 2021).
Revised: We follow the experimental settings outlined in CReST (Wei et al., 2021).

Original: As done in Wei et al.
Revised: As done in Wei et al.

Original: (2021), we make the assumption that the marginal class distributions of labeled and unlabeled data are equal.
Revised: (2021), we make the assumption that the marginal class distributions of labeled and unlabeled data are equal.

Original: We show the performance of UDAL over long-tailed versions of CIFAR10 and
Revised: We show the performance of UDAL over long-tailed versions of CIFAR10 and

Original: CIFAR100, as well as CIFAR10 in the “DARP” (Kim et al., 2020) regime.
Revised: CIFAR100, as well as CIFAR10 in the “DARP” (Kim et al., 2020) regime.

Original: Finally, we perform large-
Revised: Finally, we perform large-

Original: [Table]
Revised: [Table]

Original: data:
Revised: data:

Original: [Table]
Revised: [Table]

Original: distribution of labeled data # p_model
Revised: distribution of labeled data # p_model

Original: [Table]
Revised: [Table]

Original: moving average of model’s predictions on unlabeled data # current_epoch
Revised: moving average of model’s predictions on unlabeled data # current_epoch

Original: (g): current epoch of training (out of max_epoch total) # k
Revised: (g): current epoch of training (out of max_epoch total) # k

Original: [Table]
Revised: [Table]

Original: rate at which a_min is approached # a_min: minimum value of alpha for distribution alignment def compute_adjustment_dist(current_dist):
Revised: rate at which a_min is approached # a_min: minimum value of alpha for distribution alignment def compute_adjustment_dist(current_dist):

Original: factor=
Revised: factor=

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: a_min)
Revised: a_min)

Original: [Table]
Revised: [Table]

Original: current_epoch / max_epoch)
Revised: current_epoch / max_epoch)

Original: [Table]
Revised: [Table]

Original: # normalize ensures the argument sums to
Revised: # normalize ensures the argument sums to

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: labeled) loss on examples
Revised: labeled) loss on examples

Original: x_l
Revised: x_l

Original: # f: classifier
Revised: # f: classifier

Original: [Table] [Table]
Revised: [Table] [Table]

Original: true labels
Revised: true labels

Original: [Table]
Revised: [Table]

Original: SoftmaxCE(
Revised: SoftmaxCE(

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: compute_adjustment
Revised: compute_adjustment

Original: [Table] [Table]
Revised: [Table] [Table]

Original: compute unsupervised
Revised: compute unsupervised

Original: [Table]
Revised: [Table]

Original: unlabeled
Revised: unlabeled

Original: [Table] [Table]
Revised: [Table] [Table]

Original: predicted
Revised: predicted

Original: [Table]
Revised: [Table]

Original: weakly
Revised: weakly

Original: [Table]
Revised: [Table]

Original: confidence
Revised: confidence

Original: [Table] [Table]
Revised: [Table] [Table]

Original: (ours
Revised: (ours

Original: [Table]
Revised: [Table]

Original: Sohn et al.,
Revised: Sohn et al.,

Original: [Table]
Revised: [Table]

Original: w/ DA (Berthelot et al., 2020)
Revised: w/ DA (Berthelot et al., 2020)

Original: ( α
Revised: ( α

Original: [Table]
Revised: [Table]

Original: =
Revised: =

Original: [Table]
Revised: [Table]

Original: 4. CIFAR-LT 4. D ATASET CREATION
Revised: 4. CIFAR-LT 4. D ATASET CREATION

Original: CIFAR10LT and CIFAR100-LT (Cao et al., 2019; Cui et al., 2019) are modiﬁcations of CIFAR10/CIFAR100 with a long-tailed (Zipﬁan) distribution.
Revised: CIFAR10LT and CIFAR100-LT (Cao et al., 2019; Cui et al., 2019) are modiﬁcations of CIFAR10/CIFAR100 with a long-tailed (Zipﬁan) distribution.

Original: Given a desired imbalance ratio γ and some class ordering C i , 1 ≤ i ≤ C , we sample N i examples from the dataset for class C i according to N i = N 1 ∗ γ Ci − 1 C − 1 .
Revised: Given a desired imbalance ratio γ and some class ordering C i , 1 ≤ i ≤ C , we sample N i examples from the dataset for class C i according to N i = N 1 ∗ γ Ci − 1 C − 1 .

Original: For CIFAR10, N 1 = 5000 , C = 10 while N 1 = 500 and C =for CIFAR100.
Revised: For CIFAR10, N 1 = 5000 , C = 10 while N 1 = 500 and C =for CIFAR100.

Original: In order to create suitable splits for semi-supervised learning, labeled subsets are randomly sampled according to β = 10% and 30% .
Revised: In order to create suitable splits for semi-supervised learning, labeled subsets are randomly sampled according to β = 10% and 30% .

Original: Imbalance ratios of γ ∈ { 50 , 100 , 200 } are explored for CIFAR10-LT while γ ∈ { 50 , 100 } for CIFAR100-LT.
Revised: Imbalance ratios of γ ∈ { 50 , 100 , 200 } are explored for CIFAR10-LT while γ ∈ { 50 , 100 } for CIFAR100-LT.

Original: We evaluate with respect to the original, balanced test set which ensures that we can use ordinary accuracy metrics.
Revised: We evaluate with respect to the original, balanced test set which ensures that we can use ordinary accuracy metrics.

Original: 4. T RAINING
Revised: 4. T RAINING

Original: We follow the same model guidelines in CReST (Wei et al., 2021).
Revised: We follow the same model guidelines in CReST (Wei et al., 2021).

Original: Therefore, we use Wide ResNet28 (with width 2) as a backbone.
Revised: Therefore, we use Wide ResNet28 (with width 2) as a backbone.

Original: FixMatch (Sohn et al., 2020) and MixMatch (Berthelot et al., 2019) are used as base semi-supervised learners.
Revised: FixMatch (Sohn et al., 2020) and MixMatch (Berthelot et al., 2019) are used as base semi-supervised learners.

Original: With respect to FixMatch, the unlabeled ratio ( 7 ) and conﬁdence threshold ( 0 . 95 ) are untouched from the original paper’s settings.
Revised: With respect to FixMatch, the unlabeled ratio ( 7 ) and conﬁdence threshold ( 0 . 95 ) are untouched from the original paper’s settings.

Original: The same cosine learning rate schedule is adopted as in Wei et al.
Revised: The same cosine learning rate schedule is adopted as in Wei et al.

Original: Notably different from (Wei et al., 2021), however, is our ability to use vastly shorter training times.
Revised: Notably different from (Wei et al., 2021), however, is our ability to use vastly shorter training times.

Original: Generally, we ﬁnd only 64 epochs ( 2iterations at a batch size of 64) is necessary to reach optimal performance, and therefore all CIFAR models are trained for 64 epochs — constituting a 5x reduction in training time compared to CReST.
Revised: Generally, we ﬁnd only 64 epochs ( 2iterations at a batch size of 64) is necessary to reach optimal performance, and therefore all CIFAR models are trained for 64 epochs — constituting a 5x reduction in training time compared to CReST.

Original: Like CReST (Wei et al., 2021), we ﬁnd it useful to train MixMatch models slightly longer and use a less aggressive α min , therefore, MixMatch models are trained for 128 epochs and use α min = 0 .
Revised: Like CReST (Wei et al., 2021), we ﬁnd it useful to train MixMatch models slightly longer and use a less aggressive α min , therefore, MixMatch models are trained for 128 epochs and use α min = 0 .

Original: Apart from those of the base semi-supervised learner, our method introduces only two hyperparameters: α min which controls the ﬁnal strength of the re-balancing for ˜ p
Revised: Apart from those of the base semi-supervised learner, our method introduces only two hyperparameters: α min which controls the ﬁnal strength of the re-balancing for ˜ p

Original: α t and k , which controls the rate at which we approach α min .
Revised: α t and k , which controls the rate at which we approach α min .

Original: We use α min = 0 .
Revised: We use α min = 0 .

Original: 10 and k = 2 which allows most of training to be aligned to p data and spend the very last stages of training aligning to a relatively balanced class distribution.
Revised: 10 and k = 2 which allows most of training to be aligned to p data and spend the very last stages of training aligning to a relatively balanced class distribution.

Original: This is supported by other work (Kang et al., 2020) which empirically ﬁnds that a when training long-tailed, fully-supervised models, the bulk of training should be done with respect to random sampling, and only a small amount of time of class-balanced sampling is necessary near the end of training.
Revised: This is supported by other work (Kang et al., 2020) which empirically ﬁnds that a when training long-tailed, fully-supervised models, the bulk of training should be done with respect to random sampling, and only a small amount of time of class-balanced sampling is necessary near the end of training.

Original: We provide an ablation of these hyperparameters in Section 4.5.2.
Revised: We provide an ablation of these hyperparameters in Section 4.5.2.

Original: Training is carried out over 5 random folds of the data.
Revised: Training is carried out over 5 random folds of the data.

Original: We report ﬁnal test accuracy along with standard deviation using the exponential moving average of the model’s parameters.
Revised: We report ﬁnal test accuracy along with standard deviation using the exponential moving average of the model’s parameters.

Original: We use the same underlying codebase, written in TensorFlow (Abadi et al., 2015), and data splits as CReST as recommended in (Oliver et al., 2018) to prevent framework or dataset dependent uncertainty in performance.
Revised: We use the same underlying codebase, written in TensorFlow (Abadi et al., 2015), and data splits as CReST as recommended in (Oliver et al., 2018) to prevent framework or dataset dependent uncertainty in performance.

Original: 4. R ESULTS 4.4. CIFAR
Revised: 4. R ESULTS 4.4. CIFAR

Original: CIFAR10-LT and CIFAR100-LT results are summarized in Table 2.
Revised: CIFAR10-LT and CIFAR100-LT results are summarized in Table 2.

Original: Across the board, we ﬁnd
Revised: Across the board, we ﬁnd

Original: UDAL is competitive with CReST+ (Wei et al., 2021) and is only outperformed in a single setting.
Revised: UDAL is competitive with CReST+ (Wei et al., 2021) and is only outperformed in a single setting.

Original: Notably, we ﬁnd that CReST+ especially struggles in the more label rich regimes.
Revised: Notably, we ﬁnd that CReST+ especially struggles in the more label rich regimes.

Original: In particular, despite a heavy imbalance of γ ∈ { 100 , 200 } , UDAL signiﬁcantly outpeforms CReST+ when β
Revised: In particular, despite a heavy imbalance of γ ∈ { 100 , 200 } , UDAL signiﬁcantly outpeforms CReST+ when β

Original: = 30% for both CIFAR10-LT (up to a 4 point increase) and CIFAR100-LT (up to a 1.7 point increase).
Revised: = 30% for both CIFAR10-LT (up to a 4 point increase) and CIFAR100-LT (up to a 1.7 point increase).

Original: We hypothesize that CReST+ is unable to efﬁciently use more labeled data since it only weakly addresses imbalance in the supervised branch.
Revised: We hypothesize that CReST+ is unable to efﬁciently use more labeled data since it only weakly addresses imbalance in the supervised branch.

Original: The imbalance in the supervised branch can only be indirectly alleviated by adding pseudo-labeled to the dataset under some amount of resampling.
Revised: The imbalance in the supervised branch can only be indirectly alleviated by adding pseudo-labeled to the dataset under some amount of resampling.

Original: While this could have a high impact when there is scarce labeled data to begin with, it appears to have much less of an effect when more labeles are available.
Revised: While this could have a high impact when there is scarce labeled data to begin with, it appears to have much less of an effect when more labeles are available.

Original: UDAL, however, directly aligns the supervised branch to a more balanced distribution over the course of training.
Revised: UDAL, however, directly aligns the supervised branch to a more balanced distribution over the course of training.

Original: Not only would this improve the balanced performance of the classiﬁer in the fully supervised setting
Revised: Not only would this improve the balanced performance of the classiﬁer in the fully supervised setting

Original: [Table]
Revised: [Table]

Original: but it additionally aids the feedback loop to provide more balanced pseudo-labels in the unsupervised branch
Revised: but it additionally aids the feedback loop to provide more balanced pseudo-labels in the unsupervised branch

Original: [Table]
Revised: [Table]

Original: CIFAR10-LT and CIFAR100-LT under a variety of label fractions β and imbalance ratios γ , each averaged over 5 folds.
Revised: CIFAR10-LT and CIFAR100-LT under a variety of label fractions β and imbalance ratios γ , each averaged over 5 folds.

Original: DARP
Revised: DARP

Original: We show results within the DARP (Kim et al., 2020) setting in Table 3.
Revised: We show results within the DARP (Kim et al., 2020) setting in Table 3.

Original: While DARP also produces an imbalanced dataset from CIFAR10, it has slight differences.
Revised: While DARP also produces an imbalanced dataset from CIFAR10, it has slight differences.

Original: Whereas the CReST (Wei et al., 2021) setting assigns every example from the original CIFAR dataset to either the labeled or unlabeled dataset, DARP uses an unlabeled to labeled ratio of 2:1, which may or may not utilize all data points.
Revised: Whereas the CReST (Wei et al., 2021) setting assigns every example from the original CIFAR dataset to either the labeled or unlabeled dataset, DARP uses an unlabeled to labeled ratio of 2:1, which may or may not utilize all data points.

Original: One can consider the DARP setting as a β = 33% but with only 95% of the original dataset.
Revised: One can consider the DARP setting as a β = 33% but with only 95% of the original dataset.

Original: Nonetheless, UDAL continues to signiﬁcantly outperforms CReST+ in all settings and shows the largest gains of all in the most imbalanced settings — capable of taking advantage of the increased amount of labeled data.
Revised: Nonetheless, UDAL continues to signiﬁcantly outperforms CReST+ in all settings and shows the largest gains of all in the most imbalanced settings — capable of taking advantage of the increased amount of labeled data.

Original: 4.4. I MAGE N ET - 
Revised: 4.4. I MAGE N ET -

Original: We additionally provide experimental results on ImageNet-127 in Table 4.
Revised: We additionally provide experimental results on ImageNet-127 in Table 4.

Original: As presented in (Wei et al., 2021), ImageNet-127 is a coarser version of ImageNet (Krizhevsky et al., 2012), containing
Revised: As presented in (Wei et al., 2021), ImageNet-127 is a coarser version of ImageNet (Krizhevsky et al., 2012), containing

Original: [Table]
Revised: [Table]

Original: the same number of examples but with 127 class groupings rather than 1000.
Revised: the same number of examples but with 127 class groupings rather than 1000.

Original: This grouping results in an imbalance ratio of γ = 286 .
Revised: This grouping results in an imbalance ratio of γ = 286 .

Original: We conduct experiments with β = 10% along with all base training settings identical to CReST+, with exception to the duration of training, where we observe that only 1000 epochs ( 1 / 6 the total training time of CReST+) is necessary for convergence (Figure 1).
Revised: We conduct experiments with β = 10% along with all base training settings identical to CReST+, with exception to the duration of training, where we observe that only 1000 epochs ( 1 / 6 the total training time of CReST+) is necessary for convergence (Figure 1).

Original: For
Revised: For

Original: UDAL, we continue to ﬁnd k = 2 to be optimal, however, a less aggressive α min ∈ [0 . 5 , 0 . 6] was found to be more effective in this setting.
Revised: UDAL, we continue to ﬁnd k = 2 to be optimal, however, a less aggressive α min ∈ [0 . 5 , 0 . 6] was found to be more effective in this setting.

Original: Overall, we ﬁnd UDAL provides quite healthy increases in performance over CReST and is only 0 .
Revised: Overall, we ﬁnd UDAL provides quite healthy increases in performance over CReST and is only 0 .

Original: 7 absolute points worse than a fully supervised baseline with access to 100% of the labels.
Revised: 7 absolute points worse than a fully supervised baseline with access to 100% of the labels.

Original: [Table]
Revised: [Table]

Original: w/ DA ( α min = 0 .
Revised: w/ DA ( α min = 0 .

Original: [Table]
Revised: [Table]

Original: w/ UDAL (ours, α min =0 .
Revised: w/ UDAL (ours, α min =

Original: [Table]
Revised: [Table]

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: w/ UDAL (ours, α min =0 .
Revised: w/ UDAL (ours, α min =

Original: [Table]
Revised: [Table]

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: 4. A BLATION STUDIES
Revised: 4. A BLATION STUDIES

Original: We carry out ablation studies for our method using the CIFAR10-LT dataset with γ = 100 and β = 10% .
Revised: We carry out ablation studies for our method using the CIFAR10-LT dataset with γ = 100 and β = 10% .

Original: We plot all graphs with respect to the same scale so that they may be readily compared.
Revised: We plot all graphs with respect to the same scale so that they may be readily compared.

Original: 4.5. R EPLACING CR E ST’ S SAMPLING PROCESS
Revised: 4.5. R EPLACING CR E ST’ S SAMPLING PROCESS

Original: We explore whether it would be sufﬁcient to simply replace CReST’s generational sampling process with the supervised component of our method.
Revised: We explore whether it would be sufﬁcient to simply replace CReST’s generational sampling process with the supervised component of our method.

Original: The essential difference between this and our method would be the usage of distribution alignment within the pseudo-label inference (i.e., this affects the pseudo-labeling process directly) versus enforcing a modiﬁed loss computation without any change to the pseudo-label inference process.
Revised: The essential difference between this and our method would be the usage of distribution alignment within the pseudo-label inference (i.e., this affects the pseudo-labeling process directly) versus enforcing a modiﬁed loss computation without any change to the pseudo-label inference process.

Original: We consider a variety of schedules in Figure 3a.
Revised: We consider a variety of schedules in Figure 3a.

Original: While this approach performs well, and can even slightly outperforms CReST+, it still falls short of UDAL, which uses the same approach for both supervised and unsupervised losses — without the need to alter the pseudo-label inference process.
Revised: While this approach performs well, and can even slightly outperforms CReST+, it still falls short of UDAL, which uses the same approach for both supervised and unsupervised losses — without the need to alter the pseudo-label inference process.

Original: 4.5. S CHEDULE PARAMETERS
Revised: 4.5. S CHEDULE PARAMETERS

Original: Since UDAL uses a form of progressive distribution alignment in a similar way to CReST, it introduces two hyperparameters: k and α min which dictates the extent and rate at which the alignment approaches the uniform distribution.
Revised: Since UDAL uses a form of progressive distribution alignment in a similar way to CReST, it introduces two hyperparameters: k and α min which dictates the extent and rate at which the alignment approaches the uniform distribution.

Original: An ablation of these values on CIFAR10-LT is shown in Figure 3b which supports that a quadratic schedule and relatively low α min are optimal.
Revised: An ablation of these values on CIFAR10-LT is shown in Figure 3b which supports that a quadratic schedule and relatively low α min are optimal.

Original: 4.5. I S A SCHEDULE NECESSARY ?
Revised: 4.5. I S A SCHEDULE NECESSARY ?

Original: We investigate whether a schedule is necessary for good performance.
Revised: We investigate whether a schedule is necessary for good performance.

Original: This is equivalent to a “schedule” with the distribution alignment hyperparameter k = 0 .
Revised: This is equivalent to a “schedule” with the distribution alignment hyperparameter k = 0 .

Original: We include a variety of settings for α min
Revised: We include a variety of settings for α min

Original: 0.5 _min
Revised: 0.5 _min

Original: A cc u r a cy * denotes adding LA at inference k = 1 k = 

Revised: A cc u r a cy * denotes adding LA at inference k = 1 k =

Original: k
Revised: k

Original: =
Revised: =

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: A cc u r a cy k
Revised: A cc u r a cy k

Original: =
Revised: =

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: A cc u r a cy 64 epochs 128 epochs 256 epochs (c
Revised: A cc u r a cy 64 epochs 128 epochs 256 epochs (c

Original: Removing and the duration of training in Figure 3c to ensure we adequately explore the trends in performance.
Revised: Removing and the duration of training in Figure 3c to ensure we adequately explore the trends in performance.

Original: We observe a signiﬁcant loss in performance compared to the progressive version of UDAL presented in the paper.
Revised: We observe a signiﬁcant loss in performance compared to the progressive version of UDAL presented in the paper.

Original: We attribute this to the fact that the supervised branch is immediately pushed to produce a marginal distribution of pseudo-labels that are more balanced.
Revised: We attribute this to the fact that the supervised branch is immediately pushed to produce a marginal distribution of pseudo-labels that are more balanced.

Original: This, however, goes against empirical evidence (Kang et al., 2020) that representations are best learned from data sampled according to the data distribution, even if it is imbalanced.
Revised: This, however, goes against empirical evidence (Kang et al., 2020) that representations are best learned from data sampled according to the data distribution, even if it is imbalanced.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Semi-supervised learning (Grandvalet et al., 2005; Lee, 2013; Laine & Aila, 2017; Berthelot et al., 2019; 2020; Sohn et al., 2020) has recently seen strong advances in performance.
Revised: Semi-supervised learning (Grandvalet et al., 2005; Lee, 2013; Laine & Aila, 2017; Berthelot et al., 2019; 2020; Sohn et al., 2020) has recently seen strong advances in performance.

Original: This can be attributed to the success of pseudo-labeling (Lee, 2013) combined with consistency of predictions (Berthelot et al., 2019; Sohn et al., 2020) among varying types of augmentations of unlabeled data.
Revised: This can be attributed to the success of pseudo-labeling (Lee, 2013) combined with consistency of predictions (Berthelot et al., 2019; Sohn et al., 2020) among varying types of augmentations of unlabeled data.

Original: While supervised learning has progressed signiﬁcantly, a large amount of work has tackled the setting of class imbalanced learning (Kang et al., 2020;
Revised: While supervised learning has progressed signiﬁcantly, a large amount of work has tackled the setting of class imbalanced learning (Kang et al., 2020;

Original: Jamal et al., 2020; Cui et al., 2019; Menon et al., 2021; Ren et al., 2020; Hong et al., 2021; Tang et al., 2020; Khan et al., 2017).
Revised: Jamal et al., 2020; Cui et al., 2019; Menon et al., 2021; Ren et al., 2020; Hong et al., 2021; Tang et al., 2020; Khan et al., 2017).

Original: These range from modiﬁcations to the loss formulation (Jamal et al., 2020; Menon et al., 2021; Ren et al., 2020;
Revised: These range from modiﬁcations to the loss formulation (Jamal et al., 2020; Menon et al., 2021; Ren et al., 2020;

Original: Hong et al., 2021; Khan et al., 2017) to decoupling representation from the classiﬁer (Kang et al., 2020) and even modifying the optimization process itself (Tang et al., 2020).
Revised: Hong et al., 2021; Khan et al., 2017) to decoupling representation from the classiﬁer (Kang et al., 2020) and even modifying the optimization process itself (Tang et al., 2020).

Original: By combining the two previous settings, we consider imbalanced, semi-supervised learning.
Revised: By combining the two previous settings, we consider imbalanced, semi-supervised learning.

Original: Although still relatively unexplored, previous attempts (Hyun et al., 2020; Kim et al., 2020; Wei et al., 2021) to combat imbalance in the semi-supervised setting have been made.
Revised: Although still relatively unexplored, previous attempts (Hyun et al., 2020; Kim et al., 2020; Wei et al., 2021) to combat imbalance in the semi-supervised setting have been made.

Original: Both Hyun et al.
Revised: Both Hyun et al.

Original: Kim et al.
Revised: Kim et al.

Original: (2020) modify changes to the loss formulation of a base semi-supervised learner to combat bias within majority classes, while Wei et al.
Revised: (2020) modify changes to the loss formulation of a base semi-supervised learner to combat bias within majority classes, while Wei et al.

Original: (2021) involves a hybrid, generational approach that progressively aligns pseudo-labels predictions as well as augmenting the labeled set with rebalanced, conﬁdent pseudo-labels.
Revised: (2021) involves a hybrid, generational approach that progressively aligns pseudo-labels predictions as well as augmenting the labeled set with rebalanced, conﬁdent pseudo-labels.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We presented Unifying Distribution Alignment as a Loss (UDAL) which addresses the issue of classimbalance within semi-supervised learning.
Revised: We presented Unifying Distribution Alignment as a Loss (UDAL) which addresses the issue of classimbalance within semi-supervised learning.

Original: By connecting the ideas of progressive distribution alignment to logit adjustment, we provide a loss that can be applied to both the supervised and unsupervised branches rather than previous disjoint approaches.
Revised: By connecting the ideas of progressive distribution alignment to logit adjustment, we provide a loss that can be applied to both the supervised and unsupervised branches rather than previous disjoint approaches.

Original: Furthermore, this approach incurs no additional training time on top of the underlying semi-supervised learner, achieves improved performance across multiple imbalanced settings and datasets, and scales to larger, more realistic datasets like ImageNet — all while requiring only a few lines of code.
Revised: Furthermore, this approach incurs no additional training time on top of the underlying semi-supervised learner, achieves improved performance across multiple imbalanced settings and datasets, and scales to larger, more realistic datasets like ImageNet — all while requiring only a few lines of code.

Original: Future work includes applying
Revised: Future work includes applying

Original: UDAL in other settings, e.g. object detection where both imbalance and missing labels are pervasive.
Revised: UDAL in other settings, e.g. object detection where both imbalance and missing labels are pervasive.

Original: E THICS S TATEMENT
Revised: E THICS S TATEMENT

Original: Our work is concerned with building more data-efﬁcient classiﬁcation models while reducing bias present in the distribution of data in the wild.
Revised: Our work is concerned with building more data-efﬁcient classiﬁcation models while reducing bias present in the distribution of data in the wild.

Original: While our work may be subject to similar claims of fairness as any supervised or partially supervised model, we do not believe it introduces harm and, if anything, attempts to mitigate effects that might be seen in biased datasets
Revised: While our work may be subject to similar claims of fairness as any supervised or partially supervised model, we do not believe it introduces harm and, if anything, attempts to mitigate effects that might be seen in biased datasets

Original: R EPRODUCIBLITY
Revised: R EPRODUCIBLITY

Original: Our work is based off of an open source release of CReST available here.
Revised: Our work is based off of an open source release of CReST available here.

Original: The modiﬁcations we make are self-contained and Pythonic code is provided within the paper.
Revised: The modiﬁcations we make are self-contained and Pythonic code is provided within the paper (e g., Algorithm 1).

Original: All hyperparameters we introduce are documented within the paper.
Revised: All hyperparameters we introduce are documented within the paper.

Revised: A A PPENDIX A. A LIGNMENT OF PSEUDO - LABELS OVER THE COURSE OF TRAINING

Revised: We examine the KL-divergence of the pseudo-labels (as a moving average) over the course of

Revised: CIFAR10-LT training using UDAL.

Revised: [Figure]