Original: E XPLORING B ALANCED F EATURE S PACES FOR R EPRESENTATION L EARNING
Revised: E XPLORING B ALANCED F EATURE S PACES FOR R EPRESENTATION L EARNING

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Existing self-supervised learning (SSL) methods are mostly applied for training representation models from artiﬁcially balanced datasets ( e.g . ImageNet).
Revised: Existing self-supervised learning (SSL) methods are mostly applied for training representation models from artiﬁcially balanced datasets ( e.g . ImageNet).

Original: It is unclear how well they will perform in the practical scenarios where datasets are often imbalanced w.r.t.
Revised: It is unclear how well they will perform in the practical scenarios where datasets are often imbalanced w.r.t.

Original: the classes.
Revised: the classes.

Original: Motivated by this question, we conduct a series of studies on the performance of self-supervised contrastive learning and supervised learning methods over multiple datasets where training instance distributions vary from a balanced one to a long-tailed one.
Revised: Motivated by this question, we conduct a series of studies on the performance of self-supervised contrastive learning and supervised learning methods over multiple datasets where training instance distributions vary from a balanced one to a long-tailed one.

Original: Our ﬁndings are quite intriguing.
Revised: Our ﬁndings are quite intriguing.

Original: Different from supervised methods with large performance drop, the self-supervised contrastive learning methods perform stably well even when the datasets are heavily imbalanced.
Revised: Different from supervised methods with large performance drop, the self-supervised contrastive learning methods perform stably well even when the datasets are heavily imbalanced.

Original: This motivates us to explore the balanced feature spaces learned by contrastive learning, where the feature representations present similar linear separability w.r.t.
Revised: This motivates us to explore the balanced feature spaces learned by contrastive learning, where the feature representations present similar linear separability w.r.t.

Original: all the classes.
Revised: all the classes.

Original: Our further experiments reveal that a representation model generating a balanced feature space can generalize better than that yielding an imbalanced one across multiple settings.
Revised: Our further experiments reveal that a representation model generating a balanced feature space can generalize better than that yielding an imbalanced one across multiple settings.

Original: Inspired by these insights, we develop a novel representation learning method, called k -positive contrastive learning.
Revised: Inspired by these insights, we develop a novel representation learning method, called k -positive contrastive learning.

Original: It effectively combines strengths of the supervised method and the contrastive learning method to learn representations that are both discriminative and balanced.
Revised: It effectively combines strengths of the supervised method and the contrastive learning method to learn representations that are both discriminative and balanced.

Original: Extensive experiments demonstrate its superiority on multiple recognition tasks. Remarkably, it achieves new state-of-the-art on challenging long-tailed recognition benchmarks.
Revised: Extensive experiments demonstrate its superiority on multiple recognition tasks, including both long-tailed ones and normal balanced ones.

Original: Code and models will be released.
Revised: Code is available at https://github.com/bingykang/BalFeat .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Self-supervised learning (SSL) has been popularly explored as it can learn data representations without requiring manual annotations and offer attractive potential of leveraging the vast amount of unlabeled data in the wild to obtain strong representation models (Gidaris et al., 2018; Noroozi & Favaro, 2016;
Revised: Self-supervised learning (SSL) has been popularly explored as it can learn data representations without requiring manual annotations and offer attractive potential of leveraging the vast amount of unlabeled data in the wild to obtain strong representation models (Gidaris et al., 2018; Noroozi & Favaro, 2016;

Original: He et al., 2020; Chen et al., 2020a; Wu et al., 2018).
Revised: He et al., 2020; Chen et al., 2020a; Wu et al., 2018).

Original: For instance, some recent SSL methods (H ´ enaff et al., 2019; Oord et al., 2018; Hjelm et al., 2018; He et al., 2020) use the unsupervised contrastive loss (Hadsell et al., 2006) to train the representation models by maximizing the instance discriminativeness, which are shown to generalize well across various downstream tasks, and even surpass the supervised learning counterparts in some cases (He et al., 2020; Chen et al., 2020a).
Revised: For instance, some recent SSL methods (H´enaff et al., 2019; Oord et al., 2018; Hjelm et al., 2018; He et al., 2020) use the unsupervised contrastive loss (Hadsell et al., 2006) to train the representation models by maximizing the instance discriminativeness, which are shown to generalize well across various downstream tasks, and even surpass the supervised learning counterparts in some cases (He et al., 2020; Chen et al., 2020a).

Original: Despite the great success, existing SSL methods focus on learning data representations from the artiﬁcially balanced datasets ( e.g . ImageNet (Deng et al., 2009)) where all the classes have similar numbers of training instances.
Revised: Despite the great success, existing SSL methods focus on learning data representations from the artiﬁcially balanced datasets ( e.g . ImageNet (Deng et al., 2009)) where all the classes have similar numbers of training instances.

Original: However in reality, since the classes in natural images follow the
Revised: However in reality, since the classes in natural images follow the

Original: Zipﬁan distribution, the datasets are usually imbalanced and show a long-tailed distribution (Zipf, 1999; Spain & Perona, 2007), i.e ., some classes involving signiﬁcantly fewer training instances than others.
Revised: Zipﬁan distribution, the datasets are usually imbalanced and show a long-tailed distribution (Zipf, 1999; Spain & Perona, 2007), i.e ., some classes involving signiﬁcantly fewer training instances than others.

Original: Such imbalanced datasets are very challenging for supervised learning methods to model, leading to noticeable performance drop (Wang et al., 2017; Mahajan et al., 2018; Zhong et al., 2019).
Revised: Such imbalanced datasets are very challenging for supervised learning methods to model, leading to noticeable performance drop (Wang et al., 2017; Mahajan et al., 2018; Zhong et al., 2019).

Original: Thus several interesting questions arise: How well will SSL methods perform on imbalanced datasets?
Revised: Thus several interesting questions arise: How well will SSL methods perform on imbalanced datasets?

Original: Will the quality of their learned representations deteriorate as the supervised learning methods?
Revised: Will the quality of their learned representations deteriorate as the supervised learning methods?

Original: Or can they perform stably well?
Revised: Or can they perform stably well?

Original: Answering these questions is important for understanding the behavior of SSL in practice.
Revised: Answering these questions is important for understanding the behavior of SSL in practice.

Original: But these questions remain open as no research investigations have been conducted along this direction so far.
Revised: But these questions remain open as no research investigations have been conducted along this direction so far.

Revised: [Figure]

Original: Our work is motivated by the above questions to study the properties of data representations learned with supervised/self-supervised methods in a practical scenario.
Revised: Our work is motivated by the above questions to study the properties of data representations learned with supervised/self-supervised methods in a practical scenario.

Original: We start with two representative[Figure] losses used by these methods, i.e ., the supervised cross-entropy and the unsupervised contrastive losses (Hadsell et al., 2006; Oord et al., 2018), and investigate the classiﬁcation performance of their trained representation models from multiple training datasets where the instance distribution gradually varies from a balanced one to a long-tailed one.
Revised: We start with two representative losses used by these methods, i.e ., the supervised cross-entropy and the unsupervised contrastive losses (Hadsell et al., 2006; Oord et al., 2018), and investigate the classiﬁcation performance of their trained representation models from multiple training datasets where the instance distribution gradually varies from a balanced one to a long-tailed one.

Original: We surprisingly observe that, different from the ones learned from supervised cross-entropy loss where performance drops quickly, the representation models learned from the unsupervised contrastive loss perform stably well, no matter how much the training instance distribution is skewed to be imbalanced.
Revised: We surprisingly observe that, different from the ones learned from supervised cross-entropy loss where performance drops quickly, the representation models learned from the unsupervised contrastive loss perform stably well, no matter how much the training instance distribution is skewed to be imbalanced.

Original: Such a stark difference between the two representation learning methods drives us to explore why SSL performs so stably.
Revised: Such a stark difference between the two representation learning methods drives us to explore why SSL performs so stably.

Original: We ﬁnd that using the contrastive loss can obtain representation models generating a balanced feature space that has similar separability (and classiﬁcation performance) for all the classes, as illustrated in Figure 1.
Revised: We ﬁnd that using the contrastive loss can obtain representation models generating a balanced feature space that has similar separability (and classiﬁcation performance) for all the classes, as illustrated in Figure 1.

Original: Such a balanced property of the feature spaces from SSL is intriguing and provides a new perspective to understand the behavior of SSL methods.
Revised: Such a balanced property of the feature spaces from SSL is intriguing and provides a new perspective to understand the behavior of SSL methods.

Original: We dig deeper into its beneﬁts via a systematic study.
Revised: We dig deeper into its beneﬁts via a systematic study.

Original: In particular, since a pre-trained representation model is often used as initialization for downstream tasks (He et al., 2020; Newell & Deng, 2020; H ´ enaff et al., 2019), we evaluate and compare the generalization ability of the models that produce feature spaces of different balanced levels (or ‘balancedness’).
Revised: In particular, since a pre-trained representation model is often used as initialization for downstream tasks (He et al., 2020; Newell & Deng, 2020; H ´ enaff et al., 2019), we evaluate and compare the generalization ability of the models that produce feature spaces of different balanced levels (or ‘balancedness’).

Original: We ﬁnd that a more balanced model tends to generalize better across a variety of settings , including the out-of-distribution recognition as well as the cross-domain and cross-task applications.
Revised: We ﬁnd that a more balanced model tends to generalize better across a variety of settings , including the out-of-distribution recognition as well as the cross-domain and cross-task applications.

Original: These studies imply that feature space balancedness is an important but often neglected factor for learning high-quality representations.
Revised: These studies imply that feature space balancedness is an important but often neglected factor for learning high-quality representations.

Original: Inspired by the above insights, we propose a new representation learning method, the k -positive constrastive learning, which inherits the strength of constrastive learning in learning balanced feature spaces and meanwhile improves the feature spaces’ discriminative capability.
Revised: Inspired by the above insights, we propose a new representation learning method, the k -positive constrastive learning, which inherits the strength of constrastive learning in learning balanced feature spaces and meanwhile improves the feature spaces’ discriminative capability.

Original: Speciﬁcally, different from the contrastive learning methods lacking semantic discriminativeness, the proposed k -positive constrastive method leverages the available instance semantic labels by taking k instances of the same label with the anchor instance to embed semantics into the contrastive loss.
Revised: Speciﬁcally, different from the contrastive learning methods lacking semantic discriminativeness, the proposed k -positive constrastive method leverages the available instance semantic labels by taking k instances of the same label with the anchor instance to embed semantics into the contrastive loss.

Original: As such, it can learn representations with desirable balancedness and discriminativeness (Figure 1).
Revised: As such, it can learn representations with desirable balancedness and discriminativeness (Figure 1).

Original: Extensive experiments and analyses clearly demonstrate its superiority over the supervised learning and latest contrastive learning methods (He et al., 2020) for various recognition tasks. In particular, the proposed method establishes new state-of-the-art on the ImageNet-LT (Liu et al., 2019) and iNaturalist (iNatrualist, 2018) benchmarks for long-tailed recognition.
Revised: Extensive experiments and analyses clearly demonstrate its superiority over the supervised learning and latest contrastive learning methods (He et al., 2020) for various recognition tasks, including visual recognition in both long-tailed setting ( e.g ., ImageNet-LT, iNaturalist) and balanced setting.

Original: This work makes the following important observations and contributions.
Revised: This work makes the following important observations and contributions.

Original: (1) We present the ﬁrst systematic studies on the performance of self-supervised contrastive learning on imbalanced datasets which are helpful to understanding the merits and limitations of SSL in practice.
Revised: (1) We present the ﬁrst systematic studies on the performance of self-supervised contrastive learning on imbalanced datasets which are helpful to understanding the merits and limitations of SSL in practice.

Original: (2) Our studies reveal an intriguing property of the model trained by contrastive learning—the model can robustly learn balanced feature spaces—that has never been discussed before.
Revised: (2) Our studies reveal an intriguing property of the model trained by contrastive learning—the model can robustly learn balanced feature spaces—that has never been discussed before.

Original: (3) Our empirical analysis demonstrates that learning balanced feature spaces beneﬁts the generalization of representation models and offer a new perspective for understanding deep model generalizability.
Revised: (3) Our empirical analysis demonstrates that learning balanced feature spaces beneﬁts the generalization of representation models and offer a new perspective for understanding deep model generalizability.

Original: (4) We develop a new method to explicitly pursue balanced feature spaces for representation learning and it outperforms the popular cross-entropy and contrastive losses based methods.
Revised: (4) We develop a new method to explicitly pursue balanced feature spaces for representation learning and it outperforms the popular cross-entropy and contrastive losses based methods.

Original: We believe our ﬁndings and the novel k -positive contrastive method are inspiring for future research on representation learning.
Revised: We believe our ﬁndings and the novel k -positive contrastive method are inspiring for future research on representation learning.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: Self-supervised learningSelf-supervised learning is a form of unsupervised learning.
Revised: Self-supervised learning is a form of unsupervised learning.

Original: Recently there has been a surge of self-supervised data representation learning methods developed to allevi- ate the demand for manual annotations by mining free supervision information through speciﬁcally designed loss functions and pretext tasks.
Revised: Recently there has been a surge of self-supervised data representation learning methods developed to alleviate the demand for manual annotations by mining free supervision information through speciﬁcally designed loss functions and pretext tasks.

Original: The contrastive loss measures the similarities of sample pairs in a feature space and is at the core of several recent SSL methods (Chen et al., 2020a;b; He et al., 2020; Chen et al., 2020c).
Revised: The contrastive loss measures the similarities of sample pairs in a feature space and is at the core of several recent SSL methods (Chen et al., 2020a;b; He et al., 2020; Chen et al., 2020c).

Original: Adversarial losses that measure the distribution difference are also exploited for self-supervised representation learning (Donahue et al., 2016; Doersch & Zisserman, 2017).
Revised: Adversarial losses that measure the distribution difference are also exploited for self-supervised representation learning (Donahue et al., 2016; Doersch & Zisserman, 2017).

Original: A wide range of pretext tasks have been developed including image inpainting (Jenni & Favaro, 2018; Pathak et al., 2016), image colorization (Larsson et al., 2016; 2017), context prediction (Doersch et al., 2015), jigsaw puzzles (Carlucci et al., 2019; Noroozi & Favaro, 2016; Wei et al., 2019), rotation prediction (Gidaris et al., 2018).
Revised: A wide range of pretext tasks have been developed including image inpainting (Jenni & Favaro, 2018; Pathak et al., 2016), image colorization (Larsson et al., 2016; 2017), context prediction (Doersch et al., 2015), jigsaw puzzles (Carlucci et al., 2019; Noroozi & Favaro, 2016; Wei et al., 2019), rotation prediction (Gidaris et al., 2018).

Original: Though very successful, the behavior of SSL largely remains a mystery.
Revised: Though very successful, the behavior of SSL largely remains a mystery.

Original: Recently Wang & Isola (2020) analyze contrastive learning from the perspective of uniformity and alignment of learned representations.
Revised: Recently Wang & Isola (2020) analyze contrastive learning from the perspective of uniformity and alignment of learned representations.

Original: However investigations on the behavior of contrastive learning on imbalanced datasets are still absent.
Revised: However investigations on the behavior of contrastive learning on imbalanced datasets are still absent.

Original: We present the ﬁrst study on this problem and our investigation methodology is also applicable to other SSL methods.
Revised: We present the ﬁrst study on this problem and our investigation methodology is also applicable to other SSL methods.

Original: Long-tailed recognition In practice , the visual data usually follow a long-tailed distribution (Zipf, 1999; Spain & Perona, 2007), challenging supervised learning methods.
Revised: In practice, the visual data usually follow a long-tailed distribution (Zipf, 1999; Spain & Perona, 2007), challenging supervised learning methods.

Original: Due to the imbalance in training instance number  for the classes, they tend to perform much more poorly on instance-rare classes than on instance-rich ones.
Revised: Due to the imbalance in the number of training instances for different classes, conventional methods tend to perform much more poorly on instancerare classes than on instance-rich ones.

Original: To alleviate this performance bias, existing approaches either re-balance the data distribution through sampling (Chawla et al., 2002; Han et al., 2005;
Revised: To alleviate this performance bias, existing approaches either re-balance the data distribution through sampling (Chawla et al., 2002; Han et al., 2005;

Original: Shen et al., 2016; Mahajan et al., 2018) or the loss for each class (Cui et al., 2019; Khan et al., 2017; Cao et al., 2019; Khan et al., 2019).
Revised: Shen et al., 2016; Mahajan et al., 2018) or the loss for each class (Cui et al., 2019; Khan et al., 2017; Cao et al., 2019; Khan et al., 2019) by reweighting.

Original: Recently, Kang et al.
Revised: Kang et al.

Original: (2020) demonstrate that learning good feature spaces is crucial for long-tailed recognition. 
Revised: ﬁrst propose to decouple representation learning from classiﬁer learning to boost performance, and demonstrate that learning good feature spaces is crucial for long-tailed recognition. Along this direction, SSP (Yang & Xu, 2020) is among the ﬁrst methods that introduce SSL pretraining into learning the long-tailed recognition models. More speciﬁcally, instead of directly training a randomly initialized model from scratch as conventional supervised learning methods, SSP uses a model pretrained with SSL on the same dataset for initialization, which is observed to be able to to alleviate the label bias issue in imbalanced datasets and boost long-tailed recognition performance. In contrast, we conduct a series of systematic studies to directly compare SSL with supervised learning on representation learning.

Revised: We show that SSL can learn stably well feature spaces robust to the underlying distribution of a dataset.

Original: Inspired by our ﬁndings on the beneﬁts of a balanced feature space for generalization, we introduce the k -positive contrastive learning method to solve the long-tailed recognition problem by learning more balanced feature spaces.
Revised: Moreover, inspired by our ﬁndings on the beneﬁts of a balanced feature space for generalization, we introduce the k -positive contrastive learning method to explicitly pursue balancedness and discriminativeness for representation learning, which has been shown through experiments to beneﬁt not only long-tailed recognition but also normal recognition tasks.

Original:  B ALANCED F EATURE S PACES FROM C ONTRASTIVE L EARNING
Revised:  B ALANCED F EATURE S PACES FROM C ONTRASTIVE L EARNING

Original: In this section, we systematically study the performance of representation models trained by SSL from a collection of training datasets with varying instance number distributions, in contrast with the models learned by supervised learning methods, to explore how SSL performs when the training datasets are not artiﬁcially balanced.
Revised: In this section, we systematically study the performance of representation models trained by SSL from a collection of training datasets with varying instance number distributions, in contrast with the models learned by supervised learning methods, to explore how SSL performs when the training datasets are not artiﬁcially balanced.

Original: Furthermore, we investigate the generalization performance of these learned representation models under multiple settings, in order to explore the relationship between the representation model’s generalizability and the property of its learned feature space.
Revised: Furthermore, we investigate the generalization performance of these learned representation models under multiple settings, in order to explore the relationship between the representation model’s generalizability and the property of its learned feature space.

Original: Notations We deﬁne the notations used in this paper.
Revised: Notations We deﬁne the notations used in this paper.

Original: Representation learning aims to obtain a representation model f
Revised: Representation learning aims to obtain a representation model f

Original: θ that maps a sample x i into a feature space V such that its corresponding representation v i ∈ V encapsulates desired features for target applications.
Revised: θ that maps a sample x i into a feature space V such that its corresponding representation v i ∈ V encapsulates desired features for target applications.

Original: Let D rep-train =
Revised: Let D rep-train =

Original: { x i , y i } , i = 1 , . . .
Revised: { x i , y i } , i = 1 , . . .

Original: , N be the dataset for training the representation model, where y i is the class label for sample x i .
Revised: , N be the dataset for training the representation model, where y i is the class label for sample x i .

Original: Let C denote the number of total classes and n j denote the number of instances within class j .
Revised: Let C denote the number of total classes and n j denote the number of instances within class j .

Original: We use { q 1 , . . .
Revised: We use { q 1 , . . .

Original: , q C } with q j = n j /N to denote the discrete instance distribution over the
Revised: , q C } with q j = n j /N to denote the discrete instance distribution over the

Original: C classes.
Revised: C classes.

Original: An imbalanced dataset has signiﬁcant difference in the class instance numbers, e.g ., q 1 (cid:29) q C .
Revised: An imbalanced dataset has signiﬁcant difference in the class instance numbers, e.g ., q 1 (cid:29) q C .

Original: We use a multi-layer convolutional neural network f θ ( · ) :
Revised: We use a multi-layer convolutional neural network f θ ( · ) :

Original: x i (cid:55)→ v i to implement the representation model.
Revised: x i (cid:55)→ v i to implement the representation model.

Original: The ﬁnal classiﬁcation prediction ˆ y is given by a linear classiﬁer ˆ y = arg max[ W (cid:62) v + b ] , where W denotes the classiﬁer weight matrix and b denotes the bias term.
Revised: The ﬁnal classiﬁcation prediction ˆ y is given by a linear classiﬁer ˆ y = arg max[ W (cid:62) v + b ] , where W denotes the classiﬁer weight matrix and b denotes the bias term.

Original: 3. M ETHODOLOGY OF O UR S TUDY
Revised: 3. M ETHODOLOGY OF O UR S TUDY

Original: Representation learning methods Various loss functions have been developed for learning the representation model f
Revised: Representation learning methods Various loss functions have been developed for learning the representation model f

Original: θ on the training dataset.
Revised: θ on the training dataset.

Original: Among them, the most popular one is the supervised cross-entropy (CE) loss:
Revised: Among them, the most popular one is the supervised cross-entropy (CE) loss:

Original: [Equation]
Revised: [Equation]

Original: where p y i =
Revised: where p y i =

Original: softmax( W (cid:62)
Revised: softmax( W (cid:62)

Original: y
Revised: y

Original: i v i + b ) is the normalized probability prediction of sample i belonging to its ground truth class y i .
Revised: i v i + b ) is the normalized probability prediction of sample i belonging to its ground truth class y i .

Original: Using the semantic labels directly as supervision signal ( y i in Equation 1), the representation model trained by the CE loss can have strong semantic discrimination ability but its generated feature space is easily biased by the imbalance of the training instance distribution—if some classes have signiﬁcantly more training instances than the others, their data representations will occupy dominant portion of the feature space (Figure 1) and get higher classiﬁcation accuracy than the instance-rare classes (Kang et al., 2020; Wang et al., 2017).
Revised: Using the semantic labels directly as supervision signal ( y i in Equation 1), the representation model trained by the CE loss can have strong semantic discrimination ability but its generated feature space is easily biased by the imbalance of the training instance distribution—if some classes have signiﬁcantly more training instances than the others, their data representations will occupy dominant portion of the feature space (Figure 1) and get higher classiﬁcation accuracy than the instance-rare classes (Kang et al., 2020; Wang et al., 2017).

Original: Different from the supervised learning ones, self-supervised learning methods adopt semantic-free loss functions to learn representations from unlabeled data (He et al., 2020; Gidaris et al., 2018).
Revised: Different from the supervised learning ones, self-supervised learning methods adopt semantic-free loss functions to learn representations from unlabeled data (He et al., 2020; Gidaris et al., 2018).

Original: For example, the contrastive loss 1 (CL) (Oord et al., 2018) learns representations via maximizing the instance-wise discriminativeness:
Revised: For example, the contrastive loss 1 (CL) (Oord et al., 2018) learns representations via maximizing the instance-wise discriminativeness:

Original: [Equation]
Revised: [Equation]

Original: where τ is a temperature hyper-parameter, v
Revised: where τ is a temperature hyper-parameter, v

Original: + i is a positive sample for the anchor instance i (typically produced by data augmentation), v − i ∈ V − is the negative sample randomly drawn from the training samples excluding instance i .
Revised: + i is a positive sample for the anchor instance i (typically produced by data augmentation), v − i ∈ V − is the negative sample randomly drawn from the training samples excluding instance i .

Original: This contrastive loss encourages the feature representations from positive pairs to be similar, while pushing features from the sampled negative pairs apart.
Revised: This contrastive loss encourages the feature representations from positive pairs to be similar, while pushing features from the sampled negative pairs apart.

Original: We take the two loss functions ( L CE and L CL ) as representatives to study how the representation models (and the corresponding feature spaces) trained with supervised/self-supervised methods are affected by the training instance distribution { q 1 , . . .
Revised: We take the two loss functions ( L CE and L CL ) as representatives to study how the representation models (and the corresponding feature spaces) trained with supervised/self-supervised methods are affected by the training instance distribution { q 1 , . . .

Original: , q C } .
Revised: , q C } .

Original: Balancedness of feature spaces Since semantic labels are not involved in the contrastive loss (Equation 2), we hypothesize it may lead to representation models yielding feature spaces that are less biased by the imbalance of the training dataset, compared with the ones from the supervised loss (Equation 1).
Revised: Balancedness of feature spaces Since semantic labels are not involved in the contrastive loss (Equation 2), we hypothesize it may lead to representation models yielding feature spaces that are less biased by the imbalance of the training dataset, compared with the ones from the supervised loss (Equation 1).

Original: To verify this, we introduce a metric to characterize such an “unbiased” or “balanced” property of a feature space at ﬁrst.
Revised: To verify this, we introduce a metric to characterize such an “unbiased” or “balanced” property of a feature space at ﬁrst.

Original: A feature space V is balanced if the representations { v i } from different classes within it have similar degrees of linear separability.
Revised: A feature space V is balanced if the representations { v i } from different classes within it have similar degrees of linear separability.

Original: As the linear separability degree of the representations is usually evaluated by the accuracy of a linear classiﬁer over them (Vapnik, 2013), we follow this criterion to develop the balancedness metric.
Revised: As the linear separability degree of the representations is usually evaluated by the accuracy of a linear classiﬁer over them (Vapnik, 2013), we follow this criterion to develop the balancedness metric.

Original: Speciﬁcally, let a 1 , . .
Revised: Speciﬁcally, let a 1 , . .

Original: . , a C denote the classiﬁcation accuracy of a linear classiﬁer ( W, b ) over the representations { v i } ⊂ V from C classes.
Revised: . , a C denote the classiﬁcation accuracy of a linear classiﬁer ( W, b ) over the representations { v i } ⊂ V from C classes.

Original: We take the following uniformity of these accuracies as the balancedness of the feature space V :
Revised: We take the following uniformity of these accuracies as the balancedness of the feature space V :

Original: [Equation]
Revised: [Equation]

Original: Here σ is a ﬁxed scaling parameter.
Revised: Here σ is a ﬁxed scaling parameter.

Original: This metric achieves its maximum when all the class-wise accuracies are equal, i.e ., there being no separability bias of the learned representations to any class.
Revised: This metric achieves its maximum when all the class-wise accuracies are equal, i.e ., there being no separability bias of the learned representations to any class.

Revised: Note that this metric is developed to provide a quantitative measure of the balancedness of a feature space, but it has certain limitations such as it can be easily hacked.

Revised: We leave developing more rigorous metric that can better characterize balanced feature spaces as future work.

Revised: [Figure]

Revised: [Figure]

Original: Experimental protocol We adopt a multi-stage protocol for learning and evaluating the feature spaces.
Revised: Experimental protocol We adopt a multi-stage protocol for learning and evaluating the feature spaces.

Original: (1) Representation learning : pre-train the representation model f
Revised: (1) Representation learning : pre-train the representation model f

Original: θ
Revised: θ

Original: on the provided training set D rep-train using the above training losses L CE and L CL ;
Revised: on the provided training set D rep-train using the above training losses L CE and L CL ;

Original: (2) Classiﬁer learning : train a linear classiﬁer ( W, b ) on top of f θ with θ ﬁxed using another training dataset D train 2 and the supervised
Revised: (2) Classiﬁer learning : train a linear classiﬁer ( W, b ) on top of f θ with θ ﬁxed using another training dataset D train2 and the supervised

Original: CE loss; (3) Representation evaluation: evaluate the classiﬁcation accuracy of the learned classiﬁer on the test dataset D test with the representations from f θ and compute the above balancedness β ( V ) .
Revised: CE loss; (3) Representation evaluation: evaluate the classiﬁcation accuracy of the learned classiﬁer on the test dataset D test with the representations from f θ and compute the above balancedness β ( V ) .

Original: To thoroughly investigate sensitiveness of different representation learning methods to the imbalance level of training datasets, we construct six datasets from the long-tailed benchmark ImageNetLT (Liu et al., 2019)
Revised: To thoroughly investigate sensitiveness of different representation learning methods to the imbalance level of training datasets, we construct six datasets from the long-tailed benchmark ImageNetLT (Liu et al., 2019)

Original: ( D LT ) by varying its instance distribution { q 1 , . . .
Revised: ( D LT ) by varying its instance distribution { q 1 , . . .

Original: , q C } from a long-tailed one to a uniform one gradually, while keeping the total instance number similar.
Revised: , q C } from a long-tailed one to a uniform one gradually, while keeping the total instance number similar.

Original: The generated datasets, denoted as D LT0 , . . .
Revised: The generated datasets, denoted as D LT0 , . . .

Original: , D LT8 , D LT (which are increasingly more imbalanced), are used as D rep-train for representation learning in the following experiments.
Revised: , D LT8 , D LT (which are increasingly more imbalanced), are used as D rep-train for representation learning in the following experiments.

Original: See appendix for their details.
Revised: See appendix for their details.

Original: 3. C ONTRASTIVE L OSS H ELPS L EARN B ALANCED F EATURE S PACES
Revised: 3. C ONTRASTIVE L OSS H ELPS L EARN B ALANCED F EATURE S PACES

Original: We ﬁrst investigate classiﬁcation performance of the representation models trained with the CE and
Revised: We ﬁrst investigate classiﬁcation performance of the representation models trained with the CE and

Original: CL losses on the above six datasets D LT0 , . .
Revised: CL losses on the above six datasets D LT0 , . .

Original: , D LT8 , D LT that are increasingly more imbalanced.
Revised: , D LT8 , D LT that are increasingly more imbalanced.

Original: [Figure] [Figure] Since linear classiﬁers are easily biased by skewed training dataset distribution (Kang et al., 2020), it is necessary to eliminate the imblancedness of the evaluation datasets for reliable representation evaluation.
Revised: Since linear classiﬁers are easily biased by skewed training dataset distribution (Kang et al., 2020), it is necessary to eliminate the imblancedness of the evaluation datasets for reliable representation evaluation.

Original: Thus, we use the (balanced) training and test sets of ImageNet as D train and D test to learn classiﬁers and evaluate their classiﬁcation accuracy, following the above protocol.
Revised: Thus, we use the (balanced) training and test sets of ImageNet as D train and D test to learn classiﬁers and evaluate their classiﬁcation accuracy, following the above protocol.

Original: The results are summarized in Figure 2, from which we make an important observation: compared with the supervised cross-entropy loss, the model trained with the unsupervised contrastive loss generates a more balanced feature space , even in presence of highly imbalanced training instance distribution.
Revised: The results are summarized in Figure 2, from which we make an important observation: compared with the supervised cross-entropy loss, the model trained with the unsupervised contrastive loss generates a more balanced feature space , even in presence of highly imbalanced training instance distribution.

Original: As shown in Figure 2 (left), the classiﬁcation accuracy of representation models learned with the CE loss drops quickly when the dataset becomes more imbalanced—the quality of representations from these models is very sensitive to the imbalance of training datasets.
Revised: As shown in Figure 2 (left), the classiﬁcation accuracy of representation models learned with the CE loss drops quickly when the dataset becomes more imbalanced—the quality of representations from these models is very sensitive to the imbalance of training datasets.

Original: In contrast, the classiﬁcation accuracy of CL-trained models remains stable even when the training dataset transits to a heavily long-tailed one.
Revised: In contrast, the classiﬁcation accuracy of CL-trained models remains stable even when the training dataset transits to a heavily long-tailed one.

Original: Such surprising performance robustness to imbalance of the training datasets implies that using contrastive learning can consistently learn balanced feature spaces.
Revised: Such surprising performance robustness to imbalance of the training datasets implies that using contrastive learning can consistently learn balanced feature spaces.

Original: To see this, we also visualize the balancedness scores (Equation 3) of the learned feature spaces from
Revised: To see this, we also visualize the balancedness scores (Equation 3) of the learned feature spaces from

Original: CE and CL in Figure 2 (right).
Revised: CE and CL in Figure 2 (right).

Original: Even when the training set is heavily long-tailed, the feature spaces learned with CL loss are as highly balanced as the ones learned from a uniform training distribution, while the blancedness score of the feature spaces from CE loss is lower and drops quickly.
Revised: Even when the training set is heavily long-tailed, the feature spaces learned with CL loss are as highly balanced as the ones learned from a uniform training distribution, while the blancedness score of the feature spaces from CE loss is lower and drops quickly.

Original: Such a balanced feature space offered by CL loss is much desired for the representation learning in practice, where the training instance distribution is usually long-tailed.
Revised: Such a balanced feature space offered by CL loss is much desired for the representation learning in practice, where the training instance distribution is usually long-tailed.

Original: Certainly, using an unsupervised loss will sacriﬁce semantic discriminativeness of the representations, leading to the accuracy gap between the CE and CL-trained models.
Revised: Certainly, using an unsupervised loss will sacriﬁce semantic discriminativeness of the representations, leading to the accuracy gap between the CE and CL-trained models.

Original: 3. M ORE B ALANCED R EPRESENTATION M ODELS G ENERALIZE B ETTER
Revised: 3. M ORE B ALANCED R EPRESENTATION M ODELS G ENERALIZE B ETTER

Original: The above studies reveal that the representation models trained with the contrastive loss can produce more balanced feature spaces.
Revised: The above studies reveal that the representation models trained with the contrastive loss can produce more balanced feature spaces.

Original: A natural question is what are the beneﬁts from a balanced model for  recognition?
Revised: A natural question is what are the beneﬁts from a balanced model for [Table] recognition?

Original: Since a pre-trained representation model is often used to facilitate downstream tasks (
Revised: Since a pre-trained representation model is often used to facilitate downstream tasks (

Original: He et al., 2020; Newell & Deng, 2020; H ´ enaff et al., 2019), we here conduct extensive experiments to study its potential beneﬁts on model generalization performance under the following settings.
Revised: He et al., 2020; Newell & Deng, 2020; H´enaff et al., 2019) [Table] we here conduct extensive experiments to study its potential beneﬁts on model generalization performance under the following settings.

Original: Out-of-distribution generalization We ﬁrst study the relationship between the balancedness of representation models and their generalizability for recognizing new classes.
Revised: Out-of-distribution generalization We ﬁrst study the relationship between the balancedness of representation models and their generalizability for recognizing new classes.

Original: To thoroughly evaluate performance of representation models with different balancedness, we evenly divide the 1, classes into two splits (500 vs. 500 classes) on ImageNet, referred to as the source and target split respectively.
Revised: To thoroughly evaluate performance of representation models with different balancedness, we evenly divide the 1, classes into two splits (500 vs. 500 classes) on ImageNet, referred to as the source and target split respectively.

Original: We use the subsets (corresponding to the source class split) of the above
Revised: We use the subsets (corresponding to the source class split) of the above

Original: D LT0 , . .
Revised: D LT0 , . .

Original: , D LT8 , D LT datasets to construct six different D rep-train for training the representation model f θ .
Revised: , D LT8 , D LT datasets to construct six different D rep-train for training the representation model f θ .

Original: To obtain models with different balancedness, we use the CE loss for training, since the above studies reveal using the CL loss will always produce models with similar balancedness (Figure 2).
Revised: To obtain models with different balancedness, we use the CE loss for training, since the above studies reveal using the CL loss will always produce models with similar balancedness (Figure 2) [Table]

Original: We use the subsets (corresponding to the target class split) of the training and test sets of ImageNet as D train and D test for classiﬁer learning and evaluation, with the representation model ﬁxed.
Revised: We use the subsets (corresponding to the target class split) of the training and test sets of ImageNet as D train and D test for classiﬁer learning and evaluation, with the representation model ﬁxed.

Original: The testing performance of the models with different balancedness on the target classes is presented in Figure 3.
Revised: The testing performance of the models with different balancedness on the target classes is presented in Figure 3.

Original: It is observed that as the source dataset becomes increasingly more imbalanced (from
Revised: It is observed that as the source dataset becomes increasingly more imbalanced (from

Original: D LT0 to D LT ) and the corresponding models become more imbalanced, their generalization performance degrades correspondingly.
Revised: D LT0 to D LT ) and the corresponding models become more imbalanced, their generalization performance degrades correspondingly.

Original: Such a positive correlation between balancedness of the models and testing accuracy on the target classes clearly demonstrate that more balanced representation[Table] models tend to generalize better for recognizing unseen classes.
Revised: Such a positive correlation between balancedness of the models and testing accuracy on the target classes clearly demonstrate that more balanced representation models tend to generalize better for recognizing unseen classes.

Original: More details and results about the out-of-distribution generalization studies are deferred to the appendix
Revised: More details and results about the out-of-distribution generalization studies are deferred to the appendix.

Original: [Table]

Original: Cross-domain and cross-task generalization We then explore whether learning balanced representation models is able to beneﬁt model’s generalizability to new domains and tasks.
Revised: Cross-domain and cross-task generalization We then explore whether learning balanced representation models is able to beneﬁt model’s generalizability to new domains and tasks.

Original: We use the
Revised: We use the

Original: ImageNet-LT as D rep-train to train the models with the CE and CL losses, obtaining imbalanced and balanced representation models respectively.
Revised: ImageNet-LT as D rep-train to train the models with the CE and CL losses, obtaining imbalanced and balanced representation models respectively.

Original: For the cross-domain setting, we train a linear classiﬁer on the Places365 dataset (Zhou et al., 2017) with the representation model ﬁxed.
Revised: For the cross-domain setting, we train a linear classiﬁer on the Places365 dataset (Zhou et al., 2017) with the representation model ﬁxed.

Original: From the results in
Revised: From the results in

Original: Table [Table] it can be clearly observed that the balanced representation model (from CL) surpasses the less balanced one (from CE) signiﬁcantly, in terms of the top-1 accuracy (by 2.74%).
Revised: Table 1, it can be clearly observed that the balanced representation model (from CL) surpasses the less balanced one (from CE) signiﬁcantly, in terms of the top-1 accuracy (by 2.74%).

Original: For the crosstask setting, we take train/test splits of the PASCAL VOC (Everingham et al., 2010) and COCO (Lin et al., 2014) datasets as D train / D test for evaluating detection performance.
Revised: For the crosstask setting, we take train/test splits of the PASCAL VOC (Everingham et al., 2010) and COCO (Lin et al., 2014) datasets as D train / D test for evaluating detection performance.

Original: The results are given in
Revised: The results are given in

Original: Table 1.
Revised: Table 1.

Original: Again, the balanced model (from the CL loss) outperforms the less balanced one (from the
Revised: Again, the balanced model (from the CL loss) outperforms the less balanced one (from the

Original: CE loss) signiﬁcantly (up to 1.64%).
Revised: CE loss) signiﬁcantly (up to 1.64%).

Original: In comparison, the improvement from the CL-trained model over the CE-trained model is moderate (around 0.3%) when using the full ImageNet for training, as CE can learn a relatively balanced feature space from a balanced dataset.
Revised: In comparison, the improvement from the CL-trained model over the CE-trained model is moderate (around 0.3%) when using the full ImageNet for training, as CE can learn a relatively balanced feature space from a balanced dataset.

Original: This clearly shows that the generalization performance boost for the cross-domain and cross-task settings brought from
Revised: This clearly shows that the generalization performance boost for the cross-domain and cross-task settings brought from

Original: CL-trained models does not simply stem from using self-supervised pre-training, but indeed come from learning more balanced feature spaces.
Revised: CL-trained models does not simply stem from using self-supervised pre-training, but indeed come from learning more balanced feature spaces.

Original:  L EARNING B ALANCED F EATURE S PACES FOR R ECOGNITION
Revised:  L EARNING B ALANCED F EATURE S PACES FOR R ECOGNITION

Original: The above studies demonstrate the representation models trained with the contrastive loss can generate balanced feature spaces showing strong generalizability.
Revised: The above studies demonstrate the representation models trained with the contrastive loss can generate balanced feature spaces showing strong generalizability.

Original: Here we explore how to effectively leverage these ﬁndings in practice.
Revised: Here we explore how to effectively leverage these ﬁndings in practice.

Original: We introduce a new method that inherits the strength of the contrastive loss in learning balanced feature spaces and enhances the feature spaces’ semantic discrimination capability simultaneously.
Revised: We introduce a new method that inherits the strength of the contrastive loss in learning balanced feature spaces and enhances the feature spaces’ semantic discrimination capability simultaneously.

Original: We thoroughly study its superiority via two application cases, i.e ., the long-tailed recognition and pre-training representation models for downstream tasks.
Revised: We thoroughly study its superiority via two application cases, i.e ., the long-tailed recognition and pre-training representation models for downstream tasks.

Original: 4. K-P OSITIVE C ONSTRASTIVE L OSS
Revised: 4. K-P OSITIVE C ONSTRASTIVE L OSS

Original: Though balanced, the feature spaces from contrastive learning have limited capability of semantic discrimination, as shown in Figure 2 (left).
Revised: Though balanced, the feature spaces from contrastive learning have limited capability of semantic discrimination, as shown in Figure 2 (left).

Original: This is because the contrastive loss blindly encourages instance-level discrimiantiveness.
Revised: This is because the contrastive loss blindly encourages instance-level discrimiantiveness.

Original: Every two instances, even if they are from the same class, are forced to be apart from each other in the learned feature space.
Revised: Every two instances, even if they are from the same class, are forced to be apart from each other in the learned feature space.

Original: To embed semantic discriminativeness into the representations while maintaining desired balancedness, we develop a new method to leverage the provided semantic labels to adaptively compute the instance contrastive loss.
Revised: To embed semantic discriminativeness into the representations while maintaining desired balancedness, we develop a new method to leverage the provided semantic labels to adaptively compute the instance contrastive loss.

Original: Concretely, given an anchor training instance x i with its semantic label y i , our proposed method draws k instances from the same class to form the positive sample set V
Revised: Concretely, given an anchor training instance x i with its semantic label y i , our proposed method draws k instances from the same class to form the positive sample set V

Original: + i,k , instead of only using its augmentation as in Equation 2.
Revised: + i,k , instead of only using its augmentation as in Equation 2.

Original: Thus, it gives a new loss called k -positive contrastive loss (KCL):
Revised: Thus, it gives a new loss called k -positive contrastive loss (KCL):

Original: [Equation]
Revised: [Equation]

Original: where ˜ v i is generated by augmenting v i , V i is the current batch of examples excluding v i , and
Revised: where ˜ v i is generated by augmenting v i , V i is the current batch of examples excluding v i , and

Original: V + i,k ⊂ V
Revised: V + i,k ⊂ V

Original: i is a positive set containing k instances randomly drawn from the same class as v i .
Revised: i is a positive set containing k instances randomly drawn from the same class as v i .

Original: The proposed KCL loss purposely keeps the number of positive instances equal, which is crucial for balancing the learned feature spaces.
Revised: The proposed KCL loss purposely keeps the number of positive instances equal, which is crucial for balancing the learned feature spaces.

Original: It brings two beneﬁts.
Revised: It brings two beneﬁts.

Original: First, it helps learn representations[Table] [Table] with stronger discriminative ability as it leverages the label information as supervised learning.
Revised: First, it helps learn representations with stronger discriminative ability as it leverages the label information as supervised learning.

Original: Secondly

Original: [Table] it uses the same number of instances ( i [Table] e ., k ) for all the classes in positive pair construction which further balances the learned feature space.
Revised: Secondly, it uses the same number of instances ( i.e ., k ) for all the classes in positive pair construction which further balances the learned feature space.

Original: Note our proposed KCL is different from the supervised contrastive learning (Khosla et al., 2020) that leverages all the instances from the same class to construct the positive pairs, which cannot avoid the dominance of instance-rich classes in the representation learning.
Revised: Note our proposed KCL is different from the supervised contrastive learning (Khosla et al., 2020) that leverages all the instances from the same class to construct the positive pairs, which cannot avoid the dominance of instance-rich classes in the representation learning.

Original: This is also evidenced by our following experiments on long-tailed recognition.
Revised: This is also evidenced by our following experiments on long-tailed recognition.

Original: In the following experiments, we choose k = 6 via validation and use ResNet50 as the backbone.
Revised: In the following experiments, we choose k = 6 via validation and use ResNet50 as the backbone.

Original: Other hyper-parameter choices and implementation details are given in the appendix
Revised: Other hyper-parameter choices and implementation details are given in the appendix.

Original: . 4. L ONG -T AILED R ECOGNITION
Revised: 4. L ONG -T AILED R ECOGNITION

Original: KCL provides feature spaces with desirable balancedness and semantic discriminativeness, which makes it naturally ﬁt for addressing the challenges of long-tailed recognition, i.e ., severe performance bias to the instance-rich classes and poor generalization to the instance-rare classes (Mahajan et al., 2018; Zhong et al., 2019).
Revised: KCL provides feature spaces with desirable balancedness and semantic discriminativeness, which makes it naturally ﬁt for addressing the challenges of long-tailed recognition, i.e ., severe performance bias to the instance-rich classes and poor generalization to the instance-rare classes (Mahajan et al., 2018; Zhong et al., 2019).

Original: Here we implement and evaluate KCL for long-tailed recognition, following the two-stage training strategy from Kang et al.
Revised: Here we implement and evaluate KCL for long-tailed recognition, following the two-stage training strategy from Kang et al.

Original: (2020): 1) train the representation model with the KCL loss; 2) learn a linear classiﬁer with cross-entropy loss and class-balanced sampling.
Revised: (2020): 1) train the representation model with the KCL loss; 2) learn a linear classiﬁer with cross-entropy loss and class-balanced sampling.

Original: Baselines Besides well established state-of-the-arts, we consider following three kinds of baselines for justifying the advantages of KCL.
Revised: Baselines Besides well established state-of-the-arts, we consider following three kinds of baselines for justifying the advantages of KCL.

Original: (1) Classiﬁer balancing methods, i.e ., τ -norm and cRT (Kang et al., 2020), that re-train classiﬁers with class-balanced sampling as KCL but learn the representation models by supervised cross-entropy loss.
Revised: (1) Classiﬁer balancing methods, i.e ., τ -norm and cRT (Kang et al., 2020), that re-train classiﬁers with class-balanced sampling as KCL but learn the representation models by supervised cross-entropy loss.

Original: Comparison with them helps understand the effectiveness of learning balanced features in long-tailed recognition.
Revised: Comparison with them helps understand the effectiveness of learning balanced features in long-tailed recognition.

Original: (2) Methods that train the representation model and classiﬁer jointly with cross-entropy loss (SL) and various data re-sampling strategies, including instance-balanced (SL-i), class-balanced (SL-c), progressively-balanced (SL-p) and square-root re-sampling (SL-s) (Kang et al., 2020).
Revised: (2) Methods that train the representation model and classiﬁer jointly with cross-entropy loss (SL) and various data re-sampling strategies, including instance-balanced (SL-i), class-balanced (SL-c), progressively-balanced (SL-p) and square-root re-sampling (SL-s) (Kang et al., 2020).

Original: Comparison with them will show advantages of KCL over these data-enriching strategies in feature space balancing.
Revised: Comparison with them will show advantages of KCL over these data-enriching strategies in feature space balancing.

Original: (3) A full-positive variant of
Revised: (3) A full-positive variant of

Original: KCL , named full-positive contrastive learning (FCL), that uses all the available same-class samples in the current batch to construct positive pairs for computing the contrastive loss, which is similar to the supervised contrastive learning (Khosla et al., 2020).
Revised: KCL , named full-positive contrastive learning (FCL), that uses all the available same-class samples in the current batch to construct positive pairs for computing the contrastive loss, which is similar to the supervised contrastive learning (Khosla et al., 2020).

Original: Comparing KCL with FCL will show the beneﬁts of keeping the number of positive samples equal for all the anchor instances in KCL.
Revised: Comparing KCL with FCL will show the beneﬁts of keeping the number of positive samples equal for all the anchor instances in KCL.

Original: Results We evaluate KCL and compare it with the above strong baselines on two large-scale benchmark datasets, ImageNet-LT (Liu et al., 2019) and iNaturalist 2018 (iNatrualist, 2018).
Revised: Results We evaluate KCL and compare it with the above strong baselines on two large-scale benchmark datasets, ImageNet-LT (Liu et al., 2019) and iNaturalist 2018 (iNatrualist, 2018).

Original: For comprehensive evaluation, following (Liu et al., 2019), we split the classes of ImageNet-LT into many-shot ( > 100 images), medium-shot (20 ∼ 100 images) and few-shot ( < 20 images) groups.
Revised: For comprehensive evaluation, following (Liu et al., 2019), we split the classes of ImageNet-LT into many-shot ( > 100 images), medium-shot (20 ∼ 100 images) and few-shot ( < 20 images) groups.

Original: The results are summarized in Tables 2 and 3 respectively, along with the balancedness of these methods on ImageNet-LT in Figure 4.
Revised: The results are summarized in Tables 2 and 3 respectively, along with the balancedness of these methods on ImageNet-LT in Figure 4.

Original: We make the following observations.
Revised: We make the following observations.

Original: More balanced feature spaces give better performance.
Revised: More balanced feature spaces give better performance.

Original: We ﬁrst compare KCL with cRT and τnorm, the latest state-of-the-arts with feature spaces learned by supervised cross-entropy loss and thus less balanced as demonstrated in Sec. 3.
Revised: We ﬁrst compare KCL with cRT and τnorm, the latest state-of-the-arts with feature spaces learned by supervised cross-entropy loss and [Table] [Table] [Figure]

Revised: [Table]

Original: Compared with them, KCL improves the overall accuracy by a large margin (4.2% on ImageNet-LT and 3% on iNaturalist), demonstrating the importance of learning more balanced feature spaces for long-tailed recognition.
Revised: [Figure] [Table] [Figure] [Table]

Revised: thus less balanced as demonstrated in Sec. 3.

Revised: Compared with them, KCL improves the overall accuracy by a large margin (4.2% on ImageNet-LT and 3% on iNaturalist), demonstrating the importance of learning more balanced feature spaces for long-tailed recognition.

Original: KCL is more effective at learning balanced and discriminative feature spaces.
Revised: KCL is more effective at learning balanced and discriminative feature spaces.

Original: Data re-sampling is widely used as a straightforward approach to alleviate performance bias for long-tailed recognition (Kang et al., 2020).
Revised: Data re-sampling is widely used as a straightforward approach to alleviate performance bias for long-tailed recognition (Kang et al., 2020).

Original: We compare the feature space balancedness of KCL and SL methods with different data re-sampling strategies on ImageNet-LT in Figure 4.Clearly, data re-sampling cannot i n s t an c e ­ ba l an c ed c
Revised: We compare the feature space balancedness of KCL and SL methods with different data re-sampling strategies on ImageNet-LT in Figure 4.

Original: l a ss ­ ba l an c ed p r og r e ss

Original: i v e l y ­ ba

Original: l an c ed s qua r e ­r oo t k=full k=

Original: SL

Original: KCL MoCo

Original: Balancedness(%)

Original: Class Index

Original: A cc u r a c y ( % )

Original: Accuracies on ImageNet-LT SL-iSL-cSL-p SL-s FCLKCLMoCo effectively improve balancedness of the feature space as KCL.
Revised: Clearly, data re-sampling cannot effectively improve balancedness of the feature space as KCL.

Original: Besides data re-sampling, Figurealso shows the balancedness of the feature spaces learned by the latest contrastive learning method
Revised: Besides data re-sampling, Figurealso shows the balancedness of the feature spaces learned by the latest contrastive learning method

Original: MoCo (He et al., 2020) on ImageNet-LT.
Revised: MoCo (He et al., 2020) on ImageNet-LT.

Original: MoCo can balance the feature space but has lower accuracy, due to the lack of semantic discriminativeness in the learned feature space.
Revised: MoCo can balance the feature space but has lower accuracy, due to the lack of semantic discriminativeness in the learned feature space.

Original: KCL performs the best, demonstrating its effectiveness at learning both balanced and discriminative feature spaces.
Revised: KCL performs the best, demonstrating its effectiveness at learning both balanced and discriminative feature spaces.

Original: Equalizing the number of positive instances in KCL is important.
Revised: Equalizing the number of positive instances in KCL is important.

Original: To further justify the design of
Revised: To further justify the design of

Original: KCL loss in keeping the number of positive instances to be equal, we compare it with its variant
Revised: KCL loss in keeping the number of positive instances to be equal, we compare it with its variant

Original: FCL.
Revised: FCL.

Original: From Tables 2, 3 and Figure 4, though FCL outperforms other baselines, its performance is inferior to KCL, in terms of both the overall accuracy and the balancedness of the learned feature spaces.
Revised: From Tables 2, 3 and Figure 4, though FCL outperforms other baselines, its performance is inferior to KCL, in terms of both the overall accuracy and the balancedness of the learned feature spaces.

Original: Equalizing the number of positive instances as KCL is crucial for learning balanced feature spaces and improving recognition performance.
Revised: Equalizing the number of positive instances as KCL is crucial for learning balanced feature spaces and improving recognition performance.

Original: 4. P RE - TRAINING R EPRESENTATION M ODELS FOR D OWNSTREAM T ASKS
Revised: 4. P RE - TRAINING R EPRESENTATION M ODELS FOR D OWNSTREAM T ASKS

Original: The effectiveness of KCL is not limited to the cases where training datasets are imbalanced.
Revised: The effectiveness of KCL is not limited to the cases where training datasets are imbalanced.

Original: Here we apply KCL for pre-training the representation model on balanced datasets which is later ﬁne-tuned for downstream tasks, including the out-of-distribution (OOD) recognition (the source/target splitof ImageNet for pre-training/evaluation) and detection (ImageNet: pre-training; PASCAL VOC and 
Revised: In this section, we study KCL as a general representation learning method, i.e ., we apply KCL for pretraining a representation model on balanced datasets which is later ﬁne-tuned for downstream tasks, including the out-of-distribution (OOD) recognition and detection. Out-of-distribution Generalization Similar to Sec.3.3, we evenly divide the 1000 classes in ImageNet into two splits, use one split to learning representation backbone and the thoer one to learn a linear classiﬁer with the backbone ﬁxed. We adopt two different splitting strategies. Split-overlap (split with semantic overlap) allows the classes within the two splits to share the same super class ( e.g ., dog and wolf from canidae are put into different splits) in the ImageNet ontology.

Original: COCO: evaluation).
Revised: As such, though the target classes are all novel to the representation model, some of their attributes have been seen by the model before from the source classes. In contrast, Split-independent (split without semantic overlap) strictly avoids classes from the same super classes to be distributed into different splits. Split-independent presents a more challenging case for model’s generalization ability as all the target classes (and attributes) to recognize are novel. [Table]

Revised: The generalization performance comparsion of the representation models with different methods is given in Table 4.

Revised: When the source classes and target classes share similar semantics (on splitoverlap ), the CE-trained model surpasses the CL-trained model on both the source and target classes.

Revised: But when looking into the generalization gap ( i

Revised: [Table]

Revised: e ., the difference between the source and target accuracy)

Revised: [Table]

Revised: the CL-trained model suffers larger generalization gap than the CL-trained model (10. vs. 1.8).

Revised: When there is not semantic overlap between source and target (on split-independent ), the

Revised: CL-model outperforms CE-model on the target classes by 4.3% with much smaller generalization gap (13.4 v.s. 32.5).

Revised: By comparing the “full” performance from split-overlap to split-independent , one can observe that CL loss performs consistently well (58.3 and 58.2), but CE drops as large as 5%.

Revised: This implies that CL is robust to imbalanced training distribution used for representation learning, while CE is extremely sensitive to it.

Revised: These results clearly demonstrate the consistent superiority of balanced representation learning in terms of generalization for various training dataset distribution.

Revised: Notably, our proposed KCL loss surpasses both CE and CL loss on all the four different settings by a large margin (more than 3 points).

Revised: These results clearly demonstrate that KCL is able to learn a balanced and discriminative feature space, and balancedness is a general property that beneﬁts both balanced and imbalanced datasets.

Revised: [Table]

Revised: Cross-domain and cross-task generalization In this part, we ﬁrst pretrain a model on ImageNet then further ﬁnetune it for downstream object detaction tasks (including PASCAL VOC and COCO).

Original: Note we aim to study the generalizability of KCL as a representation learning method, rather than aiming at state-of-the-art performance.
Revised: Note we aim to study the generalizability of KCL as a representation learning method, rather than aiming at state-of-the-art performance.

Original: Hence we compare it with the vanilla supervised cross-entropy loss (SL) and MoCo (which the KCL is built on)
Revised: Hence we compare it with the vanilla supervised crossentropy loss (SL) and MoCo (which the KCL is built on)

Original: (He et al., 2020).
Revised: (He et al., 2020).

Original: The results are summarized in Table 4.
Revised: The results are summarized in Table 5.

Original: We also evaluate the discrminativeness of the learned representations (the “repr” in the table) from their classiﬁcation accuracy by learning a linear classiﬁer on the pretraining datasets.
Revised: We also evaluate the discrminativeness of the learned representations (the “repr” in the table) from their classiﬁcation accuracy by learning a linear classiﬁer on the pretraining datasets.

Original: Clearly, KCL outperforms SL and MoCo for both downstream tasks.
Revised: Clearly, KCL outperforms SL and MoCo for both downstream tasks.

Original: This is because KCL learns more balanced feature spaces than SL with similar discriminativeness, and learns more discriminative features than MoCo.
Revised: This is because KCL learns more balanced feature spaces than SL with similar discriminativeness, and learns more discriminative features than MoCo.

Original: [Table]
Revised: For more results, Please refer to the appendix .

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: This work piloted studies on performance of the self-supervised learning methods for imbalanced datasets, and made several intriguing ﬁndings.
Revised: This work piloted studies on performance of the self-supervised learning methods for imbalanced datasets, and made several intriguing ﬁndings.

Original: At the heart of these ﬁndings is the balanced feature space, which is identiﬁed to be an inherent property of the representations learned by the contrastive learning and bring stronger generalizability.
Revised: At the heart of these ﬁndings is the balanced feature space, which is identiﬁed to be an inherent property of the representations learned by the contrastive learning and bring stronger generalizability.

Original: It provides a new perspective for understanding the behavior of the contrastive learning.
Revised: It provides a new perspective for understanding the behavior of the contrastive learning.

Original: This work further developed a new representation learning method to leverage the beneﬁts of balanced feature spaces.
Revised: This work further developed a new representation learning method to leverage the beneﬁts of balanced feature spaces.

Original: We believe the ﬁndings and method developed here are inspiring for the future research on representation learning.
Revised: We believe the ﬁndings and method developed here are inspiring for the future research on representation learning.

Original: However, theoretical understandings on balanced feature spaces are not mature yet and worthy of future exploration.
Revised: However, theoretical understandings on balanced feature spaces are not mature yet and worthy of future exploration.

Revised: A CKNOWLEDGEMENT

Revised: We would like to express our deepest gratitude to Saining Xie for his comments and suggestions throughout this project and the writing of the paper, to Yu Sun for his insightful discussion at the begining of this project.

Original: A I MPLEMENTATION D ETAILS
Revised: A I MPLEMENTATION D ETAILS

Original: Representation Learning For supervised cross-entropy (CE) loss, we adopt the standard PyTorch distributed training implementation 4 .
Revised: Representation Learning For supervised cross-entropy (CE) loss, we adopt the standard PyTorch distributed training implementation 3 .

Original: For unsupervised contrastive loss (CL), we use the ofﬁcial implementation 5 of MoCo (He et al., 2020) with default hyper-parameters.
Revised: For unsupervised contrastive loss (CL), we use the ofﬁcial implementation 4 of MoCo (He et al., 2020) with default hyper-parameters.

Original: Our k -positive contrastive loss is implemented based on MoCo by randomly selecting k positive examples from the key memory for each of the query example.
Revised: Our k -positive contrastive loss is implemented based on MoCo by randomly selecting k positive examples from the key memory for each of the query example.

Original: When KCL is applied for a long-tailed dataset, there might be less than k positive examples in the memory for some of the classes.
Revised: When KCL is applied for a long-tailed dataset, there might be less than k positive examples in the memory for some of the classes.

Original: In such cases, we use all the positive examples.
Revised: In such cases, we use all the positive examples.

Original: Besides, we keep all the hyper-parameters ( e.g ., data augmentation, learning rate and batch size) the same as supervised learning.
Revised: Besides, we keep all the hyper-parameters ( e.g ., data augmentation, learning rate and batch size) the same as supervised learning.

Original: We only carefully tune the number of training epochs for KCL to make it achieve similar performance as supervised learning on a balanced dataset (full Imagenet) for fair comparison (see Table 5).
Revised: We only carefully tune the number of training epochs for KCL to make it achieve similar performance as supervised learning on a balanced dataset (full Imagenet) for fair comparison (see Table 6).

Original: As a result, throughout the paper our KCL is trained for 200 epochs while its supervised counterpart is trained for 90 epochs.
Revised: As a result, throughout the paper our KCL is trained for 200 epochs while its supervised counterpart is trained for 90 epochs.

Original: This is reasonable as contrastive learning usually takes much longer to converge (He et al., 2020).
Revised: This is reasonable as contrastive learning usually takes much longer to converge (He et al., 2020).

Original: We are using k =for KCL throughout the paper, which is carefully tuned on the validation set of ImageNet-LT, as shown in Fig.
Revised: We are using k =for KCL throughout the paper, which is carefully tuned on the validation set of ImageNet-LT, as shown in Fig.

Original: The detailed hyper-parameters of different loss functions are given in Table 6.
Revised: The detailed hyper-parameters of different loss functions are given in Table 7.

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: Classiﬁer Learning We need to train linear classiﬁers in two cases.
Revised: Classiﬁer Learning We need to train linear classiﬁers in two cases.

Original: 1) representation evalauation (Sec. 3 and Figure 4).
Revised: 1) representation evalauation (Sec. 3 and Figure 4).

Original: For CE-learned representations, we train a linear classiﬁer using the same parameters as representation learning in Table 6 with a smaller number of epochs (10).
Revised: For CE-learned representations, we train a linear classiﬁer using the same parameters as representation learning in Table 7 with a smaller number of epochs (10).

Original: For CL and
Revised: For CL and

Original: KCL, we adopt the classiﬁer training protocol introduced by MoCo (He et al., 2020) with default hyper-parameters ( i.e ., learning rate 30 and weight decay 0).
Revised: KCL, we adopt the classiﬁer training protocol introduced by MoCo (He et al., 2020) with default hyper-parameters ( i.e ., learning rate 30 and weight decay 0).

Original: 2) Long-tailed recognition (Sec. 4.2).
Revised: 2) Long-tailed recognition (Sec. 4.2).

Original: We obtain a re-balanced classiﬁer for CE representations following Kang et al.
Revised: We obtain a re-balanced classiﬁer for CE representations following Kang et al.

Original: (2020), and adopt the MoCo classiﬁer learning strategy with class-balanced sampling by setting the learning rate toon ImageNet-LT and 30 on iNaturalist 2018.
Revised: (2020), and adopt the MoCo classiﬁer learning strategy with class-balanced sampling by setting the learning rate toon ImageNet-LT and 30 on iNaturalist 2018.

Original: Detection model training We use exactly the same setting and evaluation metrics as He et al.
Revised: Detection model training We use exactly the same setting and evaluation metrics as He et al.

Original: R50-C4 backbone is used with BN tuned.
Revised: R50-C4 backbone is used with BN tuned.

Original: The image is rescaled to [640, 800] during training and 800 at inference.
Revised: The image is rescaled to [640, 800] during training and 800 at inference.

Original: All layers are ﬁne-tuned end-to-end with batch size = 16.
Revised: All layers are ﬁne-tuned end-to-end with batch size = 16.

Original: For Pascal
Revised: For Pascal

Original: VOC, we train Faster R-CNN (Ren et al., 2015) on trainval07+12 set with 24k schedule and evaluate on test07 set.
Revised: VOC, we train Faster R-CNN (Ren et al., 2015) on trainval07+12 set with 24k schedule and evaluate on test07 set.

Original: For COCO, we train Mask R-CNN (He et al., 2017) on train2017 set with × 2 schedule and evaluate on val2017 set.
Revised: For COCO, we train Mask R-CNN (He et al., 2017) on train2017 set with × 2 schedule and evaluate on val2017 set.

Original: B D ATASET C ONSTRUCTION
Revised: B D ATASET C ONSTRUCTION

Original: Datasets for studying balancedness of feature spaces We here explain the details on the construction of the series of datasets used in our study in Sec. 3.
Revised: Datasets for studying balancedness of feature spaces We here explain the details on the construction of the series of datasets used in our study in Sec. 3.

Original: In particular, we take the standard long-tailed training set from ImageNet-LT (Liu et al., 2019) whose instances follow the Pareto distribution as the base dataset, denoted as D LT .
Revised: In particular [Figure] we take the standard long-tailed training set from ImageNet-LT (Liu et al., 2019) whose instances follow the Pareto distribution as the base dataset, denoted as D LT .

Original: We vary its training instance distribution { q 1 , . . .
Revised: We vary its training instance distribution { q 1 , . . .

Original: , q C } gradually to obtain different datasets as follows,
Revised: , q C } gradually to obtain different datasets as follows,

Original: [Equation]
Revised: [Equation]

Original: where N D is the total number of training instances in D LT , α ∈ [0 , 1] controls the dataset balancedness.
Revised: where N D is the total number of training instances in D LT , α ∈ [0 , 1] controls the dataset balancedness.

Original: When α = 0 , it corresponds to a fully balanced dataset; when α = 1 , it becomes a heavy long-tailed ones.
Revised: When α = 0 , it corresponds to a fully balanced dataset; when α = 1 , it becomes a heavy long-tailed ones.

Original: In total, we generated 6 datasets with α ∈ { 0 , 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 , 1 . 0 } , denoted as
Revised: In total [Figure] we generated 6 datasets with α ∈ { 0 , 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 , 1 . 0 } , denoted as

Original: D LT0 , . .
Revised: D LT0 , . .

Original: , D LT8 , D LT respectively, as different examples of D rep-train for representation learning.
Revised: , D LT8 , D LT respectively, as different examples of D rep-train for representation learning.

Original: The detailed statistics and visualization of the datasets D LT0 , . .
Revised: The detailed statistics and visualization of the datasets D LT0 , . .

Original: , D LT8 , D LT are summarized in Tableand Fig.
Revised: , D LT8 , D LT are summarized in Tableand Fig.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: Datasets for generalizability studies We carefully choose the proper datasets to construct the
Revised: Datasets for generalizability studies We carefully choose the proper datasets to construct the

Original: D train and D test for evaluating generalizability of the representation models under multiple settings.
Revised: D train and D test for evaluating generalizability of the representation models under multiple settings.

Original: The choices are summarized in Table 8.
Revised: The choices are summarized in Table 9.

Original: [Table]
Revised: [Table]

Original: C A DDITIONAL R ESULTS ON M ODEL G ENERALIZATION P ERFORMANCE
Revised: C A DDITIONAL R ESULTS ON M ODEL G ENERALIZATION P ERFORMANCE

Original: We now provide more detailed generalization results of models learned with different loss functions, involving supervised cross-entropy (CE) loss, unsupervised contrastive loss (CL) and k -positive contrastive loss (KCL).

Original:  We ﬁrst consider theOOD generalization from one imbalanced dataset to another dataset without overlapping class between them, as discussed in Sec. 4.3. In the experiments, the Split-overlap (split with semantic overlap) scheme is adopted that allows the classes within the two splits to share the same super class ( e.g ., dog and wolf from canidae are put into different splits) in the  ImageNet ontology. As such, though the target classes are all novel to the representation model, some of
Revised: Cross-domain and Cross-task Generalization We evaluate the generalization ability of the representation models trained on the balanced full ImageNet datasets, for cross-domain and cross-task applications.

Original: [Table] their attributes have been seen by the model before from the source classes. The resultsfor the split-overlap are presented in the main text (Table 4) [Table]
Revised: The results are given in Table 10 (cross-domain) and Tables 11 and 12 (for detection) respectively.

Original: We have shown that a balanced feature space gives better generalization ability when when representation learning is performed on imbalanced datasets.

Original: A natural question is whether the conclusion still holds for large-scale balanced datasets, such as ImageNet? To answer this, we split the whole ImageNet into two halves using the above protocol and train the representation model on a balanced source half of ImageNet and evaluate it on the target half. In addition to the Split-overlap
Revised: From Table

Original: [Table]
Revised: [Table]

Original: we consider another split scheme Split-independent (split without semantic overlap) strictly avoids classes from the same super classes to be distributed into different splits.

Original: Split-independent presents a more challenging case for model’s generalization ability as all the target classes (and attributes) to recognize are novel. The generalization performance comparsion of therepresentation models  with different methods is given in Table 9. When the source classes and target classes share similar semantics (on splitoverlap ), the CE-trained model surpasses the CL-trained model on both the source and target classes. But when looking into the generalization gap ( i.e ., the difference between the source and target accuracy), the CL-trained model suffers larger generalization gap than the CL-trained model (10. vs. 1.8).
Revised: when the training datasets are balanced, the models trained with CL and CE achieve comparable performance.

Original: When there is not semantic overlap between source and target (on split-independent ), the

Original: CL-model outperforms CE-model on the target classes by 4.3% with much smaller generalization gap (13.4 v.s. 32.5).

Original: By comparing the “full” performance from split-overlap to split-independent , one can observe that CL loss performs consistently well (58.3 and 58.1), but CE drops as large as 5%.

Original: This implies that CL is robust to imbalanced training distribution used for representation learning, while CE is extremely sensitive to it. These results clearly demonstrate theconsistent superiority of balanced representation learning in terms of generalization for various training dataset distribution. Notably, our proposed KCL loss surpasses both CE and CL loss on all thefour different settings by a large margin (more than 3 points). These results clearly demonstrate that KCL is able to learn a balanced and discriminative feature space, and balancedness is a general property that beneﬁts both balanced and imbalanced datasets.
Revised: While when the training datasets are not balanced, the

Original: C. C ROSS - DOMAIN AND C ROSS - TASK G ENERALIZATION

Original: We also evaluate the generalization ability of the representation models trained on the balanced full

Original: ImageNet datasets, for cross-domain and cross-task applications. The results are given in Table(cross-domain) and Tables 11 and 12 (for detection) respectively. From Table 10, when the training datasets are balanced, the models trained with CL and CE achieve comparable performance. While when the training datasets are not balanced, the CL model signiﬁcantly outperforms the CE model (Table 1).
Revised: CL model signiﬁcantly outperforms the CE model (Table 1).

Original: This demonstrates that the CL loss can consistently produce balanced representation models and the model generalization performance can indeed beneﬁt from being balanced.
Revised: This demonstrates that the CL loss can consistently produce balanced representation models and the model generalization performance can indeed beneﬁt from being balanced.

Original: Similar conclusion can be drawn for the cross-task generalization.
Revised: Similar conclusion can be drawn for the cross-task generalization.

Original: From Tables 11 and 12, when training the model on the full ImageNet dataset, using self-supervised CL loss can produce the model performing slightly better than using the supervised CE loss.
Revised: From Tables 11 and 12, when training the model on the full ImageNet dataset, using self-supervised CL loss can produce the model performing slightly better than using the supervised CE loss.

Original: On PASCAL VOC, the performance advantage is as marginal as 0.02% in AP 50 .
Revised: On PASCAL VOC, the performance advantage is as marginal as 0.02% in AP 50 .

Original: In contrast, when training the model on the ImageNetLT dataset, using CL loss can boost the model performance over using the CE loss much more signiﬁcantly.
Revised: In contrast, when training the model on the ImageNetLT dataset, using CL loss can boost the model performance over using the CE loss much more signiﬁcantly.

Original: The improvement is as large as 1.64% in AP 50 .
Revised: The improvement is as large as 1.64% in AP 50 .

Original: Thus the performance beneﬁt on the generalization to detection brought by CL does not simple stem from using self-supervised pretraining, but indeed come from learning more balanced feature spaces.
Revised: Thus the performance beneﬁt on the generalization to detection brought by CL does not simple stem from using self-supervised pretraining, but indeed come from learning more balanced feature spaces.

Original: Similar to OOD generalization, the model learned with KCL gives better downstream performance on both VOC and COCO, which mean enforcing feature space balancedness with KCL is indeed able to help learning better representation
Revised: Similar to OOD generalization, the model learned with KCL gives better downstream performance on both VOC and COCO, which mean enforcing feature space balancedness with KCL is indeed able to help learning better representation

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]