Original: O N R EPRESENTATION
Revised: O N R EPRESENTATION

Original: L EARNING U NDER
Revised: L EARNING U NDER

Original: C LASS
Revised: C LASS

Original: I MBALANCE
Revised: I MBALANCE

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Unlike carefully curated academic benchmarks, real-world datasets are often highly class-imbalanced, involving training and test sets which contain few examples from certain minority classes.
Revised: Unlike carefully curated academic benchmarks, real-world datasets are often highly class-imbalanced, involving training and test sets which contain few examples from certain minority classes.

Original: While there is a common understanding that neural network generalization is negatively impacted by imbalance, the source of this problem and its resolution are unclear.
Revised: While there is a common understanding that neural network generalization is negatively impacted by imbalance, the source of this problem and its resolution are unclear.

Original: Through extensive empirical investigation, we study foundational learning behaviors for various models such as neural networks, gradient-boosted decision trees, and SVMs across a range of domains and find that (1) contrary to conventional wisdom, re-balancing the training set to include a higher proportion of minority samples degrades performance on imbalanced test sets; (2) minority samples are hard to fit, yet algorithms which fit them, such as oversampling, do not improve generalization.
Revised: Through extensive empirical investigation, we study foundational learning behaviors for various models such as neural networks, gradient-boosted decision trees, and SVMs across a range of domains and find that (1) contrary to conventional wisdom, re-balancing the training set to include a higher proportion of minority samples degrades performance on imbalanced test sets; (2) minority samples are hard to fit, yet algorithms which fit them, such as oversampling, do not improve generalization.

Original: Motivated by the observation that re-balancing class-imbalanced training data is ineffective, we show that several existing techniques for improving representation learning are effective in this setting: (3) self-supervised pre-training is insensitive to imbalance and can be used for feature learning before fine-tuning on labels; (4) Bayesian inference is effective because neural networks are especially underspecified under class imbalance; (5) flatness-seeking regularization pulls decision boundaries away from minority samples, especially when we seek minima that are particularly flat on the minority samples’ loss.
Revised: Motivated by the observation that re-balancing class-imbalanced training data is ineffective, we show that several existing techniques for improving representation learning are effective in this setting: (3) self-supervised pre-training is insensitive to imbalance and can be used for feature learning before fine-tuning on labels; (4) Bayesian inference is effective because neural networks are especially underspecified under class imbalance; (5) flatness-seeking regularization pulls decision boundaries away from minority samples, especially when we seek minima that are particularly flat on the minority samples’ loss.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In real-world data collection scenarios, some events are common while others are exceedingly rare.
Revised: In real-world data collection scenarios, some events are common while others are exceedingly rare.

Original: For example, only a miniscule proportion of credit card transactions are fraudulent, and most cancer screenings come back negative.
Revised: For example, only a miniscule proportion of credit card transactions are fraudulent, and most cancer screenings come back negative.

Original: As a result of this property, machine learning systems are routinely trained and deployed on class-imbalanced data where relatively few samples are associated with certain minority classes , while majority classes dominate the datasets.
Revised: As a result of this property, machine learning systems are routinely trained and deployed on class-imbalanced data where relatively few samples are associated with certain minority classes , while majority classes dominate the datasets.

Original: Nonetheless, the vast majority of works exclusively consider class balanced benchmarks (LeCun, 1998; Krizhevsky, 2009;
Revised: Nonetheless, the vast majority of works exclusively consider class balanced benchmarks (LeCun, 1998; Krizhevsky, 2009;

Original: Deng et al., 2009), including both foundational literature which seeks to understand how and why machine learning algorithms operate as well as applied methodological literature.
Revised: Deng et al., 2009), including both foundational literature which seeks to understand how and why machine learning algorithms operate as well as applied methodological literature.

Original: In this work, we conduct an exploration, on various machine learning approaches including neural networks, gradient-boosted decision trees, and SVMs, of what makes learning under class imbalance so difficult and the associated implications for best practices in such scenarios.
Revised: In this work, we conduct an exploration, on various machine learning approaches including neural networks, gradient-boosted decision trees, and SVMs, of what makes learning under class imbalance so difficult and the associated implications for best practices in such scenarios.

Original: Many of the widely referenced methods for remedying class-imbalance problems rely on modifying how the training data is sampled, such as oversampling or SMOTE (Chawla et al., 2002) and have been shown to be ineffective for neural networks (Buda et al., 2018).
Revised: Many of the widely referenced methods for remedying class-imbalance problems rely on modifying how the training data is sampled, such as oversampling or SMOTE (Chawla et al., 2002) and have been shown to be ineffective for neural networks (Buda et al., 2018).

Original: A common assumption underpinning these sampling methods is that learning under class imbalance is pathological, perhaps even defaulting to predicting only the majority class on all inputs when imbalance is sufficiently severe, so we must intervene by simulating balanced training.
Revised: A common assumption underpinning these sampling methods is that learning under class imbalance is pathological, perhaps even defaulting to predicting only the majority class on all inputs when imbalance is sufficiently severe, so we must intervene by simulating balanced training.

Original: An effect of sampling from minority classes disproportionately more often is that more signal from those classes is injected into model updates, helping to fit the otherwise rarely seen minority samples.
Revised: An effect of sampling from minority classes disproportionately more often is that more signal from those classes is injected into model updates, helping to fit the otherwise rarely seen minority samples.

Original: To tease out exactly why oversampling is ineffective, we begin by studying the relationship between imbalances seen at train and test time, and we investigate whether poor generalization under class-imbalance can really be explained by failures of optimization.
Revised: To tease out exactly why oversampling is ineffective, we begin by studying the relationship between imbalances seen at train and test time, and we investigate whether poor generalization under class-imbalance can really be explained by failures of optimization.

Original: We find that while minority samples are hard to fit, this optimization phenomenon has little explanatory power regarding generalization as fitting them does not affect performance.
Revised: We find that while minority samples are hard to fit, this optimization phenomenon has little explanatory power regarding generalization as fitting them does not affect performance.

Original: Additionally, we find that on the one hand, re-balancing training data to include more minority samples can negatively impact generalization under imbalanced testing, but on the other hand, gathering more majority samples to increase the size of the dataset degrades generalization as well.
Revised: Additionally, we find that on the one hand, re-balancing training data to include more minority samples can negatively impact generalization under imbalanced testing, but on the other hand, gathering more majority samples to increase the size of the dataset degrades generalization as well.

Original: Following our investigation into the role of dataset imbalances in generalization, we show why self-supervised learning (SSL), Bayesian inference, and flatness-seeking regularizers are particularly well-suited for deep learning in class-imbalanced settings.
Revised: Following our investigation into the role of dataset imbalances in generalization, we show why self-supervised learning (SSL), Bayesian inference, and flatness-seeking regularizers are particularly well-suited for deep learning in class-imbalanced settings.

Original: Self-supervised learning algorithms are less sensitive to the proportion of samples in various classes as they do not make use of label information, so we can learn better feature representations via SSL before fine-tuning even on the same data but with labels.
Revised: Self-supervised learning algorithms are less sensitive to the proportion of samples in various classes as they do not make use of label information, so we can learn better feature representations via SSL before fine-tuning even on the same data but with labels.

Original: Previous works have found that the number of high singular values of the Hessian is related to the number of classes (Sagun et al., 2017; Papyan, 2020), but these works were conducted strictly on balanced data.
Revised: Previous works have found that the number of high singular values of the Hessian is related to the number of classes (Sagun et al., 2017; Papyan, 2020), but these works were conducted strictly on balanced data.

Original: In examining such properties on imbalanced datasets, we observe that neural networks trained in such settings are significantly more underdetermined by the data.
Revised: In examining such properties on imbalanced datasets, we observe that neural networks trained in such settings are significantly more underdetermined by the data.

Original: In cases where many parameter settings, and induced functions, are compatible with the data,
Revised: In cases where many parameter settings, and induced functions, are compatible with the data,

Original: Bayesian Neural Networks (BNNs) can represent our uncertainty for improved accuracy (Wilson & Izmailov, 2020; Shwartz-Ziv et al., 2022), and we find that this capability of BNNs is especially advantageous in the class-imbalanced regime.
Revised: Bayesian Neural Networks (BNNs) can represent our uncertainty for improved accuracy (Wilson & Izmailov, 2020; Shwartz-Ziv et al., 2022), and we find that this capability of BNNs is especially advantageous in the class-imbalanced regime.

Original: Finally, whereas neural network decision boundaries tend to hug minority samples in order to expand the margins from majority data points which occur more frequently in training data, we can counteract this behavior with Sharpness-Aware Minimization (SAM) (Foret et al., 2020), and we can further improve margins with respect to minority samples by increasing flatness on their corresponding loss functions.
Revised: Finally, whereas neural network decision boundaries tend to hug minority samples in order to expand the margins from majority data points which occur more frequently in training data, we can counteract this behavior with Sharpness-Aware Minimization (SAM) (Foret et al., 2020), and we can further improve margins with respect to minority samples by increasing flatness on their corresponding loss functions.

Original: In summary, our work questions the motivation of orthodox sampling methods and proposes new directions by which improved representation learning can benefit classifiers in class-imbalanced settings.
Revised: In summary, our work questions the motivation of orthodox sampling methods and proposes new directions by which improved representation learning can benefit classifiers in class-imbalanced settings.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: A long line of research has been conducted on imbalanced classification.
Revised: A long line of research has been conducted on imbalanced classification.

Original: There are several general approaches to address this problem: (1) Re-sampling the data - In early ensemble learning studies, boosting and bagging algorithms were adjusted to take account of imbalanced data by re-sampling.
Revised: There are several general approaches to address this problem: (1) Re-sampling the data - In early ensemble learning studies, boosting and bagging algorithms were adjusted to take account of imbalanced data by re-sampling.

Original: Traditionally, re-sampling involves oversampling minority class samples by simply copying them (Guo & Viktor, 2004; Chawla et al., 2002; Han et al., 2005), or undersampling majority classes by removing samples (Drummond et al., 2003; Hu et al., 2020; Ando & Huang, 2017; Buda et al., 2018), so that minority and majority class samples appear equally frequently in the training process.
Revised: Traditionally, re-sampling involves oversampling minority class samples by simply copying them (Guo & Viktor, 2004; Chawla et al., 2002; Han et al., 2005), or undersampling majority classes by removing samples (Drummond et al., 2003; Hu et al., 2020; Ando & Huang, 2017; Buda et al., 2018), so that minority and majority class samples appear equally frequently in the training process.

Original: (2) Loss re-weighting : Loss re-weighting assigns different weights to majority and minority classes, thus reducing optimization difficulty under class imbalance (Cui et al., 2019; Huang et al., 2019a).
Revised: (2) Loss re-weighting : Loss re-weighting assigns different weights to majority and minority classes, thus reducing optimization difficulty under class imbalance (Cui et al., 2019; Huang et al., 2019a).

Original: For instance, one may scale the loss by inverse class frequency He & Garcia (2009) or re-weight it using the effective number of samples Cui et al.
Revised: For instance, one may scale the loss by inverse class frequency He & Garcia (2009) or re-weight it using the effective number of samples Cui et al.

Original: As an alternative approach, one may focus on hard examples by down-weighing the loss of well-classified examples (Lin et al., 2017) or dynamically re-scaling the cross-entropy loss based on the difficulty of classifying a sample (Ryou et al., 2019).
Revised: As an alternative approach, one may focus on hard examples by down-weighing the loss of well-classified examples (Lin et al., 2017) or dynamically re-scaling the cross-entropy loss based on the difficulty of classifying a sample (Ryou et al., 2019).

Original: Bertsimas et al.
Revised: Bertsimas et al.

Original: (2018) propose to encourage larger margins for rare classes, while Goh & Sim (2010) learn robust features for classify minority classes using class-uncertainty information which approximates Bayesian methods.
Revised: (2018) propose to encourage larger margins for rare classes, while Goh & Sim (2010) learn robust features for classify minority classes using class-uncertainty information which approximates Bayesian methods.

Original: Two-stage fine-tuning and meta-learning approaches:
Revised: Two-stage fine-tuning and meta-learning approaches:

Original: Two-stage methods separate the training process into representation learning and classifier learning (Liu et al., 2019;
Revised: Two-stage methods separate the training process into representation learning and classifier learning (Liu et al., 2019;

Original: Ouyang et al., 2016; Kang et al., 2019; Bansal et al., 2021).
Revised: Ouyang et al., 2016; Kang et al., 2019; Bansal et al., 2021).

Original: In the first stage, the data is unmodified, and no re-sampling or re-weighting is used to train good representations.
Revised: In the first stage, the data is unmodified, and no re-sampling or re-weighting is used to train good representations.

Original: In the second stage, the classifier is balanced by freezing the backbone and fine-tuning the last layers with re-sampling techniques or by learning to debias the confidence.
Revised: In the second stage, the classifier is balanced by freezing the backbone and fine-tuning the last layers with re-sampling techniques or by learning to debias the confidence.

Original: These methods assume that the bias towards majority classes exists only in the classifier layer or that tweaking the classifier layer can correct the underlying biases.
Revised: These methods assume that the bias towards majority classes exists only in the classifier layer or that tweaking the classifier layer can correct the underlying biases.

Original: Several works have also inspected representations learned under class imbalance.
Revised: Several works have also inspected representations learned under class imbalance.

Original: Kang et al.
Revised: Kang et al.

Original: (2019) find that representations learned on class-imbalanced training data via supervised learning perform better when the linear head is fine-tuned on balanced samples.
Revised: (2019) find that representations learned on class-imbalanced training data via supervised learning perform better when the linear head is fine-tuned on balanced samples.

Original: Yang & Xu (2020) instead examine the effect of self- and semi-supervised training on imbalanced data and conclude that imbalanced labels are significantly more useful when accompanied by auxiliary data for semi-supervised learning.
Revised: Yang & Xu (2020) instead examine the effect of self- and semi-supervised training on imbalanced data and conclude that imbalanced labels are significantly more useful when accompanied by auxiliary data for semi-supervised learning.

Original: Kotar et al.
Revised: Kotar et al.

Original: ; Yang & Xu (2020); Liu et al.
Revised: ; Yang & Xu (2020); Liu et al.

Original: (2021) make the observation that self-supervised pre-training is insensitive to imbalance in the upstream training data.
Revised: (2021) make the observation that self-supervised pre-training is insensitive to imbalance in the upstream training data.

Original: These works study SSL pretraining for the purpose of transfer learning, sometimes using linear probes to evaluate the quality of representations.
Revised: These works study SSL pretraining for the purpose of transfer learning, sometimes using linear probes to evaluate the quality of representations.

Original: Inspired by their observations, we find that SSL pre-training
Revised: Inspired by their observations, we find that SSL pre-training

Original: and then fine-tuning on the same exact class-imbalanced dataset can significantly improve generalization.
Revised: and then fine-tuning on the same exact class-imbalanced dataset can significantly improve generalization.

Original:  E XPERIMENTAL S ETUP
Revised:  E XPERIMENTAL S ETUP

Original: Class-imbalance ratio.
Revised: Class-imbalance ratio.

Original: Let r denote the ratio between the number of samples in the rarest class to the number of samples in the most frequent class.
Revised: Let r denote the ratio between the number of samples in the rarest class to the number of samples in the most frequent class.

Original: In the case where r = 1 , the dataset is fully balanced, whereas r = 0 .
Revised: In the case where r = 1 , the dataset is fully balanced, whereas r = 0 .

Original: 1 indicates that there are 10 times as many samples in the majority class as in the minority class.
Revised: 1 indicates that there are 10 times as many samples in the majority class as in the minority class.

Original: In this paper, we will construct and investigate both train and test sets with varying imbalance ratios.
Revised: In this paper, we will construct and investigate both train and test sets with varying imbalance ratios.

Original: It is important to note that if the train dataset ratio is fixed and the test ratio is varied, then we may train a single model but evaluate it on a variety of test sets.
Revised: It is important to note that if the train dataset ratio is fixed and the test ratio is varied, then we may train a single model but evaluate it on a variety of test sets.

Original: On the other hand, if the test ratio is fixed and the training ratio is varied, we must train many models.
Revised: On the other hand, if the test ratio is fixed and the training ratio is varied, we must train many models.

Original: We will vary our setup along both these axes.
Revised: We will vary our setup along both these axes.

Original: Datasets.
Revised: Datasets.

Original: For experiments with neural networks, we use CIFAR-10 (Krizhevsky, 2009) as well as a binary variant in which we simply use two of the CIFAR-10 classes.
Revised: For experiments with neural networks, we use CIFAR-10 (Krizhevsky, 2009) as well as a binary variant in which we simply use two of the CIFAR-10 classes.

Original: For tabular data, we use the Adult dataset and Forest Cover dataset from the UCI Machine Learning Repository (Dua &
Revised: For tabular data, we use the Adult dataset and Forest Cover dataset from the UCI Machine Learning Repository (Dua &

Original: Graff, 2017).
Revised: Graff, 2017).

Original: We adopt the imbalanced datasets of Liu et al.
Revised: We adopt the imbalanced datasets of Liu et al.

Original: (2019) which imbalance classes via exponential distribution, closely reflecting natural long-tailed class distributions.
Revised: (2019) which imbalance classes via exponential distribution, closely reflecting natural long-tailed class distributions.

Original: Models.
Revised: Models.

Original: We use ResNet-34 (He et al., 2016) on CIFAR-10.
Revised: We use ResNet-34 (He et al., 2016) on CIFAR-10.

Original: We use XGBoost (Chen & Guestrin, 2016) and SVM on tabular datasets.
Revised: We use XGBoost (Chen & Guestrin, 2016) and SVM on tabular datasets.

Original: For supervised pre-training, we follow the standard protocol of (Kang et al., 2020).
Revised: For supervised pre-training, we follow the standard protocol of (Kang et al., 2020).

Original: For self-supervised pre-training, we use SimCLR (Chen et al., 2020) and fine-tune as in (Kotar et al., 2021).
Revised: For self-supervised pre-training, we use SimCLR (Chen et al., 2020) and fine-tune as in (Kotar et al., 2021).

Original: For each evaluation in our experiments, we run five seeds and report the mean along with one standard error.
Revised: For each evaluation in our experiments, we run five seeds and report the mean along with one standard error.

Original: Appendix A.1 contains additional details.
Revised: Appendix A.1 contains additional details.

Original:  T HE R OLE OF I MBALANCED D ATA IN G ENERALIZATION AND O PTIMIZATION
Revised:  T HE R OLE OF I MBALANCED D ATA IN G ENERALIZATION AND O PTIMIZATION

Original: In this section, we first explore the impact of training set imbalances on generalization across various imbalanced testing scenarios, finding that re-balancing training data actually harms accuracy under imbalanced testing, across multiple ML models.
Revised: In this section, we first explore the impact of training set imbalances on generalization across various imbalanced testing scenarios, finding that re-balancing training data actually harms accuracy under imbalanced testing, across multiple ML models.

Original: We then find that severe class imbalance makes fitting minority samples difficult, but solving this problem is irrelevant to generalization in neural networks.
Revised: We then find that severe class imbalance makes fitting minority samples difficult, but solving this problem is irrelevant to generalization in neural networks.

Original: Re-balancing the training data and fitting minority samples are both primary effects of oversampling, so these observations will inspire us to look past sampling algorithms in our subsequent search for improved training methods for neural networks.
Revised: Re-balancing the training data and fitting minority samples are both primary effects of oversampling, so these observations will inspire us to look past sampling algorithms in our subsequent search for improved training methods for neural networks.

Original: 4. T HE R ELATIONSHIP B ETWEEN T RAIN -T IME AND T EST -T IME I MBALANCE
Revised: 4. T HE R ELATIONSHIP B ETWEEN T RAIN -T IME AND T EST -T IME I MBALANCE

Original: The literature on training routines for class imbalance in machine learning is filled with methods designed for scenarios in which training data is highly imbalanced, but testing data is balanced.
Revised: The literature on training routines for class imbalance in machine learning is filled with methods designed for scenarios in which training data is highly imbalanced, but testing data is balanced.

Original: However, in industrial applications of machine learning, data encountered during deployment is typically also imbalanced.
Revised: However, in industrial applications of machine learning, data encountered during deployment is typically also imbalanced.

Original: Therefore, we disentangle training and testing balances and investigate how sensitive models are to discrepancies between the two.
Revised: Therefore, we disentangle training and testing balances and investigate how sensitive models are to discrepancies between the two.

Original: This study may be particularly important if one is considering collecting training data for a downstream application.
Revised: This study may be particularly important if one is considering collecting training data for a downstream application.

Original: Should we gather training data with the same balance as we anticipate during testing?
Revised: Should we gather training data with the same balance as we anticipate during testing?

Original: If the data we encounter during deployment is more or less balanced than the training data we gathered, how worried should we be?
Revised: If the data we encounter during deployment is more or less balanced than the training data we gathered, how worried should we be?

Original: A priori, the optimal train set balance is unclear.
Revised: A priori, the optimal train set balance is unclear.

Original: On the one hand, one might want to train on the same distribution as they anticipate testing on.
Revised: On the one hand, one might want to train on the same distribution as they anticipate testing on.

Original: On the other hand, previous studies have suggested that imbalanced training data impedes representation learning and have suggested sampling procedures for simulating more balanced training (Gosain & Sardana, 2017).
Revised: On the other hand, previous studies have suggested that imbalanced training data impedes representation learning and have suggested sampling procedures for simulating more balanced training (Gosain & Sardana, 2017).

Original: To answer this question, we train on datasets with a wide range of imbalance ratios and evaluate each trained model on a variety of testing ratios.
Revised: To answer this question, we train on datasets with a wide range of imbalance ratios and evaluate each trained model on a variety of testing ratios.

Original: We illustrate three scenarios in Figure 1: (1) identical training and testing ratios, (2) balanced training, and (3) the training ratio with the lowest test error (optimal training ratio).
Revised: We illustrate three scenarios in Figure 1: (1) identical training and testing ratios, (2) balanced training, and (3) the training ratio with the lowest test error (optimal training ratio).

Original: We see that training on data with the same imbalance as the testing data is typically superior to training on balanced data, and the two strategies only approach equal performance when the testing data
Revised: We see that training on data with the same imbalance as the testing data is typically superior to training on balanced data, and the two strategies only approach equal performance when the testing data

Original: [Figure]
Revised: [Figure]

Original: becomes balanced.
Revised: becomes balanced.

Original: Moreover, training on data with the same imbalance as the testing data is nearly optimal, so there is very little to gain by balancing the training set.
Revised: Moreover, training on data with the same imbalance as the testing data is nearly optimal, so there is very little to gain by balancing the training set.

Original: To determine the optimal training distribution, we plot the best train ratio versus the best test ratio in
Revised: To determine the optimal training distribution, we plot the best train ratio versus the best test ratio in

Original: Figure 2.
Revised: Figure 2.

Original: In this figure, if there is a perfect match between the training and testing distributions, we would expect points to lie on the diagonal.
Revised: In this figure, if there is a perfect match between the training and testing distributions, we would expect points to lie on the diagonal.

Original: Indeed, the points are close to the diagonal, indicating that it is best to train with a very similar ratio to that of the test dataset, especially for highly imbalanced testing scenarios.
Revised: Indeed, the points are close to the diagonal, indicating that it is best to train with a very similar ratio to that of the test dataset, especially for highly imbalanced testing scenarios.

Original: Interestingly, when the optimal training distribution is not exactly the same as the testing distribution, it is usually very close to it and generally more balanced than the testing distribution.
Revised: Interestingly, when the optimal training distribution is not exactly the same as the testing distribution, it is usually very close to it and generally more balanced than the testing distribution.

Original: Even in cases where the best ratio for training is more balanced, there is minimal difference in test error between the best ratio for training and the test-equivalent ratio for training.
Revised: Even in cases where the best ratio for training is more balanced, there is minimal difference in test error between the best ratio for training and the test-equivalent ratio for training.

Original: This phenomena is consistent across different training methods (including oversampling training
Revised: This phenomena is consistent across different training methods (including oversampling training

Original: Drummond et al.
Revised: Drummond et al.

Original: and
Revised: and

Original: loss re-weighting Cui et al.
Revised: loss re-weighting Cui et al.

Original: This observation calls into question oversampling routines which attempt to improve performance by re-balancing the training data.
Revised: This observation calls into question oversampling routines which attempt to improve performance by re-balancing the training data.

Original: Takeaway: Training with a train set class imbalance similar to that of the test set is near optimal across the ML models and datasets we consider.
Revised: Takeaway: Training with a train set class imbalance similar to that of the test set is near optimal across the ML models and datasets we consider.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: 4. W HEN M ORE D ATA D EGRADES P ERFORMANCE
Revised: 4. W HEN M ORE D ATA D EGRADES P ERFORMANCE

Original: In the previous section, we fixed the total number of training samples and saw that models perform best when trained on a similar data distribution to their testing data.
Revised: In the previous section, we fixed the total number of training samples and saw that models perform best when trained on a similar data distribution to their testing data.

Original: However, in practice, a practitioner likely will not have precise control over the data they collect.
Revised: However, in practice, a practitioner likely will not have precise control over the data they collect.

Original: Will collecting additional samples always help performance?
Revised: Will collecting additional samples always help performance?

Original: If not, practitioners must be cautious when collecting new data that its balance matches their testing data or else the additional data could result in even worse models.
Revised: If not, practitioners must be cautious when collecting new data that its balance matches their testing data or else the additional data could result in even worse models.

Original: Instead of fixing the total number of samples and varying their class ratio, we now fix the number of samples from the minority class and vary the number of others.
Revised: Instead of fixing the total number of samples and varying their class ratio, we now fix the number of samples from the minority class and vary the number of others.

Original: In Figure 3, we see that increasing the number of samples from the majority class, both for neural networks (Figure 3a) and XGBoost (Figure 3b), initially boosts performance on a balanced test set.
Revised: In Figure 3, we see that increasing the number of samples from the majority class, both for neural networks (Figure 3a) and XGBoost (Figure 3b), initially boosts performance on a balanced test set.

Original: Nevertheless, in both cases, the performance reaches an optimum before the ever increasing imbalance in training data eventually degrades test accuracy.
Revised: Nevertheless, in both cases, the performance reaches an optimum before the ever increasing imbalance in training data eventually degrades test accuracy.

Original: Thus, adding training data can be helpful, even without considering the balance of the additional data, but if we add enough samples, we need to be careful not to cause too sharp a mismatch between training and testing distributions.
Revised: Thus, adding training data can be helpful, even without considering the balance of the additional data, but if we add enough samples, we need to be careful not to cause too sharp a mismatch between training and testing distributions.

Original: Notably, the optimal training set ratio is nearly balanced, matching the test set, even when we are allowed to gather extra samples from one class without having to forego samples from another.
Revised: Notably, the optimal training set ratio is nearly balanced, matching the test set, even when we are allowed to gather extra samples from one class without having to forego samples from another.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: 4. M INORITY D ATA IS H ARD TO F IT , BUT F ITTING I T D OES N OT H ELP
Revised: 4. M INORITY D ATA IS H ARD TO F IT , BUT F ITTING I T D OES N OT H ELP

Original: We now investigate optimization on imbalanced data.
Revised: We now investigate optimization on imbalanced data.

Original: Does the classifier correctly label minority training points?
Revised: Does the classifier correctly label minority training points?

Original: Our goal is to determine if low performance under class imbalance can be attributed to poor optimization.
Revised: Our goal is to determine if low performance under class imbalance can be attributed to poor optimization.

Original: This question will inform our future exploration of methods for training on imbalanced data as it will tell us what underlying problem needs solving.
Revised: This question will inform our future exploration of methods for training on imbalanced data as it will tell us what underlying problem needs solving.

Original: In imbalanced training, the vast majority of the gradient signal during training comes from majority class samples since they are seen much more often, making it hard to fit minority samples as shown in Figure 4.
Revised: In imbalanced training, the vast majority of the gradient signal during training comes from majority class samples since they are seen much more often, making it hard to fit minority samples as shown in Figure 4.

Original: In order to re-balance imbalanced training data, practitioners employ various methods, including oversampling and undersampling.
Revised: In order to re-balance imbalanced training data, practitioners employ various methods, including oversampling and undersampling.

Original: These methods sample minority data more often, thus boosting their contribution to the gradient and helping to fit these minority samples.
Revised: These methods sample minority data more often, thus boosting their contribution to the gradient and helping to fit these minority samples.

Original: We confirm in
Revised: We confirm in

Original: Appendix A.2 that such sampling methods indeed help to fit minority samples across models.
Revised: Appendix A.2 that such sampling methods indeed help to fit minority samples across models.

Original: But is the inability to fit minority samples a culprit responsible for worse test accuracy?
Revised: But is the inability to fit minority samples a culprit responsible for worse test accuracy?

Original: To answer the above question, we train our model using two additional methods: (1) oversampling the minority class, and (2) overfitting the training examples using a large number of epochs and a small learning rate.
Revised: To answer the above question, we train our model using two additional methods: (1) oversampling the minority class, and (2) overfitting the training examples using a large number of epochs and a small learning rate.

Original: Both methods successfully fit minority and majority classes.
Revised: Both methods successfully fit minority and majority classes.

Original: Nonetheless, they fail to improve neural network test performance in contrast to XGBoost which benefits from oversampling, as seen in Figure 5.
Revised: Nonetheless, they fail to improve neural network test performance in contrast to XGBoost which benefits from oversampling, as seen in Figure 5.

Original: We include additional details and results for neural networks and
Revised: We include additional details and results for neural networks and

Original: XGBoost, as well as results on imbalanced test sets, in Appendix A.2.
Revised: XGBoost, as well as results on imbalanced test sets, in Appendix A.2.

Original: In light of this study, while minority samples are hard to fit, we will search elsewhere for improving neural network generalization in subsequent sections.
Revised: In light of this study, while minority samples are hard to fit, we will search elsewhere for improving neural network generalization in subsequent sections.

Original: Takeaway: Since minority samples are rarely seen in training, they are hard to fit.
Revised: Takeaway: Since minority samples are rarely seen in training, they are hard to fit.

Original: Nonetheless, strategies such as oversampling which fit these samples do not improve generalization.
Revised: Nonetheless, strategies such as oversampling which fit these samples do not improve generalization.

Original:  C OMBINING S ELF -S UPERVISED L EARNING WITH S UPERVISED F INE -T UNING FOR R OBUST F EATURE L EARNING
Revised:  C OMBINING S ELF -S UPERVISED L EARNING WITH S UPERVISED F INE -T UNING FOR R OBUST F EATURE L EARNING

Original: Self-supervised learning (SSL) has recently achieved great success for representation learning in computer vision, NLP, and tabular data (Chen et al., 2020; Kenton & Toutanova, 2019; Somepalli et al., 2021).
Revised: Self-supervised learning (SSL) has recently achieved great success for representation learning in computer vision, NLP, and tabular data (Chen et al., 2020; Kenton & Toutanova, 2019; Somepalli et al., 2021).

Original: SSL pre-trained networks often exhibit even more transferable representations than their supervised counterparts (Grill et al., 2020).
Revised: SSL pre-trained networks often exhibit even more transferable representations than their supervised counterparts (Grill et al., 2020).

Original: Liu et al.
Revised: Liu et al.

Original: (2021) notes that self-supervised pretraining training strategies for transfer learning are more robust to upstream imbalance than supervised pre-training.
Revised: (2021) notes that self-supervised pretraining training strategies for transfer learning are more robust to upstream imbalance than supervised pre-training.

Original: However, many use-cases for deep learning are not accompanied by massive pre-training datasets.
Revised: However, many use-cases for deep learning are not accompanied by massive pre-training datasets.

Original: We thus propose a two-step procedure in which we first perform SSL pretraining and then supervised fine-tuning, all on the same imbalanced dataset.
Revised: We thus propose a two-step procedure in which we first perform SSL pretraining and then supervised fine-tuning, all on the same imbalanced dataset.

Original: By first learning a feature extractor via SSL, which is insensitive to class imbalance, we can improve the quality of features and as a result generalization too.
Revised: By first learning a feature extractor via SSL, which is insensitive to class imbalance, we can improve the quality of features and as a result generalization too.

Original: [Figure]
Revised: [Figure]

Original: We use SimCLR
Revised: We use SimCLR

Original: ) for self-supervised pre-training and try two fine-tuning routines: (1) fine-tune all layers in an end-to-end fashion and (2) train only a fullyconnected layer on top of the fixed
Revised: ) for self-supervised pre-training and try two fine-tuning routines: (1) fine-tune all layers in an end-to-end fashion and (2) train only a fullyconnected layer on top of the fixed

Original: SSL feature extractor.
Revised: SSL feature extractor.

Original: We train on
Revised: We train on

Original: CIFAR-10 using a wide range of class-imbalance ratios reporting accuracy on a balanced test set, and we compare to three baselines including standard supervised learning, oversampling, and SimCLR ImageNet pre-training.
Revised: CIFAR-10 using a wide range of class-imbalance ratios reporting accuracy on a balanced test set, and we compare to three baselines including standard supervised learning, oversampling, and SimCLR ImageNet pre-training.

Original: The latter baseline uses an extra one million training samples for pre-training, which may not be available for a particular data domain in practice, but will serve as an aspiration for representation learning.
Revised: The latter baseline uses an extra one million training samples for pre-training, which may not be available for a particular data domain in practice, but will serve as an aspiration for representation learning.

Original: In Figure 6, we see that our twostep procedure which uses SSL pre-training on the small imbalanced CIFAR-10 dataset achieves almost as high performance as ImageNet pre-training and far superior performance to standard supervised learning and oversampling, even when the training set is relatively balanced.
Revised: In Figure 6, we see that our twostep procedure which uses SSL pre-training on the small imbalanced CIFAR-10 dataset achieves almost as high performance as ImageNet pre-training and far superior performance to standard supervised learning and oversampling, even when the training set is relatively balanced.

Original: See Appendix
Revised: See Appendix

Original: Appendix A.2 for an identical experiment but where testing data has the same imbalance as the training data.
Revised: Appendix A.2 for an identical experiment but where testing data has the same imbalance as the training data.

Original: We see a similar trend there, namely that SSL pre-training improves generalization significantly.
Revised: We see a similar trend there, namely that SSL pre-training improves generalization significantly.

Original: Takeaway:
Revised: Takeaway:

Original: A two-step process of (a) SSL pre-training on the imbalanced training set and (b) supervised fine-tuning on the same data can boost accuracy across imbalance ratios.
Revised: A two-step process of (a) SSL pre-training on the imbalanced training set and (b) supervised fine-tuning on the same data can boost accuracy across imbalance ratios.

Original:  U NDERSPECIFICATION AND B AYESIAN I NFERENCE
Revised:  U NDERSPECIFICATION AND B AYESIAN I NFERENCE

Original: Expressive models such as neural networks are often capable of representing numerous functions which are each compatible with the training data.
Revised: Expressive models such as neural networks are often capable of representing numerous functions which are each compatible with the training data.

Original: Standard neural network training protocols only choose one of these solutions.
Revised: Standard neural network training protocols only choose one of these solutions.

Original: However, we can better represent this uncertainty using an ensemble.
Revised: However, we can better represent this uncertainty using an ensemble.

Original: Bayesian Neural Networks (BNNs) approach this problem by estimating the average prediction over all parameters weighted by their posterior probabilities conditioned on the training data, p ( y | x, D )
Revised: Bayesian Neural Networks (BNNs) approach this problem by estimating the average prediction over all parameters weighted by their posterior probabilities conditioned on the training data, p ( y | x, D )

Original: = (cid:82) p ( y | x, w ) p ( w |D ) dw .
Revised: = (cid:82) p ( y | x, w ) p ( w |D ) dw .

Original: This approach is particularly effective when the model is underspecified by the data, or in other words when there are many solutions that each achieve a high likelihood (Wilson & Izmailov, 2020).
Revised: This approach is particularly effective when the model is underspecified by the data, or in other words when there are many solutions that each achieve a high likelihood (Wilson & Izmailov, 2020).

Original: In this section, we see that underspecification is particularly strong on class-imbalanced data so that BNNs are especially useful in this setting.
Revised: In this section, we see that underspecification is particularly strong on class-imbalanced data so that BNNs are especially useful in this setting.

Original:  F LATNESS -S EEKING R EGULARIZATION P ULLS D ECISION B OUNDARIES A WAY FROM M INORITY S AMPLES
Revised:  F LATNESS -S EEKING R EGULARIZATION P ULLS D ECISION B OUNDARIES A WAY FROM M INORITY S AMPLES

Original: A number of works have found that the number of high singular values of the loss function Hessian is equal to the number of classes (Sagun et al., 2017; Papyan, 2020).
Revised: A number of works have found that the number of high singular values of the loss function Hessian is equal to the number of classes (Sagun et al., 2017; Papyan, 2020).

Original: Papyan (2020) notes that the theoretical underpinnings of this connection may depend on the training data being class balanced.
Revised: Papyan (2020) notes that the theoretical underpinnings of this connection may depend on the training data being class balanced.

Original: We thus measure the 10 leading singular values of the Hessian on models trained on balanced or highly imbalanced CIFAR-10 data, where the loss function is evaluated on the models’ respective training data.
Revised: We thus measure the 10 leading singular values of the Hessian on models trained on balanced or highly imbalanced CIFAR-10 data, where the loss function is evaluated on the models’ respective training data.

Original: We observe in Figure 7a that imbalanced data leads to lower singular values.
Revised: We observe in Figure 7a that imbalanced data leads to lower singular values.

Original: We also perturb the parameter vector in random directions in Figure 7b using filter-normalization (Huang et al., 2019b) to normalize distance measurements, and we see that the loss increases slightly slower
Revised: We also perturb the parameter vector in random directions in Figure 7b using filter-normalization (Huang et al., 2019b) to normalize distance measurements, and we see that the loss increases slightly slower

Original: [Figure]
Revised: [Figure]

Original: i ngu l a r
Revised: i ngu l a r

Original: V a l u e 50/50 Ratio
Revised: V a l u e 50/50 Ratio

Original: [Figure]
Revised: [Figure]

Original: Perturbation (Filter Normalized)
Revised: Perturbation (Filter Normalized)

Original: [Figure]
Revised: [Figure]

Original: on imbalanced data.
Revised: on imbalanced data.

Original: Note that the volume of a basin scales with its radius raised to a power equal to the dimension of the space.
Revised: Note that the volume of a basin scales with its radius raised to a power equal to the dimension of the space.

Original: As a neural network contains millions of parameters, a very small increase in basin radius will therefore lead to a massive increase in volume (Huang et al., 2019b).
Revised: As a neural network contains millions of parameters, a very small increase in basin radius will therefore lead to a massive increase in volume (Huang et al., 2019b).

Original: Flat minima have the property that their parameters can be perturbed without unfitting the data, indicating that there are many solutions, even locally, which are all consistent with the training samples.
Revised: Flat minima have the property that their parameters can be perturbed without unfitting the data, indicating that there are many solutions, even locally, which are all consistent with the training samples.

Original: But which of these solutions should we choose?
Revised: But which of these solutions should we choose?

Original: By using Bayesian inference methods, we need not commit to only a single one.
Revised: By using Bayesian inference methods, we need not commit to only a single one.

Original: [Figure]
Revised: [Figure]

Original: We now try two such Bayesian inference procedures.
Revised: We now try two such Bayesian inference procedures.

Original: The Laplace approximation (Daxberger et al., 2021) fits a Gaussian distribution with diagonal covariance to the posterior so that lower curvature, which we observed above on imbalanced datasets, will induce higher variance.
Revised: The Laplace approximation (Daxberger et al., 2021) fits a Gaussian distribution with diagonal covariance to the posterior so that lower curvature, which we observed above on imbalanced datasets, will induce higher variance.

Original: SGHMC (Chen et al., 2014) is a Markov chain Monte Carlo sampler which instead introduces noise to the training procedure so that parameters spend more time in places with higher probability under the posterior and samples checkpoints periodically.
Revised: SGHMC (Chen et al., 2014) is a Markov chain Monte Carlo sampler which instead introduces noise to the training procedure so that parameters spend more time in places with higher probability under the posterior and samples checkpoints periodically.

Original: We see in Figure 8 that such methods confer especially large boosts in accuracy on class-imbalanced training data with virtually no additional training cost.
Revised: We see in Figure 8 that such methods confer especially large boosts in accuracy on class-imbalanced training data with virtually no additional training cost.

Original: Takeaway: Under class imbalance, models are particularly underspecified by the data.
Revised: Takeaway: Under class imbalance, models are particularly underspecified by the data.

Original: We can reflect uncertainty over solutions with Bayesian inference, leading to increased accuracy.
Revised: We can reflect uncertainty over solutions with Bayesian inference, leading to increased accuracy.

Original: Sharpness-Aware Minimization (SAM) (Foret et al., 2020) is an optimizer for finding flat minima of the loss function which often generalize better than those found by SGD.
Revised: Sharpness-Aware Minimization (SAM) (Foret et al., 2020) is an optimizer for finding flat minima of the loss function which often generalize better than those found by SGD.

Original: To this end, SAM entails an inner ascent step followed by a descent step which finds parameters such that an ascent step minimally increases the loss.
Revised: To this end, SAM entails an inner ascent step followed by a descent step which finds parameters such that an ascent step minimally increases the loss.

Original: Huang et al.
Revised: Huang et al.

Original: (2019b) connect flat minima to wide margin decision boundaries.
Revised: (2019b) connect flat minima to wide margin decision boundaries.

Original: By plotting the decision boundaries of a small multi-layer perceptron on a toy 2D dataset in Figure 9, we see small margins surrounding minority class samples, and SAM expands these margins supporting the intuition of Huang et al.
Revised: By plotting the decision boundaries of a small multi-layer perceptron on a toy 2D dataset in Figure 9, we see small margins surrounding minority class samples, and SAM expands these margins supporting the intuition of Huang et al.

Original: (2019b).
Revised: (2019b).

Original: Following this observation, we employ SAM on our CIFAR-10 setup and find that SAM especially improves generalization on class-imbalanced training data in Figure 10.
Revised: Following this observation, we employ SAM on our CIFAR-10 setup and find that SAM especially improves generalization on class-imbalanced training data in Figure 10.

Original: In light of this result, we now further increase flatness specifically on minority class loss terms by increasing the ascent step size in SAM’s inner loop.
Revised: In light of this result, we now further increase flatness specifically on minority class loss terms by increasing the ascent step size in SAM’s inner loop.

Original: We see in Figure 10 that this adaptation can yield even greater performance boosts.
Revised: We see in Figure 10 that this adaptation can yield even greater performance boosts.

Original: See Liu et al.
Revised: See Liu et al.

Original: (2021) for an adaptation of SAM which, rather than increasing the size of ascent steps on minority sample loss, instead performs self-supervised pre-training where the ascent step size is related to the density of examples estimated via kernel density estimation.
Revised: (2021) for an adaptation of SAM which, rather than increasing the size of ascent steps on minority sample loss, instead performs self-supervised pre-training where the ascent step size is related to the density of examples estimated via kernel density estimation.

Original: [Figure]
Revised: [Figure]

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: [Figure]
Revised: [Figure]

Original: In this work, we first examined the effects of class-imbalanced data on optimization and generalization.
Revised: In this work, we first examined the effects of class-imbalanced data on optimization and generalization.

Original: The results of these experiments indicated that oversampling solves exactly the wrong problems, namely re-balancing the training data is not helpful, and fitting minority samples does not improve generalization.
Revised: The results of these experiments indicated that oversampling solves exactly the wrong problems, namely re-balancing the training data is not helpful, and fitting minority samples does not improve generalization.

Original: The observation that adding additional majority class samples actually hurts performance raises a critical open problem: how can we best harness extra majority class samples?
Revised: The observation that adding additional majority class samples actually hurts performance raises a critical open problem: how can we best harness extra majority class samples?

Original: Following the above study, we suggest three existing methods for improving performance on classimbalanced data without ad-hoc interventions specific to imbalance: (1) since SSL is insensitive to imbalance, pre-training followed by endto-end fine-tuning can result in enhanced representations; (2) due to amplified underspecification, Bayesian inference is particularly effective on class-imbalanced problems; (3) sharpness-aware minimization can pull decision boundaries away from minority samples.
Revised: Following the above study, we suggest three existing methods for improving performance on classimbalanced data without ad-hoc interventions specific to imbalance: (1) since SSL is insensitive to imbalance, pre-training followed by endto-end fine-tuning can result in enhanced representations; (2) due to amplified underspecification, Bayesian inference is particularly effective on class-imbalanced problems; (3) sharpness-aware minimization can pull decision boundaries away from minority samples.

Original: Given the pervasiveness of imbalanced data in real-world applications, the success of these simple existing methods should redirect our attention away from re-sampling methods and towards better feature representations.
Revised: Given the pervasiveness of imbalanced data in real-world applications, the success of these simple existing methods should redirect our attention away from re-sampling methods and towards better feature representations.

Original: A A PPENDIX A. M ODEL D ETAILS
Revised: A A PPENDIX A. M ODEL D ETAILS

Original: XGBoost : ‘XGBClassifier’ from XGBoost version 1.6.
Revised: XGBoost : ‘XGBClassifier’ from XGBoost version 1.6.

Original: • ‘n estimators’ =
Revised: • ‘n estimators’ =

Original: • ‘subsample’ = 0.
Revised: • ‘subsample’ = 0.

Original: • ‘eta’ =
Revised: • ‘eta’ =

Original: • ‘max depth’ =
Revised: • ‘max depth’ =

Original: SVM : ‘LinearSVM’ from sklearn version 1.1.
Revised: SVM : ‘LinearSVM’ from sklearn version 1.1.

Original: • ‘dual’ = false • ‘max iter’ =
Revised: • ‘dual’ = false • ‘max iter’ =

Original: A. S UPPLEMENTAL F IGURES
Revised: A. S UPPLEMENTAL F IGURES

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure] 
Revised: [Figure] [Figure] [Figure]

Original: [Figure] [Figure] 
Revised: [Figure] [Figure] [Figure]