Original: Learning Debiased Classifier with Biased Committee
Revised: Learning Debiased Classifier with Biased Committee

Original: Anonymous Authors

Original: Abstract
Revised: Abstract

Original: This paper proposes a new method for training debiased classifiers with no bias supervision.
Revised: This paper proposes a new method for training debiased classifier with no bias supervision.

Original: The key idea of the method is to employ a committee of classifiers as an auxiliary module that identifies bias-conflicting data and assigns large weights to them when training the main classifier.
Revised: The key idea of the method is to employ a committee of classifiers as an auxiliary module that identifies bias-conflicting data and assigns large weights to them when training the main classifier.

Original: The committee is learned as a bootstrapped ensemble so that a majority of its classifiers are biased as well as being diverse, and intentionally fail to predict classes of bias-conflicting data accordingly.
Revised: The committee is learned as a bootstrapped ensemble so that a majority of its classifiers are biased as well as being diverse, and intentionally fail to predict classes of bias-conflicting data accordingly.

Original: The consensus within the committee on prediction difficulty provides a reliable cue for identifying and weighting bias-conflicting data.
Revised: The consensus within the committee on prediction difficulty provides a reliable cue for identifying and weighting bias-conflicting data.

Original: Moreover, the committee is trained also with knowledge transferred from the main classifier so that it gradually becomes debiased and emphasizes more difficult data as training progresses.
Revised: Moreover, the committee is trained also with knowledge transferred from the main classifier so that it gradually becomes debiased and emphasizes more difficult data as training progresses.

Original: On five real-world datasets, our method outperforms previous arts using no bias label like ours and even surpasses those relying on bias labels occasionally.
Revised: On five real-world datasets, our method outperforms previous arts using no bias label like ours and even surpasses those relying on bias labels occasionally.

Original: 1. Introduction
Revised: 1. Introduction

Original: Most supervised learning algorithms for classification rely on the empirical risk minimization (ERM) principle (Vapnik, 1999).
Revised: Most supervised learning algorithms for classification rely on the empirical risk minimization (ERM) principle (Vapnik, 1999).

Original: However, ERM has been known to cause a learned classifier to be biased toward spurious correlations between predefined classes and latent attributes that appear in a majority of training data (Geirhos et al., 2020).
Revised: However, ERM has been known to cause a learned classifier to be biased toward spurious correlations between predefined classes and latent attributes that appear in a majority of training data (Geirhos et al., 2020).

Original: We call data with spurious correlations and holding a majority of training data bias-guiding samples , and the other biasconflicting samples , respectively.
Revised: We call data with spurious correlations and holding a majority of training data bias-guiding samples , and the other biasconflicting samples , respectively.

Original: The issue of model bias has often been addressed by exploiting explicit spurious attribute labels (Kim et al., 2019; Li & Vasconcelos, 2019;
Revised: The issue of model bias has often been addressed by exploiting explicit spurious attribute labels (Kim et al., 2019; Li & Vasconcelos, 2019;

Original: Sagawa et al., 2019;
Revised: Sagawa et al., 2019;

Original: Arjovsky et al., 2019; Teney et al., 2021;
Revised: Arjovsky et al., 2019; Teney et al., 2021;

Original: Tartaglione et al., 2021; Zhu et al., 2021) or knowledge about bias types given a priori (Bahng et al., 2020).
Revised: Tartaglione et al., 2021; Zhu et al., 2021) or knowl- edge about bias types given a priori (Bahng et al., 2020).

Original: However, these methods are impractical because such su- pervision and prior knowledge are costly, and the methods demand extensive post hoc analysis.
Revised: However, these methods are impractical because such supervision and prior knowledge are costly, and the methods demand extensive post hoc analysis.

Original: Hence, a body of research has been conducted for learning debiased classifiers with no additional label for spurious attributes (Wang et al., 2021;
Revised: Hence, a body of research has been conducted for learning debiased classifiers with no additional label for spurious attributes (Wang et al., 2021;

Original: Levy et al., 2020; Liu et al., 2021b;
Revised: Levy et al., 2020; Liu et al., 2021b;

Original: Nam et al., 2020; Kim et al., 2021; Lee et al., 2021).
Revised: Nam et al., 2020; Kim et al., 2021; Lee et al., 2021).

Original: A common approach in this line of work is to employ an intentionally biased classifier as an auxiliary module (Liu et al., 2021b;
Revised: A common approach in this line of work is to employ an intentionally biased classifier as an auxiliary module (Liu et al., 2021b;

Original: Nam et al., 2020; Kim et al., 2021; Lee et al., 2021):
Revised: Nam et al., 2020; Kim et al., 2021; Lee et al., 2021):

Original: Samples that the biased classifier has trouble handling are regarded as bias-conflicting ones and assigned large weights when used for training the main classifier to reduce the effect of bias-guiding counterparts.
Revised: Samples that the biased classifier has trouble handling are regarded as bias-conflicting ones and assigned large weights when used for training the main classifier to reduce the effect of bias-guiding counterparts.

Original: Although it has driven remarkable success, this approach has drawbacks due to the use of a single biased classifier.
Revised: Although it has driven remarkable success, this approach has drawbacks due to the use of a single biased classifier.

Original: First, the quality of the biased classifier could vary by hyper-parameters (Liu et al., 2021a) and its initial parameter values (Fort et al., 2019).
Revised: First, the quality of the biased classifier could vary by hyper-parameters (Liu et al., 2021a) and its initial parameter values (Fort et al., 2019).

Original: Further, data that the biased classifier fails to handle could include not only bias-conflicting samples but also biasguiding ones, which differs by the quality of the classifier.
Revised: Further, data that the biased classifier fails to handle could include not only bias-conflicting samples but also biasguiding ones, which differs by the quality of the classifier.

Original: These drawbacks limit the reliability and performance of debiasing methods depending on a single biased classifier, which is demonstrated in Appendix A.1.
Revised: These drawbacks limit the reliability and performance of debiasing methods depending on a single biased classifier, which is demonstrated in Appendix A.1.

Original: To overcome these limitations, we propose a new method using a committee of biased classifiers as the auxiliary module, coined learning with biased committee (LBC).
Revised: To overcome these limitations, we propose a new method using a committee of biased classifiers as the auxiliary module, coined learning with biased committee (LWBC).

Original: LBC identifies bias-conflicting samples and determines their weights through consensus on their prediction difficulty within the committee.
Revised: LWBC identifies bias-conflicting samples and determines their weights through consensus on their prediction difficulty within the committee.

Original: The committee is built as a bootstrapped ensemble, i .
Revised: The committee is built as a bootstrapped ensemble, i .

Original: e ., each of its classifiers is trained from a randomly sampled subset of the entire training dataset.
Revised: e ., each of its classifiers is trained from a randomly sampled subset of the entire training dataset.

Original: This strategy not only guarantees the diversity among the classifiers, but also lets a majority of the classifiers be biased.
Revised: This strategy not only guarantees the diversity among the classifiers, but also lets a majority of the classifiers be biased.

Original: Accordingly, a majority of the committee tends to classify bias-guiding samples correctly and fail to deal with bias-conflicting ones.
Revised: Accordingly, a majority of the committee tends to classify bias-guiding samples correctly and fails to deal with bias-conflicting ones.

Original: The consensus on prediction difficulty within the committee thus gives a strong cue for identifying and weighting biasconflicting samples.
Revised: The consensus on prediction difficulty within the committee thus gives a strong cue for identifying and weighting bias-conflicting samples.

Original: Also, using the consensus of multiple classifiers enables LBC to be robust to the varying quality of individual classifiers and consequently to focus more precisely on bias-conflicting samples.
Revised: Also, using the consensus of multiple classifiers enables LWBC to be robust to the varying quality of individual classifiers and consequently to focus more precisely on bias-conflicting samples.

Original: Moreover, unlike the biased classifier trained independently of the main classifier in the previous work, the committee

Original: [Figure]
Revised: [Figure]

Original: conflicting
Revised: : Bias-guiding sample : Bias-conflicting sample [Figure] Sample

Original: [Figure]
Revised: [Figure]

Revised: Moreover, unlike the biased classifier trained independently of the main classifier in the previous work, the committee in

Original: in LBC is trained with knowledge of the main classifier as well as the random subsets of training data.
Revised: LWBC is trained with knowledge of the main classifier as well as the random subsets of training data.

Original: Specifically, the knowledge is distilled in the form of classification logits of the main classifier (Hinton et al., 2015), and each classifier of the committee utilizes the knowledge as pseudo labels of training data other than its own training set.
Revised: Specifically, the knowledge is distilled in the form of classification logits of the main classifier (Hinton et al., 2015), and each classifier of the committee utilizes the knowledge as pseudo labels of training data other than its own training set.

Original: We expect that this strategy allows the committee to become debiased gradually so that it does not give large weights to easy biasconflicting samples, i .
Revised: We expect that this strategy allows the committee to become debiased gradually so that it does not give large weights to easy biasconflicting samples, i .

Original: e .
Revised: e .

Original: , those already well handled by the main classifier, and focuses more on difficult ones.
Revised: , those already well handled by the main classifier, and focuses more on difficult ones.

Original: Finally, we further improve the proposed method by adopting a self-supervised representation as the frozen backbone of the committee and the main classifier.
Revised: Finally, we further improve the proposed method by adopting a self-supervised representation as the frozen backbone of the committee and the main classifier.

Original: Since selfsupervised learning is not dependent on class labels, it is less affected by the spurious correlations between classes and latent attributes, leading to a robust and less-biased representation.
Revised: Since selfsupervised learning is not dependent on class labels, it is less affected by the spurious correlations between classes and latent attributes, leading to a robust and less-biased representation.

Original: Also, by installing the committee and the main classifier on top of the representation, the classifiers can be implemented efficiently in both space and time while enjoying the rich and bias-free features given by the backbone.
Revised: Also, by installing the committee and the main classifier on top of the representation, the classifiers can be implemented efficiently in both space and time while enjoying the rich and bias-free features given by the backbone.

Original: LBC is validated extensively on five real-world datasets.
Revised: LWBC is validated extensively on five real-world datasets.

Original: It substantially outperforms existing methods using no bias label and even occasionally surpasses previous arts demanding bias labels.
Revised: It substantially outperforms existing methods using no bias label and even occasionally surpasses previous arts demanding bias labels.

Original: We also demonstrate that all of the main components contribute to the outstanding performance.
Revised: We also demonstrate that all of the main components contribute to the outstanding performance.

Original: The main contribution of this paper is three-fold:
Revised: The main contribution of this paper is three-fold:

Original: • We present LBC, a new method for learning a debiased classifier with no spurious attribute label.
Revised: • We present LWBC, a new method for learning a debiased classifier with no spurious attribute label.

Original: The use of consensus within the committee allows LBC to address the limitations of previous work.
Revised: The use of consensus within the committee allows LWBC to address the limitations of previous work.

Original: • We propose to learn the committee using knowledge of the main classifier, unlike the previous work whose auxiliary modules do not consider the main classifier.
Revised: • We propose to learn the committee using knowledge of the main classifier, unlike the previous work whose auxiliary modules do not consider the main classifier.

Original: • LBC demonstrates superior performance on five realworld datasets.
Revised: • LWBC demonstrates superior performance on five realworld datasets.

Original: It outperforms existing methods using no additional supervision like ours and even surpasses those relying on spurious attribute labels occasionally.
Revised: It outperforms existing methods using no additional supervision like ours and even surpasses those relying on spurious attribute labels occasionally.

Original: 2. Proposed method
Revised: 2. Proposed method

Original: LBC first learns a feature representation with self supervision, which is used as the frozen backbone providing rich and bias-free features to downstream modules (Section 2.1).
Revised: LWBC first learns a feature representation with self supervision, which is used as the frozen backbone providing rich and bias-free features to downstream modules (Section 2.1).

Original: Next, it trains a committee of m auxiliary classifiers f 1 , f 2 , . . .
Revised: Next, it trains a committee of m auxiliary classifiers f 1 , f 2 , . . .

Original: , f m and the main classifier g on top of the self-supervised representation (Section 2.2); thanks to the self-supervised representation, the classifiers are designed concisely, using only two fully-connected layers for each.
Revised: , f m and the main classifier g on top of the self-supervised representation (Section 2.2); thanks to the self-supervised representation, the classifiers are designed concisely, using only two fully-connected layers for each.

Original: The committee identifies bias-conflicting samples and assigns them large weights to reduce the effect of bias-guiding samples during the training of the main classifier.
Revised: The committee identifies bias-conflicting samples and assigns them large weights to reduce the effect of bias-guiding samples during the training of the main classifier.

Original: To this end, the committee is trained as a bootstrapped ensemble of classifiers so that a majority of its classifiers are biased as well as diverse, and intentionally fail to predict classes of bias-conflicting samples accordingly.
Revised: To this end, the committee is trained as a bootstrapped ensemble of classifiers so that a majority of its classifiers are biased as well as diverse, and intentionally fail to predict classes of bias-conflicting samples accordingly.

Original: The consensus within the committee on prediction difficulty of a sample ( e .
Revised: The consensus within the committee on prediction difficulty of a sample ( e .

Original: g ., the number of classifiers that fail to predict its class label) thus indicates how much likely the sample is bias-conflicting, and is used to compute weights for training samples.
Revised: g ., the number of classifiers that fail to predict its class label) thus indicates how much likely the sample is bias-conflicting, and is used to compute weights for training samples.

Original: Moreover, the committee is trained also with knowledge of the main classifier so that it gradually becomes debiased along with the main classifier and emphasizes more difficult samples as training progresses.
Revised: Moreover, the committee is trained also with knowledge of the main classifier so that it gradually becomes debiased along with the main classifier and emphasizes more difficult samples as training progresses.

Original: Note that the committee is an auxiliary module used only in training and thus does not impose additional computation or memory footprint in testing.
Revised: Note that the committee is an auxiliary module used only in training and thus does not impose additional computation or memory footprint in testing.

Original: The overall process of LBC is illustrated in Figure 1, and the following sections elaborate on each step of LBC.
Revised: The overall process of LWBC is illustrated in Figure 1, and the following sections elaborate on each step of LWBC.

Original: 2.1. Self-supervised representation learning
Revised: 2.1. Self-supervised representation learning

Original: As the feature extractor, we train a backbone network by selfsupervised learning with BYOL (Grill et al., 2020) on the target dataset.
Revised: As the feature extractor, we train a backbone network by selfsupervised learning with BYOL (Grill et al., 2020) on the target dataset.

Original: A self-supervised model can capture diverse patterns shared by data without being biased towards a particular class even the training set is biased.
Revised: A self-supervised model can capture diverse patterns shared by data without being biased towards a particular class even the training set is biased.

Original: We empirically demonstrate that adopting a self-supervised representation leads to a model less biased compared with a supervised representation.
Revised: We empirically demonstrate that adopting a self-supervised representation leads to a model less biased compared with a supervised representation.

Original: Although the representation offers rich and less-biased features, the main classifier can be still biased when it is trained by ERM; the need to explore a debiasing method for a classifier arises.
Revised: Although the representation offers rich and less-biased features, the main classifier can be still biased when it is trained by ERM; the need to explore a debiasing method for a classifier arises.

Original: We thus propose LBC, a new debiasing method illustrated in the next section.
Revised: We thus propose LWBC, a new debiasing method illustrated in the next section.

Original: 2.2. Learning debiased classifier with biased committee
Revised: 2.2. Learning debiased classifier with biased committee

Original: We randomly sample m subsets of the same size, denoted by S 1 , ..., S m , from the training set D with replacement.
Revised: We randomly sample m subsets of the same size, denoted by S 1 , ..., S m , from the training set D with replacement.

Original: Then m auxiliary classifiers f 1 , ..., f m of the committee are initialized randomly, and each of the subsets S l is assigned to each auxiliary classifier f l as its training data.
Revised: Then m auxiliary classifiers f 1 , ..., f m of the committee are initialized randomly, and each of the subsets S l is assigned to each auxiliary classifier f l as its training data.

Original: The first step of LBC is warm-up training of the committee; this is required to ensure that the committee is capable of identifying and weighting bias-conflicting samples at the beginning of the main training process.
Revised: The first step of LWBC is warm-up training of the committee; this is required to ensure that the committee is capable of identifying and weighting bias-conflicting samples at the beginning of the main training process.

Original: Given a mini-batch
Revised: Given a mini-batch

Original: B at each warm-up iteration, the committee is trained by minimizing the cross-entropy loss below:
Revised: B at each warm-up iteration, the committee is trained by minimizing the cross-entropy loss below:

Original: [Equation]
Revised: [Equation]

Original: Since each subset is sampled from the training set dominated by bias-guiding samples, a majority of auxiliary classifiers are also biased.
Revised: Since each subset is sampled from the training set dominated by bias-guiding samples, a majority of auxiliary classifiers are also biased.

Original: At the same time, the classifiers are diverse due to their difference in initialization and training data.
Revised: At the same time, the classifiers are diverse due to their difference in initialization and training data.

Original: After the warm-up stage, the main classifier and the committee are trained while interacting with each other.
Revised: After the warm-up stage, the main classifier and the committee are trained while interacting with each other.

Original: First, the main classifier is trained by the weighted cross entropy loss with the entire training set, where the sample weights are computed by considering consensus within the committee on prediction difficulty of the samples.
Revised: First, the main classifier is trained by the weighted cross entropy loss with the entire training set, where the sample weights are computed by considering consensus within the committee on prediction difficulty of the samples.

Original: Since a majority of auxiliary classifiers have trouble to handle bias-conflicting samples, we identity and weight bias-conflicting samples based on the number of auxiliary classifiers predicting correctly for the samples.
Revised: Since a majority of auxiliary classifiers have trouble to handle bias-conflicting samples, we identity and weight bias-conflicting samples based on the number of auxiliary classifiers predicting correctly for the samples.

Original: The weight function w is given by
Revised: The weight function w is given by

Original: [Equation]
Revised: [Equation]

Original: where m is the size of the committee, f l is the l -th classifier of the committee, and α is a scale hyper-parameter.
Revised: where m is the size of the committee, f l is the l -th classifier of the committee, and α is a scale hyper-parameter.

Original: The weight reflects how much the sample x is likely to be bias-conflicting and decreases rapidly when the number of correctly predicting classifiers increases.
Revised: The weight reflects how much the sample x is likely to be bias-conflicting and decreases rapidly when the number of correctly predicting classifiers increases.

Original: Then we train the main classifier g with emphasis on the bias-conflicting samples through w by the weighted cross entropy loss below:
Revised: Then we train the main classifier g with emphasis on the bias-conflicting sam- [Table]

Original: [Equation]
Revised: [Equation]

Original: where B is the mini-batch.
Revised: where B is the mini-batch.

Original: [Table]

Original: During training, as the main classifier is gradually debiased, samples useful for debiasing the main classifier change accordingly.
Revised: During training, as the main classifier is gradually debiased, samples useful for debiasing the main classifier change accordingly.

Original: To focus more on bias-conflicting samples difficult for the main classifier, we inform the quality of the main classifier to the committee by distilling the knowledge of the main classifier in the form of its classification logits (Hinton et al., 2015) and transferring the knowledge by minimizing the following KD loss:
Revised: To focus more on bias-conflicting samples difficult for the main classifier, we inform the quality of the main classifier to the committee by distilling the knowledge of the main classifier in the form of its classification logits (Hinton et al., 2015) and transferring the knowledge by minimizing the following KD loss:

Original: [Equation]
Revised: [Equation]

Original: where ‘soft’ is softmax function and τ is a temperature parameter.
Revised: where ‘soft’ is softmax function and τ is a temperature parameter.

Original: Note that we apply L KD to the complement set of S l to avoid auxiliary classifiers in the committee being identical to each other.
Revised: Note that we apply L KD to the complement set of S l to avoid auxiliary classifiers in the committee being identical to each other.

Original: By interacting with the main classifier, the committee gradually becomes debiased along with the main classifier.
Revised: By interacting with the main classifier, the committee gradually becomes debiased along with the main classifier.

Original: Hence, samples correctly predicted by the main classifier are less weighted and those with incorrect predictions are more weighted by the committee.
Revised: Hence, samples correctly predicted by the main classifier are less weighted and those with incorrect predictions are more weighted by the committee.

Original: After the warm-up training, the main and the auxiliary classifiers are alternately updated with a given mini-batch at each iteration.
Revised: After the warm-up training, the main and the auxiliary classifiers are alternately updated with a given mini-batch at each iteration.

Original: First, we forward every sample in a mini-batch to each auxiliary classifier and compute weights of the samples through Eq.
Revised: First, we forward every sample in a mini-batch to each auxiliary classifier and compute weights of the samples through Eq.

Original: With the weights, the main classifier is updated by Eq.
Revised: With the weights, the main classifier is updated by Eq.

Original: (3) and the knowledge of the updated main classifier is transferred to the auxiliary classifiers.
Revised: (3) and the knowledge of the updated main classifier is transferred to the auxiliary classifiers.

Original: The auxiliary classifiers are in turn updated by a linear combination of the losses in Eq.
Revised: The auxiliary classifiers are in turn updated by a linear combination of the losses in Eq.

Original: (1) and (4).
Revised: (1) and (4).

Original: We also provide the full description of our algorithm in Appendix A.2.
Revised: We also provide the full description of our algorithm in Appendix A.2.

Original: 3. Experiments 3.1. Evaluation metrics
Revised: 3. Experiments 3.1. Evaluation metrics

Original: We adopt six metrics.
Revised: We adopt six metrics.

Original: V ALIDATION / T EST : average accuracy on validation / test splits.
Revised: V ALIDATION / T EST : average accuracy on validation / test splits.

Original: G UIDING : average accuracy  on bias guiding samples per class.
Revised: G UIDING : average accuracy [Table] [Table] on bias guiding samples per class.

Original: C ONFLICTING : average accuracy on bias conflicting samples per class.
Revised: C ONFLICTING : average accuracy on bias conflicting samples per class.

Original: U NBIASED : average of ‘Guiding’ and ‘Conflicting’ per class.
Revised: U NBIASED : average of ‘Guiding’ and ‘Conflicting’ per class.

Original: W ORSTGROUP : minimum average accuracy of group.
Revised: W ORSTGROUP : minimum average accuracy of group.

Original: We also provide detailed experimental settings in Appendix A.3 and description of datasets in Appendix A.4.[Table] [Table] [Table]
Revised: We also provide detailed experimental settings in Appendix A.3 and description of datasets in Appendix A.4.

Original: [Table]

Original: 3.2. Quantitative results
Revised: 3.2. Quantitative results

Original: LBC shows superior classification accuracy among the methods that do not use the spurious attribute label on the five real-world datasets.
Revised: LWBC shows superior classification accuracy among the methods that do not use the spurious attribute label on the five real-world datasets.

Original: In Tables 1 &
Revised: In Tables 1 &

Original: [Table]
Revised: [Table]

Original: we observe LBC outperforms existing debiasing methods using no spurious attribute label and shows comparable C ONFLICTING performance with methods that exploit spurious attribute labels on CelebA, which reflects gender bias in the real world.
Revised: we observe LWBC outperforms existing debiasing methods using no spurious attribute label and shows comparable C ONFLICTING performance with methods that exploit spurious attribute labels on CelebA, which reflects gender bias in the real world [Table]

Original:  Especially, the gap between the average accuracy of groups and worst-group accuracy of LBC is much smaller than the other methods, this means that our model fairly predicts a sample belongs to each group.
Revised: [Table] [Table] [Table] Especially, the gap between the average accuracy of groups and worst-group accuracy of LWBC is much smaller than the other methods, this means that our model fairly predicts a sample belongs to each group.

Original: Table 3 shows the results on the ImageNet-9, which is dominated by texture bias, and the ImageNet-A, which is regarded as a bias-conflicting set of ImageNet.
Revised: Table 3 shows the results on the ImageNet-9, which is dominated by texture bias, and the ImageNet-A, which is regarded as a bias-conflicting set of ImageNet.

Original: LBC is 9.7% better than CaaM
Revised: LWBC is 9.7% better than CaaM

Original: (Woo et al.,

Original: [Table]

Original: [Table] [Figure] 2018) on the ImageNet-A dataset, i .
Revised: (Woo et al., 2018) on the ImageNet-A dataset, i .

Original: e ., LBC is robust to texture bias and various ImageNet biases.
Revised: e ., LWBC is robust to texture bias and various ImageNet biases.

Original: In Table 4, LBC is 18.6% better than LDD, which is the previous state of the art.
Revised: In Table 4, LWBC is 18.6% better than LDD, which is the previous state of the art.

Original: Compared to LfF (Nam et al., 2020) and LDD (Lee et al., 2021), which are debiasing methods with a single biased classifier, we demonstrate that learning a debiased classifier with the biased committee is more effective than a single classifier scheme.
Revised: Compared to LfF (Nam et al., 2020) and LDD (Lee et al., 2021), which are debiasing methods with a single biased classifier, we demonstrate that learning a debiased classifier with the biased committee is more effective than a single classifier scheme.

Original: In Table 5, LBC is 13.3% better than CaaM, i .
Revised: In Table 5, LWBC is 13.3% better than CaaM, i .

Original: e ., LBC is well generalized to the unseen spurious attributes.
Revised: e ., LWBC is well generalized to the unseen spurious attributes.

Original: Note that the validation and test set of NICO have unseen context classes and are unbiased.
Revised: Note that the validation and test set of NICO have unseen context classes and are unbiased.

Original: We also provide ablation studies in Appendix A.5 and qualitative results in
Revised: Importance of each module in LWBC. Table 6 demonstrates through ablation studies (Liu et al., 2021b; Nam et al., 2020; Kim et al., 2021; Lee et al., 2021): (1) learning from

Revised: [Figure]

Revised: a single biased classifier, (2) learning with committee, and (3) transferring the knowledge of the main classifier.

Revised: First, we train a classifier by ERM (row 1) then assign a weight value 50 to the wrongly predicted samples and 1 to other samples.

Revised: Then re-training the classifier with the weights (row 2).

Revised: Comparing these two results demonstrates that up-weighting scheme with a biased classifier is effective to debiasing a classifier.

Revised: Then we increase the number of biased classifiers and compute the sample weights using our weight function Eq.

Revised: (2) (row 3).

Revised: Learning with the committee shows a remarkable improvement in the worst-group accuracy.

Revised: Moreover, knowledge distillation that enables the committee to interact with the main classifier further improves performance (row 4).

Revised: Effectiveness of transferring knowledge of the main classifier.

Revised: Figure 2(a) shows the range of unbiased validation accuracy of classifiers in the committee and unbiased validation accuracy of the main classifier during training.

Revised: The mean accuracy of classifiers in the committee gradually increases following the accuracy of the main classifier.

Revised: Also, the accuracy of the main classifier gradually increases as training progresses.

Revised: On the other hand, the accuracy of classifiers in the committee trained without KD loss does not increase or even decrease.

Revised: Number of classifiers.

Revised: We compare the results of the main classifier trained with the different number of auxiliary classifiers.

Revised: Figure 2(b) shows the worst-group accuracy versus the number of auxiliary classifiers m on CelebA. With a single auxiliary classifier, the main classifier shows the lowest worst-group accuracy, but the accuracy increases as m increases.

Revised: When larger than 40, the number of classifiers has little effect on learning the main classifier.

Revised: Impact of bootstrapping.

Revised: In Table 7, we study the impact of auxiliary classifiers trained on a subset of the training set (‘subset’) and auxiliary classifiers trained on the full training set (‘full’).

Revised: Each auxiliary classifier in the ‘full’ setting learns from the same data, but they are differently initialized.

Revised: Unlike LWBC, the KD loss in Eq.

Revised: (2) is calculated using the entire training set for the ‘full’ experiment.

Revised: So, the comparison between ‘full’ and ‘subset’ shows the impact

Original: Appendix A.6.
Revised: [Table] [Figure] [Table] [Figure]

Revised: [Table]

Revised: of bootstrapping excluding the impact of random initialization.

Revised: As shown in Table 7, learning with auxiliary classifiers trained on the same full training dataset degrades the performance at every metric.

Revised: We believe that this is because the auxiliary classifiers trained on a subset of the training set are more diverse and biased than those trained on the full training set.

Revised: We also provide more ablation studies in

Revised: Appendix A.5 and qualitative results in Appendix A.6.

Original: 4. Conclusion
Revised: 4. Conclusion

Original: We have proposed a new method for learning a debiased classifier with a committee of auxiliary classifiers.
Revised: We have proposed a new method for learning a debiased classifier with a committee of auxiliary classifiers.

Original: The committee is learned in a way that consensus on predictions of its classifiers to identify and weight bias-conflicting data.
Revised: The committee is learned in a way that consensus on predictions of its classifiers to identify and weight bias-conflicting data.

Original: The main debiased classifier is then trained with an emphasis on the bias-conflicting data.
Revised: The main debiased classifier is then trained with an emphasis on the bias-conflicting data.

Original: Moreover, we demonstrated that self-supervised learning is a solid yet unexplored baseline for debiasing.
Revised: Moreover, we demonstrated that self-supervised learning is a solid yet unexplored baseline for debiasing.

Original: Coupled with a self-supervised feature extractor, our method achieved state-of-the-art by large margins on real-world datasets.
Revised: Coupled with a self-supervised feature extractor, our method achieved state-of-the-art by large margins on real-world datasets.

Revised: Acknowledgement

Revised: This work was supported by IITP grant funded by the Korea government (MSIT)

Revised: (No. 2022-0-00926).

Original: A. Appendix A.1. Limitations of using a single biased classifier
Revised: A. Appendix A.1. Limitations of using a single biased classifier

Original: We empirically demonstrate the limitations of debiasing methods depending on an single biased classifier in Figure 3.
Revised: We empirically demonstrate the limitations of debiasing methods depending on an single biased classifier in Figure 4.

Original: Figure 3(a) shows the ratio of bias-conflicting samples to all incorrectly predicted by a single biased classifier.
Revised: Figure 4(a) shows the ratio of bias-conflicting samples to all incorrectly predicted by a single biased classifier.

Original: The ratio highly fluctuates by the learning rate of the classifier and varies up to 4%p due to different initialization even with a fixed learning rate, meaning that using a single biased classifier is sensitive to hyper-parameters.
Revised: The ratio highly fluctuates by the learning rate of the classifier and varies up to 4%p due to different initialization even with a fixed learning rate, meaning that using a single biased classifier is sensitive to hyper-parameters.

Original: Figure 3(b) is a matrix of disagreement on predictions of biased classifiers.
Revised: Figure 4(b) is a matrix of disagreement on predictions of biased classifiers.

Original: For pairs of biased classifiers initialized differently, we measure the number of bias-conflicting samples for which the classifiers predict differently.
Revised: For pairs of biased classifiers initialized differently, we measure the number of bias-conflicting samples for which the classifiers predict differently.

Original: The results suggest that individual biased classifiers are sensitive to initialization.
Revised: The results suggest that individual biased classifiers are sensitive to initialization.

Original: Figure 3(c) shows comparisons between a single biased classifier and a committee of biased classifiers.
Revised: Figure 4(c) shows comparisons between a single biased classifier and a committee of biased classifiers.

Original: Higher enrichment implies more precise mining and weighting of bias-conflicting samples (Liu et al., 2021a).
Revised: Higher enrichment implies more precise mining and weighting of bias-conflicting samples (Liu et al., 2021a).

Original: The committee clearly outperforms the single biased classifier in terms of enrichment.
Revised: The committee clearly outperforms the single biased classifier in terms of enrichment.

Original: A.2. Algorithm for LBC
Revised: A.2. Algorithm for LWBC

Original: The overall process of LBC is given formally in Algorithm 1.
Revised: The overall process of LWBC is given formally in Algorithm 1.

Original: A.3. Experimental setup
Revised: A.3. Experimental setup

Original: Implementation details.
Revised: Implementation details.

Original: For the training self-supervised model, we train ResNet-18 (He et al., 2016) following selfsupervised learning with BYOL (Grill et al., 2020) on the target dataset.
Revised: For the training self-supervised model, we train ResNet-18 (He et al., 2016) following selfsupervised learning with BYOL (Grill et al., 2020) on the target dataset.

Original: During the self-supervised learning, a random patch of input image is cropped, resized to 224 × 224 pixels, flipped horizontally at random, and distorted by a random sequence of

Original: [Table] [Equation] [Table] brightness, contrast, saturation, hue adjustments, and grayscale conversion; since the color information is key feature to classify HairColor class on the celebA dataset, we adopt only brightness and contrast as color distortion when it trains celebA. Since NICO and BAR are very small dataset, we train self-supervised model from ImageNet (Deng et al., 2009) pretrained parameters as an initial point.
Revised: During the self-supervised learning, a random patch of input image is cropped, resized to 224 × 224 pixels, flipped horizontally at random, and distorted by a random sequence of brightness, contrast, saturation, hue adjustments, and grayscale conversion; since the color information is key feature to classify HairColor class on the celebA dataset, we adopt only brightness and contrast as color distortion when it trains celebA. Since NICO and BAR are very small dataset, we train self-supervised model from ImageNet (Deng et al., 2009) pretrained parameters as an initial point.

Original: Except for the experiments using BAR and NICO, we train self-supervised models from scratch.
Revised: Except for the experiments using BAR and NICO, we train self-supervised models from scratch.

Original: We use the self-supervised ResNet-18 as a backbone network except for the last fully connected layer.
Revised: We use the self-supervised ResNet-18 as a backbone network except for the last fully connected layer.

Original: We set the batch size to { 64, 64, 128, 256 } , learning rate { 1e-3, 1e-3, 1e-4, 6e-3 } , the size of the committee m to { 30, 30, 30, 40 } , the size of subset S l to { 10, 10, 80, 300 } , λ to { 0.9, 0.6, 0.6, 0.6 } , and τ to { 1, 1, 1, 2.5 } , respectively for { BAR, NICO,
Revised: We set the batch size to { 64, 64, 128, 256 } , learning rate { 1e-3, 1e-3, 1e-4, 6e-3 } , the size of the committee m to { 30, 30, 30, 40 } , the size of subset S l to { 10, 10, 80, 300 } , λ to { 0.9, 0.6, 0.6, 0.6 } , and τ to { 1, 1, 1, 2.5 } , respectively for { BAR, NICO,

Original: Imagenet-9, CelebA } and α to 0.02.
Revised: Imagenet-9, CelebA } and α to 0.02.

Original: Note that we average the accuracy of each training over three independent trials with random seeds.
Revised: Note that we average the accuracy of each training over three independent trials with random seeds.

Original: A.4. Datasets
Revised: A.4. Datasets

Original: CelebA. CelebA
Revised: CelebA. CelebA

Original: (Liu et al., 2015) is a dataset for face recognition where each sample is labeled with 40 attributes.
Revised: (Liu et al., 2015) is a dataset for face recognition where each sample is labeled with 40 attributes.

Original: Following the experiment configuration suggested by Nam et al.
Revised: Following the experiment configuration suggested by Nam et al.

Original: (Nam et al., 2020), we focus on HairColor and HeavyMakeup attributes that are spuriously correlated with Gender attributes, i .
Revised: (Nam et al., 2020), we focus on HairColor and HeavyMakeup attributes that are spuriously correlated with Gender attributes, i .

Original: e ., most of the CelebA images with blond hair are women.
Revised: e ., most of the CelebA images with blond hair are women.

Original: As a result, the biased model suffers from performance degradation when predicting HairColor attribute on males.
Revised: As a result, the biased model suffers from performance degradation when predicting HairColor attribute on males.

Original: Therefore, we use HairColor as the target attribute and Gender as a spurious attribute, the same as HeavyMakeup .
Revised: Therefore, we use HairColor as the target attribute and Gender as a spurious attribute, the same as HeavyMakeup .

Original: ImageNet-9.
Revised: ImageNet-9.

Original: ImageNet-9 (Ilyas et al., 2019) is a subset of ImageNet (Russakovsky et al., 2015) containing nine super-classes.
Revised: ImageNet-9 (Ilyas et al., 2019) is a subset of ImageNet (Russakovsky et al., 2015) containing nine super-classes.

Original: Following the setting adopted by (Bahng et al., 2020), we conduct experiments with 54,600 training images and 2, validation images.
Revised: Following the setting adopted by (Bahng et al., 2020), we conduct experiments with 54,600 training images and 2, validation images.

Original: ImageNet-9 tempts to have a correlated object class and image texture.
Revised: ImageNet-9 tempts to have a correlated object class and image texture.

Original: We follow the evaluation scheme adopted by (Bahng et al., 2020), and we report the unbiased accuracy of the validation set, which is computed as average accuracy on every object-texture combination.
Revised: We follow the evaluation scheme adopted by (Bahng et al., 2020), and we report the unbiased accuracy of the validation set, which is computed as average accuracy on every object-texture combination.

Original: ImageNet-A. ImageNet-A (Hendrycks et al., 2021) contains real-world images misclassified by ImageNet-trained ResNet 

Revised: ImageNet-A. ImageNet-A (Hendrycks et al., 2021) contains real-world images misclassified by ImageNet-trained ResNet

Original: (He et al., 2016).
Revised: (He et al., 2016).

Original: Since such failure is caused when a model too heavily relies on the color, textures, and backgrounds.
Revised: Since such failure is caused when a model too heavily relies on the color, textures, and backgrounds.

Original: The ImageNet-A dataset could be a bias-conflicting set w.r.t.
Revised: The ImageNet-A dataset could be a bias-conflicting set w.r.t.

Original: various ImageNet biases.
Revised: various ImageNet biases.

Original: This dataset is used only for evaluating a model trained on ImageNet-9.
Revised: This dataset is used only for evaluating a model trained on ImageNet-9.

Original: BAR.
Revised: BAR.

Original: The Biased Action Recognition (BAR) dataset (Nam et al., 2020) is a real-world image dataset designed to evaluate the spurious correlation between human action and place on real-world images.
Revised: The Biased Action Recognition (BAR) dataset (Nam et al., 2020) is a real-world image dataset designed to evaluate the spurious correlation between human action and place on real-world images.

Original: Originally the given training set of BAR consists of only 100% bias-guiding samples, and its test set consists of only bias-conflicting samples.
Revised: Originally the given training set of BAR consists of only 100% bias-guiding samples, and its test set consists of only bias-conflicting samples.

Original: In our setting, we use 10% of the original BAR training set as validation and set the bias-conflicting ratio of the training set to 1%.
Revised: In our setting, we use 10% of the original BAR training set as validation and set the bias-conflicting ratio of the training set to 1%.

Original: NICO.
Revised: NICO.

Original: NICO (He et al., 2021) is a real-world dataset for simulating the out-of-distribution image classification task.
Revised: NICO (He et al., 2021) is a real-world dataset for simulating the out-of-distribution image classification task.

Original: Following the setting used by Wang et al.
Revised: Following the setting used by Wang et al.

Original: (Wang et al., 2021), we use an animal subset of NICO, which is labeled withobject and 10 context classes for evaluating the debiasing methods.
Revised: (Wang et al., 2021), we use an animal subset of NICO, which is labeled withobject and 10 context classes for evaluating the debiasing methods.

Original: The training set consists 7 context classes per object class and they are long-tailed distributed (e.g., most dog images are appeared ‘on grass’ the others are appeared on 6 contexts).
Revised: The training set consists 7 context classes per object class and they are long-tailed distributed (e.g., most dog images are appeared ‘on grass’ the others are appeared on 6 contexts).

Original: The validation and test set consists of 10 context classes with 3 unseen context classes per object class.
Revised: The validation and test set consists of 10 context classes with 3 unseen context classes per object class.

Original: We verify the ability of debiasing a model from object-context correlations by evaluating on NICO.[Figure]
Revised: We verify the ability of debiasing a model from object-context correlations by evaluating on NICO.

Original: A.5. Ablation study
Revised: A.5. Ablation study

Original: Self-supervised representation as a solid baseline.
Revised: Self-supervised representation as a solid baseline.

Original: We empirically investigate the potential of self-supervised representation as a solid baseline for the debiasing task.
Revised: We empirically investigate the potential of self-supervised representation as a solid baseline for the debiasing task.

Original: We train a classifier on the top of the self-supervised representation by ERM.
Revised: We train a classifier on the top of the self-supervised representation by ERM.

Original: The results are denoted by ‘SSL+ERM’ and compared with ‘ERM’, which is a fully supervised classification model in
Revised: The results are denoted by ‘SSL+ERM’ and compared with ‘ERM’, which is a fully supervised classification model in

Original: Table 1, 2, 3, 4, 5.
Revised: Table 1, 2, 3, 4, 5.

Original: ‘SSL+ERM’ outperforms not only ERM but also the previous state-of-the-art on all the datasets except for CelebA. ‘SSL+ERM’ is less biased than the model trained by fully supervised learning.
Revised: ‘SSL+ERM’ outperforms not only ERM but also the previous state-of-the-art on all the datasets except for CelebA. ‘SSL+ERM’ is less biased than the model trained by fully supervised learning.

Original: Importance of each module in LBC.

Original: Table 6 demonstrates through ablation studies (Liu et al., 2021b;

Original: Nam et al., 2020;

Original: Kim et al., 2021; Lee et al., 2021): (1) learning from a single biased classifier, (2) learning with committee, and (3) transferring the knowledge of the main classifier.

Original: First, we train a classifier by ERM (row 1) then assign a weight value 50 to the wrongly predicted samples and 1 to other samples.

Original: Then re-training the classifier with the weights (row 2).

Original: Comparing these two results demonstrates that up-weighting scheme with a biased classifier is effective to debiasing a classifier.

Original: Then we increase the number of biased classifiers and compute the sample weights using our weight function

Original: Eq.

Original: (2) (row 3).

Original: Learning with the committee shows a remarkable improvement in the worst-group accuracy.

Original: Moreover, knowledge distillation that enables the committee to interact with the main classifier further improves performance (row 4).

Original: Effectiveness of transferring knowledge of the main classifier.

Original: Figure 4(a) shows the range of unbiased validation accuracy of classifiers in the committee and unbiased validation accuracy of the main classifier during training.

Original: The mean accuracy of classifiers in the committee gradually increases following the accuracy of the main classifier.

Original: Also, the accuracy of the main classifier gradually increases as training progresses.

Original: On the other hand, the accuracy of classifiers in the committee trained without KD loss does not increase or even decrease.

Original: Number of classifiers.

Original: We compare the results of the main classifier trained with the different number of auxiliary classifiers.

Original: Figure 4(b) shows the worst-group accuracy versus the number of auxiliary classifiers m on CelebA. With a single auxiliary classifier, the main classifier shows the lowest worst-group accuracy, but the accuracy increases as m increases.

Original: When larger than 40, the number of classifiers has little effect on learning the main classifier.

Original: [Table]

Original: Impact of bootstrapping.

Original: In Table 7, we study the impact of auxiliary classifiers trained on a subset of the training set (‘subset’) and auxiliary classifiers trained on the full training set (‘full’).

Original: Each auxiliary classifier in the ‘full’ setting learns from the same data, but they are differently initialized.

Original: Unlike LBC, the KD loss in Eq.

Original: (2) is calculated using the entire training set for the ‘full’ experiment.

Original: So, the comparison between ‘full’ and ‘subset’ shows the impact of bootstrapping excluding the impact of random initialization.

Original: As shown in Table 7, learning with auxiliary classifiers trained on the same full training dataset degrades the performance at every metric.

Original: We believe that this is because the auxiliary classifiers trained on a subset of the training set are more diverse and biased than those trained on the full training set

Original: [Table]

Original: [Table]

Original: pretrained

Original: [Table]

Original: Impact of backbone.
Revised: Impact of backbone.

Original: In Table 8, we study the impact of a frozen backbone trained by self-supervised learning (row 7-9) compared to supervised learning (row 1-3 for ERM backbone and row 4-6 for ImageNet pretrained backbone).
Revised: In Table 8, we study the impact of a frozen backbone trained by self-supervised learning (row 7-9) compared to supervised learning (row 1-3 for ERM backbone and row 4-6 for ImageNet pretrained backbone).

Original: Within the results using the same backbone, learning with the committee and transferring the knowledge of the main classifier to the committee improve performance in all metrics compared with the ERM classifier, regardless of the backbone.
Revised: Within the results using the same backbone, learning with the committee and transferring the knowledge of the main classifier to the committee improve performance in all metrics compared with the ERM classifier, regardless of the backbone.

Original: Regarding the performance of the ERM classifier on top of each backbone (row 1, 4, 7), the ERM backbone leads to the best performance among the three backbones since the ERM backbone is trained with class labels.
Revised: Regarding the performance of the ERM classifier on top of each backbone (row 1, 4, 7), the ERM backbone leads to the best performance among the three backbones since the ERM backbone is trained with class labels.

Original: However, the ERM backbone was not useful when coupled with our method (learning with the committee and KD) dedicated to debiasing.
Revised: However, the ERM backbone was not useful when coupled with our method (learning with the committee and KD) dedicated to debiasing.

Original: This shows the limitation of conventional representation based on supervised learning.
Revised: This shows the limitation of conventional representation based on supervised learning.

Revised: Comparing ImageNet pretrained backbone and self-supervised

Original: Comparing ImageNet pretrained backbone and self-supervised trained backbone (both are target-label-free schemes), the backbone trained by self-supervised learning is always better than the ImageNet pretrained backbone in our experiments.
Revised: [Table] ± [Table] trained backbone (both are target-label-free schemes), the backbone trained by self-supervised learning is always better than the ImageNet pretrained backbone in our experiments.

Original: We believe that this is because a frozen backbone trained by self-supervised learning on a target dataset gives rich and bias-free features.
Revised: We believe that this is because a frozen backbone trained by self-supervised learning on a target dataset gives rich and bias-free features.

Original: Surprisingly, the main classifier learned by the committee and KD on top of ImageNet pretrained frozen backbone using the same hyper-parameters outperforms LfF (Nam et al., 2020), which demonstrates that the advantage of our method is not limited to a specific backbone network.
Revised: Surprisingly, the main classifier learned by the committee and KD on top of ImageNet pretrained frozen backbone using the same hyper-parameters outperforms LfF (Nam et al., 2020), which demonstrates that the advantage of our method is not limited to a specific backbone network.

Original: A.6. Qualitative results A.6.1. C LASS ACTIVATION MAP
Revised: A.6. Qualitative results A.6.1. C LASS ACTIVATION MAP

Original: Figure 5 shows the class activation map (CAM) of the main classifier, those of auxiliary classifiers of the committee, and a consensus graph on a bias-guiding sample of CelebA, and Figure 6 shows them on a bias-conflicting sample of CelebA. We mark a classifier that correctly predicts the class of the sample in ‘correct’, otherwise ‘incorrect’.
Revised: Figure 5 shows the class activation map (CAM) of the main classifier, those of auxiliary classifiers of the committee, and a consensus graph on a bias-guiding sample of CelebA, and Figure 6 shows them on a bias-conflicting sample of CelebA. We mark a classifier that correctly predicts the class of the sample in ‘correct’, otherwise ‘incorrect’.

Original: In Figure 5, a majority of auxiliary classifiers correctly predict the class of the bias-guiding sample, but they focus on facial appearance.
Revised: In Figure 5, a majority of auxiliary classifiers correctly predict the class of the bias-guiding sample, but they focus on facial appearance.

Original: Since auxiliary classifiers have a consensus on ‘correct’, the main classifier less focus on the sample during training.
Revised: Since auxiliary classifiers have a consensus on ‘correct’, the main classifier less focus on the sample during training.

Original: In Figure 6, a majority of auxiliary classifiers wrongly predict the class of the bias-conflicting sample because they focus on facial appearance.
Revised: In Figure 6, a majority of auxiliary classifiers wrongly predict the class of the bias-conflicting sample because they focus on facial appearance.

Original: Since auxiliary classifiers have a consensus on ‘incorrect’, the main classifier focuses more on the sample during training.
Revised: Since auxiliary classifiers have a consensus on ‘incorrect’, the main classifier focuses more on the sample during training.

Original: The main classifier does not focus on facial appearance to correctly predict both the bias-guiding and bias-conflicting samples.
Revised: The main classifier does not focus on facial appearance to correctly predict both the bias-guiding and bias-conflicting samples.

Original: As we expected, a majority of auxiliary classifiers focus on facial appearance, i .
Revised: As we expected, a majority of auxiliary classifiers focus on facial appearance, i .

Original: e ., auxiliary classifiers exploit gender feature rather than HairColor feature to classify an image.
Revised: e ., auxiliary classifiers exploit gender feature rather than HairColor feature to classify an image.

Original: However, the main classifier focuses more on HairColor feature than the auxiliary classifiers.
Revised: However, the main classifier focuses more on HairColor feature than the auxiliary classifiers.

Original: A.6.2. Q UALITATIVE EXAMPLES WITH WEIGHTS
Revised: A.6.2. Q UALITATIVE EXAMPLES WITH WEIGHTS

Original: Figure 7, 8, 9 show the graph of weight function in Eq. 2 and examples on weight values.
Revised: Figure 7, 8, 9 show the graph of weight function in Eq. 2 and examples on weight values.

Original: As illustrated on Figure 7, 8, 9, our method not only up-weights the bias-conflicting samples and down-weights the bias-guiding samples but also imposes finer weight according to difficulty of a sample.Main
Revised: As illustrated on Figure 7, 8, 9, our method not only up-weights the bias-conflicting samples and down-weights the bias-guiding samples but also imposes finer weight according to difficulty of a sample.

Original: [Figure]
Revised: [Figure]

Original: correctcorrect incorrect
Revised: correct incorrect

Original: [Figure]
Revised: [Figure]

Original: Aux0: correct
Revised: Aux0: correct

Original: Aux1: correct
Revised: Aux1: correct

Original: Aux2: correct Aux3: correct
Revised: Aux2: correct Aux3: correct

Original: Aux4: incorrect
Revised: Aux4: incorrect

Original: Aux5: correct
Revised: Aux5: correct

Original: Aux6: incorrect
Revised: Aux6: incorrect

Original: Aux7: correct Aux8: correct
Revised: Aux7: correct Aux8: correct

Original: Aux9: correct
Revised: Aux9: correct

Original: Aux10: correct
Revised: Aux10: correct

Original: Aux11: correct Aux12: incorrect Aux13: correct Aux14: incorrect Aux15: correct
Revised: Aux11: correct Aux12: incorrect Aux13: correct Aux14: incorrect Aux15: correct

Original: Aux16: incorrect Aux17: incorrect Aux18: correct Aux19: incorrect Aux20: incorrect Aux21: correct
Revised: Aux16: incorrect Aux17: incorrect Aux18: correct Aux19: incorrect Aux20: incorrect Aux21: correct

Original: Aux22: correct
Revised: Aux22: correct

Original: Aux23: correct
Revised: Aux23: correct

Original: Aux24: correct
Revised: Aux24: correct

Original: Aux25: correct
Revised: Aux25: correct

Original: Aux26: correct
Revised: Aux26: correct

Original: Aux27: correct Aux28: incorrect Aux29: correct Aux30: incorrect Aux31: correct
Revised: Aux27: correct Aux28: incorrect Aux29: correct Aux30: incorrect Aux31: correct

Original: Aux32: correct
Revised: Aux32: correct

Original: Aux33: correct
Revised: Aux33: correct

Original: Aux34: correct Aux35: correct
Revised: Aux34: correct Aux35: correct

Original: Aux36: correct
Revised: Aux36: correct

Original: Aux37: correct
Revised: Aux37: correct

Original: Aux38: correct Aux39: incorrect
Revised: Aux38: correct Aux39: incorrect

Original: [Figure]
Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Original: The ratio of correctly predicted classifiers
Revised: The ratio of correctly predicted classifiers

Original: S a m p l e w e i gh t
Revised: S a m p l e w e i gh t

Original: The ratio of correctly predicted classifiers
Revised: The ratio of correctly predicted classifiers

Original: The ratio of correctly predicted classifiers
Revised: The ratio of correctly predicted classifiers

Original: S a m p l e w e i gh t
Revised: S a m p l e w e i gh t

Original: The ratio of correctly predicted classifiers
Revised: The ratio of correctly predicted classifiers

Original: [Figure]
Revised: [Figure]

Original: The ratio of correctly predicted classifiers
Revised: The ratio of correctly predicted classifiers

Revised: [Figure]

Original: S a m p l e w e i g h t
Revised: m [Figure] l e w e i g h t

Original: The ratio of correctly predicted classifiers
Revised: The ratio of correctly predicted classifiers