Paper: Mining latent labels for imbalance classification: a regrouping perspective (ICLR-2024)

Reviewer comment (Experiments):
The efficacy of RG is not well demonstrated. In Table 1, RG only outperforms in one of 3 metrics in one of 2 datasets. For the multi-class problem only one dataset is used.

Author response:
- More datasets: We acknowledge the limitation that we only have one multi-class dataset, and have addressed this issue by extending our experiment to include a new multi-class task. Specifically, we take the financial phasebank dataset (Malo et al. (2014)), which consists of 4840 sentences of English-language financial news categorized by sentiment with a label frequency of 2,879 (neutral), 1,363 (positive), and 604 (negative). We have shown that in the new dataset, RG surpasses other baselines and remains dominant. For more information about the experiment setups and results, we would kindly refer to Appendix C for details.

Reviewer comment (Experiments):
The choice of datasets could be improved. The 9 classes in CIFAR 10 are quite different and merging to simulate a single majority class might have a quite diversified class. For example, a majority class has many images of dogs, but the different kinds (subclasses/clusters) of dogs have commonalities to be dogs. Hence, Binary CIFAR 10 might not be a good dataset to use. Binary HAM10000 on "dermatoscopic images of 7 common skin lesions" is more appropriate.

Author response:
- Choice of dataset for visualization experiments: We appreciate the reviewer’s comment on how binary HAM10000 can form a more realistic dataset for visualization experiments. The choice of the CIFAR-10 dataset for the visualization experiment is based on three arguments:
    - Popularity: CIFAR-10 is one of the most popular datasets in image classification, and the class labels are easily interpretable by the readers, and hence a natural choice for a motivating example
    - Relevance to the regrouping idea: The fact that the majority class is diverse makes it an ideal situation for regrouping. Hence, it helps in highlighting the failure of naively using clustering algorithms to separate samples of different classes, for example in raw image space and autoencoder latent representation space.
    - Difficulty:  As seen from the results in Section 4, the classification of HAM10000 images is a much harder task in comparison to CIFAR10 image classification, and hence presents a more clear motivation for the readers.
	
    While a dataset like CATSvsDOGS could potentially be a more realistic setting for imbalance classification, the lack of fine-grained labels (for example: breeds of the animals) makes it difficult to visualize the effect of regrouping. Additionally, the discrepancy between how DNNs interpret label structures and human understanding further complicates the analysis.

Reviewer comment (Presentation):
The presentation could be improved. For example: Sec 2.2: Sum aggregation was discussed, but it seems to be not used in any experiments. Also, the motivation for Sum aggregation was not discussed.

Author response:
- Presentation: We would like to thank the reviewer for the comments on the presentation, we have addressed them as follows:
    - Aggregation: We performed an ablation study to show the impact aggregation method in Appendix B, where max aggregation is found to be marginally better in comparison to sum aggregation, but since the difference is not too large it is recommended to try both methods. We have added some explanation for both aggregation methods in Sec 2.2 to avoid confusion.

Reviewer comment (Reproducibility):
Sec. 2.2 does not discuss how clusters are formed via regrouping. k-means is mentioned in 4.1 Setting of experiment.

Author response:
- Presentation: We would like to thank the reviewer for the comments on the presentation, we have addressed them as follows:
    - Clustering Method: In Appendix B, we also explored the role of clustering algorithms in the performance of the proposed method, especially focussing on whether enforcing equal cluster sizes can be beneficial to the method. Two balanced clustering algorithms: constrained K-means (Bradley et al. (2000)) and Normalized Cut (Shi et al. (2000)) have been compared with the naive K-means method. We find that balanced clustering can lead to slightly better BA, but perform much worse in terms of AP metric, and hence the choice of a simple K-means algorithm is proposed. To avoid confusion, we have also added brief information about the clustering algorithm in Sec 2.2.

Reviewer comment (Experiments):
RG+WCE: since RG tries to balanced class sizes, why do you need WCE (weighted cross entropy)? Why did WCE help? Could you describe WCE or cite a source?

Author response:
- Why WCE is required with RG?: WCE is the weighted version of cross entropy loss where each sample is weighted by the inverse of the corresponding class frequency, so that classes with fewer samples get higher weights, check Elkan (2001) for the general concept of reweighting. Since we are using K-means algorithm to perform clustering and assign pseudo labels, the resulting classes are not guaranteed to be balanced. Hence, along with CE, we have also reported results that use WCE loss function.

Reviewer comment (Evaluation):
p7. "Considering that AUPRC is agnostic to the decision rule and is often approximated by AP given finite sample sizes"--any citations or evidence to support the statement?

Author response:
- AUPRC/AP metric clarification: AUPRC is a metric that calculates the area under precision-recall curve, where each point is based on a specific threshold (decision rule). Hence, AUPRC (similar to AUROC) is independent of the choice of decision rule. Average Precision (AP) is known to be an unbiased estimate of AUPRC (Boyd et al. (2013)). We have added the reference in the paper as well.

Reviewer comment (Novelty):
The novelty concern can be a big problem. As the authors mentioned about the COG method (local clustering for imbalanced learning in Wu, et. al., 2010), both the proposed method and COG share the same spirit for imbalanced learning, and the technical major difference is COG follows the SVM classifier. Despite being in different data contexts, they are both for imbalanced learning, which weakens the novelty of this work.

Author response:
Novelty: We agree with the reviewer that the proposed method shares a high-level overview with Classification using local clustering (COG), proposed by Wu, et. al. (2010) to tackle imbalanced classification. However, we would like to highlight some key arguments in the paper that differ from COG:
    - Motivation: COG was proposed in the context of linear SVM models in order to: a) decompose the majority class samples so that they become linearly separable from minority class samples, and b) produce sub-classes with relatively uniform sizes. Note that a) is irrelevant in the context of deep neural network (DNN) models, and b) is not supported (in Wu, et. al. (2010)) with any theoretical/empirical evidence. In contrast, our motivation for this method stems from the advantages demonstrated through the use of multiclass labels, for the one-vs-rest CIFAR-10 classification problem, over binary labels.
    - Insights: Wu, et. al. (2010) justify their method by showing how linear decision boundaries between pseudo-classes found by COG can separate linearly inseparable classes. However, for DNN classifiers it is known that even random labels can fit perfectly on a training set (Zhang et al. (2021)), and hence the insight from Wu et al. (2010) is not valid for DNNs. In comparison, in this paper, we show the ability of the proposed method to learn meaningful representations (Figure 3 bottom pane), synchronize training across different labels (Figure 5), and achieve better performance (Figure 6). These insights shed light on the training dynamics in binary vs multiclass setting for the first time to the best of our knowledge, which could also be of independent interest.
    - Implementation: In COG, pseudo labels are assigned by clustering the data in data space, we propose to cluster in the image encoding space. The choice of image encoder is central to the merits of the proposed method – it can be seen in Figure 3 (top pane) that naively clustering the data in raw image space, or even the representation space of autoencoder, will lead to very noisy pseudo labels and would result in poor final performance. We argue that this implementation detail of the proposed method is non-trivial for a practitioner implementing COG method for DNNs.

	To conclude, we argue that this work provides a fresh perspective for exploring and expanding on the method proposed by Wu et al. (2010) as a powerful baseline method to tackle imbalance.

Reviewer comment (Reproducibility):
The technical description is not sufficient, as there is a lack of explanation regarding the clustering methods for pseudo labels that are used in the regrouping method. This also connects to the lack of corresponding experiments to verify the clustering impact on the final performance. Especially, as shown in Figure 5 and Figure 6, how to assign the pseudo labels matters for performance, which makes the readers care about the clustering effectiveness.

Author response:
Clustering Method: We would like to thank the reviewer for pointing out the lack of clarity about the clustering algorithm in the method description, we have updated the paper to make it more clear. Essentially, the proposed method is agnostic to the choice of clustering algorithm, but the conventional K-means algorithm seems to perform well empirically. Note that the proposed method depends heavily on the choice of image encoder (which has been explored in detail in Section 2.1), but less so on the choice of clustering algorithm used in the image encoder space. A natural question could be if enforcing the clustering algorithm to generate relatively balanced clusters could be helpful, which is explored in the ablation study in Appendix B. Two balanced clustering algorithms: constrained K-means(Bradley et al. (2000)) and Normalized Cut (Shi et al. (2000)) have been compared with the naive K-means method. We find that balanced clustering can lead to slightly better BA, but perform much worse in terms of AP metric, and hence the choice of a simple K-means algorithm is proposed.

Reviewer comment (Experiments):
The experiments are not very persuasive although some experiments have shown improvement for RG. The major concern is about the datasets and the baselines, especially for the multi-class classification experiments. There are a range of explorations in long-tailed learning for multi-class classification problems; however, there is no sufficient comparison with recent advances like Decoupling, LA (logit adjustment), BCL, and so on. For the datasets in long-tailed learning, CIFAR100-LT, ImageNet-LT, or INaturalist are all widely adopted benchmarks, which should be included in this submission.

Author response:
Comparison for long-tailed learning dataset: Extension of the proposed method to long-tailed learning setting has been left open for future work. This requires careful handling of the tail classes, and addressing the explosion in numbers of pseudo-classes due to the extremely small sample size in tail classes. We have briefly discussed the merits of exploring classical imbalance classification methods in the last paragraph of Section 3: numerous practical applications still fall under classical imbalance setting, and classical methods often form a base for long-tailed learning methods. The competing methods have also been chosen accordingly. To show that our proposed method can work well in a classical imbalance classification setting, we added an extended experiment using the financial phasebank dataset (Malo et al. (2014)). The details of the experiment setup and results can be found in the revised appendix. For a quick summary, we observed consistent improvements of RG over other baselines, both in terms of BA and AP.

Reviewer comment (Experiments):
There is no experiment that any model that is not data hungry has been applied to compare it with DNN.

Author response:
Gaussian Processes: In this paper, we mainly focus on classification based on deep learning models, especially in the image domain. We could find only two works that address Gaussian Process-based classification for image data (Blomqvist et al. (2020), Van der Wilk et al. (2017)), but their public code repositories have not been maintained and we are unable to implement these methods from scratch due to limited time of the discussion period. However, the method proposed by Wu et al. (2010) is similar to the proposed method for tabular data where results on linear SVM models are provided, which may be of interest to the reviewer.

Reviewer comment (Evaluation):
I would like to see the experiments results with error bars included. For example if you run the experiment n times and calculate standard deviation.

Author response:
Error Bars: We have rerun all the experiments with 3 random seeds and report the mean with standard deviation, please see the updated results in the revised manuscript.

Reviewer comment (Experiments):
I would also like to see how the results change if you apply non-hungry methods such as Gaussian processes.

Author response:
Gaussian Processes: In this paper, we mainly focus on classification based on deep learning models, especially in the image domain. We could find only two works that address Gaussian Process-based classification for image data (Blomqvist et al. (2020), Van der Wilk et al. (2017)), but their public code repositories have not been maintained and we are unable to implement these methods from scratch due to limited time of the discussion period. However, the method proposed by Wu et al. (2010) is similar to the proposed method for tabular data where results on linear SVM models are provided, which may be of interest to the reviewer.

Reviewer comment (Theory):
Sometimes groping the classes to small sub-groups is a difficult task by itself, how do you decide what type of data you can use to have this meaningful sub-groups? What happens if you can't put them into smaller groups?

Author response:
Meaningful subgroups: A potential way to check whether the data can be partitioned into sub-groups is to make use of a visualization tool such as t-SNE plot (For example: As used in Figure 3 top pane). When data cannot be put into meaningful subgroups, it might point to the choice of encoder model (i.e. CLIP model in our case), and other encoder models can be tried.

Reviewer comment (Evaluation):
What are the limitations of your method?

Author response:
Limitations: The key limitation of the proposed method is to handle long-tailed datasets, where the tail classes lie in the few-shot learning setting (i.e. tail classes have few training samples). This leads to an explosion in the number of pseudo-classes, and hence extension to this setting is non-trivial and left open for future work. Another limitation is that the number of pseudo-classes (K) in the current format is a tuning hyper-parameter, a future work could try to deduce this in a more methodological manner to save excess computation for tuning the hyper-parameter.