Paper: Feedback-guided Data Synthesis for Imbalanced Classification (ICLR-2024)

Reviewer comment (Presentation):
The font format of the article is not uniform. Do the words in italics want to express any special meaning? Make it difficult for readers to read.

Author response:
We have followed ICLR 2024's official style guidelines, including the use of italics for introducing either new concepts (e.g., “usefulness”) or emphasizing key points (e.g. “close to the support of the real data distribution”). If any instances create readability issues please let us know, we will modify them. However, these formatting concerns could be easily rectified and do not reflect the quality of the research presented.

Reviewer comment (Presentation):
The charts are mixed up, for example, Figure 5. Is it a table or a graph? The sizes of some pictures also don’t match.

Author response:
We have now made the size of the tables consistent across the paper and modified Figure 5 (now Table 1) to include only the table and moved the figure to appendix.

Reviewer comment (Theory):
How about the time complexity of this method?

Author response:
Thank you for raising this question. We have now included an additional Section (G1) in the Appendix which compares the time-complexity of feedback guidance versus regular sampling.

Reviewer comment (Evaluation):
Are there more evaluation metrics to evaluate the performance of the proposed method versus the baseline method?

Author response:
We used standard metrics used to report results on ImageNet-LT and NICO++ [1, 2] such as average accuracy, many-med-few accuracy and worst group accuracy. In addition to the benchmark’s metrics, in our submission we include standard  generative model metrics to evaluate the effect of our criteria on the quality of the generated images such as FID, coverage and density – see Table 2. Thus, we argue that our evaluation metrics are in-line with both representation learning and generative model literatures. If the reviewer has suggestions on which metrics they would like to see included in the paper, please let us know and we would be happy to include them.

Reviewer comment (Novelty):
The proposed solution for the problem definition is too naïve. For active learning methods, in addition to the confidence-based or entropy-based approach, margin-based approach is also possible. For the recent active learning criteria, such as BALD [1], VAAL [2], or MCDAL [3]. To claim the contribution of a complete research paper, the authors should devise an idea to leverage such recent active learning methods to find more novel solutions suitable for this problem.  
[1] Deep Bayesian Active Learning with Image Data. ICML 2017.  
[2] Variational Adversarial Active Learning. ICCV 2019.  
[3] MCDAL: Maximum Classifier Discrepancy for Active Learning. TNNLS 2022.

Author response:
We would like to thank the reviewer for their insightful review. We are glad to see that the reviewer finds our framework novel. We would like to highlight important differences between Active Learning (AL) and our method. The goal of AL is to interactively query a user (or a model) to label new data points from a given static dataset of unlabelled data points, thus the dataset is known but the labels are unknown. However, in our setup the labels are given and the dataset is unknown. The dataset has to be obtained by sampling from a parametric generative model – our framework describes how to class-conditionally sample from a generative model to obtain useful data points to train a classifier. We agree with the reviewer that our criteria bear some similarities with AL acquisition functions. However, we argue that our main contribution is the design of a queryable framework for generative models that “smartly” balances the training data for a classification model, and not the criteria itself. Note that we do not claim the criteria to be our contribution – see contributions list at the end of the introduction. Moreover, in our experiments we show that the criteria we use are effective enough to obtain state-of-the-art results on three challenging datasets (ImageNet-LT and NICO++ and Places-LT). Thus, we disagree with the main premise of the reviewer’s critique that our simple and effective criteria weakens our main contributions. Contrarily, we see this as a proof of strength for our framework as it can reach the state-of-the-art with simple and broadly used criteria. Thus, we kindly ask the reviewer to reconsider their initial recommendation. Below, we provide detailed responses to the reviewer's questions.

> .. using the recent active learning criteria, such as BALD [1], VAAL [2], or MCDAL [3]

Thank you for suggesting the relevant literature. We have included them in the future work section (6) in our paper. In general, any differentiable criterion which is a function of the classifier can be applied as a feedback guidance in the sampling process, as long as it is computationally efficient. Among the mentioned methods, in theory, both BALD and MCDAL could be applied as a feedback criterion. However, compared to the entropy, they are computationally 10x and 3x more expensive, respectively, limiting their application in practice. The suggested VAAL method is not applicable in our framework since it is task-agnostic and its acquisition function does not depend on the classifier of the task.

Reviewer comment (Novelty):
Instead of simply comparing among naïve active learning criteria, the authors could consider combining multiple losses (at least linear combination in the loss), which would be more novel than the proposed solution.

Author response:
> ..how about combining multiple losses (at least linear combination in the loss)? That would be more novel than the proposed solution.

Thank you for this suggestion. Although our framework has been proven practical and efficient by the extensive experimental analysis provided in the paper, we considered your suggestion and ran an experiment that studies how the combination of different criteria affect the performance of the model. 

In this setup, for the three criteria studied in the paper (loss, hardness, entropy), we generate a dataset of size 1.3M by selecting each criteria with 0.33 probability for generating a sample. We tune the hyper-parameters of the classifier. Below, you can find a table that summarizes the results and compares with previous analysis that we had in the paper. You can also find this result and details in Section G.2 of the paper. Overall, the entropy criterion is proven to be the best in all the datasets that we studied and outperforms using a combination of criteria.


| Method | # Syn. data | Overall | Many | Medium | Few |
| --- | --- | --- | --- | --- | --- |
| LDM-FG (Loss) | 1.3M | 60.41 | 66.14 | 57.68 | 54.1 |
| LDM-FG (Hardness) | 1.3M | 56.70 | 58.07 | 55.38 | 57.32 |
| LDM-FG (Entropy) | 1.3M | **64.7** | **69.8** | **62.3** | **59.1** |
| LDM-FG (Combined) | 1.3M | 62.38 | 67.66 | 59.96 | 56.23 |

Reviewer comment (Experiments):
A comparison with more recent state-of-the-art long-tailed recognition papers is missing. For example, CMO [4] is one of the recent long-tailed recognition methods based on sample synthesis. To claim the usefulness of the proposed method, the authors should compare the proposed method with recent long-tailed recognition papers, including [4].  
[4] The Majority Can Help The Minority: Context-rich Minority Oversampling for Long-Tailed Classification. CVPR 2022.

Author response:
>...authors should compare the proposed method with recent long-tailed recognition papers, including [4]..

Thank you for suggesting relevant literature. We have included [4] in our literature review (section A.2) and compared against it in Table 1. CMO[4] method achieves state of the art results on classes “many” among methods that do not use synthetic data. However, our method with entropy outperforms CMO across all cases, including many, med, few and overall.

Reviewer comment (Experiments):
More analysis of the detailed design choices is needed. For example, how are the hyper-parameters decided, such as w in Eqns (5), (6), (8)? As the authors proposed to add additional criteria, it would be necessary to analyze the effect of w on the performance.

Author response:
> ...how are the hyper-parameters decided..analyze the effect of w on the performance.

As mentioned in the appendix Section G, the feedback guidance coefficient $\omega$ is tuned based on the validation accuracy of the classifier. We have now included the ablation study on $\omega$ for the task of NICO++ in Table 5, in the Appendix Section G.3.
In general, among the several values of $\omega$ that we tried, all of them improved against the case where $\omega=0$. However, better improvement is achieved through careful tuning. Below you can find the summary of results:

| $\omega$  | Worst-group accuracy |
| --- | --- |
| ω=0 | 32.66 ± 1.33 |
| ω=0.01 | 45.60 ± 2.63 |
| ω=0.03 | **49.20 ± 0.97** |
| ω=0.05 | 42.10 ± 1.84 |

Reviewer comment (Experiments):
ImageNet-LT is essentially a pseudo long-tail dataset, where the tail classes may not necessarily be the minority in the actual data distribution. Therefore, generative models can sample relatively well. However, for real-world long-tail distributions, is it also difficult for generative models to obtain sufficiently good samples?

Author response:
Thank you for your valuable feedback. We are glad that you found our experimental results “comprehensive/stunning" and the writing “easy to follow”. Below we clarify your question and hope that you consider increasing your score. 

> ImageNet-LT is essentially a pseudo long-tail dataset, where the tail classes may not necessarily be the minority in the actual data distribution. Therefore, generative models can sample relatively well. However, for real-world long-tail distributions, is it also difficult for generative models to obtain sufficiently good samples?

Thanks for raising this point. As discussed in the limitations paragraphs in the conclusion section of our initial submission: “...our guidance mechanism can only explore the data manifold already captured by the generative model” and this limitation is imposed by the generative models.

Yet, we would like to add that state-of-the-art generative models are capable of generating very unlikely data points under the real world distribution of images - e.g. they are able to generate the rare concept of "an astronaut riding a horse" by combining the common concepts of “astronaut” and “horse”. This is mostly what we observed in our NICO++ experiments, where we saw that generative models can indeed generate new combinations of object and background which is useful to improve generalization.

Our findings on Imagenet-LT, NICO++, and Places-LT (recently added - refer to new results in reviewer’s tjGi answer) show that this inherent potential can be harnessed towards achieving tangible improvements in classification tasks.

Reviewer comment (Experiments):
ImageNet-LT is essentially a pseudo long-tail dataset, where the tail classes may not necessarily be the minority in the actual data distribution. Therefore, generative models can sample relatively well. However, for real-world long-tail distributions, is it also difficult for generative models to obtain sufficiently good samples?

Author response:
Thank you for your valuable feedback. We are glad that you found our experimental results “comprehensive/stunning" and the writing “easy to follow”. Below we clarify your question and hope that you consider increasing your score. 

> ImageNet-LT is essentially a pseudo long-tail dataset, where the tail classes may not necessarily be the minority in the actual data distribution. Therefore, generative models can sample relatively well. However, for real-world long-tail distributions, is it also difficult for generative models to obtain sufficiently good samples?

Thanks for raising this point. As discussed in the limitations paragraphs in the conclusion section of our initial submission: “...our guidance mechanism can only explore the data manifold already captured by the generative model” and this limitation is imposed by the generative models.

Yet, we would like to add that state-of-the-art generative models are capable of generating very unlikely data points under the real world distribution of images - e.g. they are able to generate the rare concept of "an astronaut riding a horse" by combining the common concepts of “astronaut” and “horse”. This is mostly what we observed in our NICO++ experiments, where we saw that generative models can indeed generate new combinations of object and background which is useful to improve generalization.

Our findings on Imagenet-LT, NICO++, and Places-LT (recently added - refer to new results in reviewer’s tjGi answer) show that this inherent potential can be harnessed towards achieving tangible improvements in classification tasks.