Paper: Pareto Deep Long-Tailed Recognition: A Conflict-Averse Solution (ICLR-2024)

Reviewer comment (Novelty):
The paper introduces some trivial factors, e.g. sharpness-aware minimization, which makes the contribution of this paper ad-hoc. In my humble opinion, sharpness-aware minimization does not show strong connection to the proposed MOO framework, and can be integrated to almost all the long-tailed learning baselines.

Author response:
We acknowledge that SAM has been a technique commonly used to augment existing approaches. However, its effectiveness in the context of DLTR is not as significant as anticipated. While a previous study [1] demonstrated the effectiveness of incorporating SAM into certain DLTR methods, it was limited to a few baseline models (CE, LDAM, VS). However, recent research [2] indicates that SAM does not consistently improve a wide range of DLTR approaches. This suggests that the integration of SAM and DLTR requires a specific design rather than a straightforward application.
In our paper, we employ SAM based on the observation that the convergence of CAGrad cannot be guaranteed in DLTR scenarios, particularly for tail classes that are known to exhibit sharp minima (larger H values). Although H-Lipschitz is typically assumed to hold, it fails to do so in DLTR scenarios and must be addressed. Therefore, we utilize SAM to address the challenges posed by integrating DLTR with MOO, rather than treating it as a mere technique or trick.

Reference:

[1] Harsh Rangwani, Sumukh K Aithal, Mayank Mishra, et al. Escaping saddle points for effective generalization on class-imbalanced data. Advances in Neural Information Processing Systems, 35: 22791–22805, 2022.

[2] Zhipeng Zhou, Lanqing Li, Peilin Zhao, Pheng-Ann Heng, and Wei Gong. Class-conditional sharpness-aware minimization for deep long-tailed recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3499–3509, June 2023.

Reviewer comment (Novelty):
The technical contribution of this paper seems not strong. This paper mainly adopts CAGRAD with slight modification on the training schedule (temporal design). Besides, sharpness-aware minimization is directly applied and the variability collapse loss is only a regularization term on the standard deviation of the loss.

Author response:
We would like to emphasize that this paper makes significant contributions in three key aspects. Firstly, we provide a comprehensive observation that identifies an intrinsic conflict issue present in current DLTR approaches during the early stages of representation learning. This novel observation sheds light on an unexplored aspect of DLTR research.
Secondly, we propose a framework, referred to as the temporal design, which enables the application of MOO to DLTR methods, effectively addressing the aforementioned conflict issue. This framework serves as a bridge between the distinct regimes of MOO-based MTL and DLTR. Furthermore, it holds potential for further enhancement through the incorporation of advanced MOO approaches. Lastly, we introduce two novel theoretical perspectives, i.e., SAM and variability collapse loss, to tackle the convergence and generalization challenges that arise when integrating DLTR with MOO. While these designs are not entirely novel, their purpose and re-discovery within the context of MOO-based DLTR scenarios are indeed novel and well motivated, as highlighted in the original main text.

Reviewer comment (Experiments):
The effectiveness of the proposed PLOT on top of baseline methods shows significant variability, see Table 1-2. However, the authors only conduct experiments on top of cRT + Mixup/MiSLAS on ImageNet-LT, Places-LT, iNaturalist, thus casting doubts on the confidence of the results.

Author response:
We acknowledge that the improvements in Table 1-2 show variability, which is in turns aligns with our conclusion, i.e., PLOT bring more improvements to those that have more serious conflict issue since a key contribution of PLOT is just to address the conflict issue for representation learning of DLTR. Besides, we have to clarify that most of the DLTR methods haven’t released their official implementations for large-scale datasets, making it hard for us to apply PLOT on them. But we would like to provide you some additional empirical evidence to demonstrate the effectiveness of PLOT on large-scale datasets in the next few days.

Reviewer comment (Theory):
According to Theorem 4.1, the generalization bound is bounded by the weighted intra-class loss variability, i.e., $w_KM$. It raises the question of whether uniformly constraining the intra-class loss variability is the optimal solution. Besides, in the neural collapse theory, minimizing loss function (e.g. cross-entropy loss) can lead to the intra-class collapse at the optimal case, why there should be an explicit constraint on intra-class variability?

Author response:
Although uniformly imposing intra-class loss variability may appear to offer an optimal solution, achieving it becomes challenging in imbalanced scenarios due to the unstable variability across different categories caused by significant differences in sample numbers. Consequently, we consider it solely as an auxiliary loss function.
Regarding the second question, we believe that recent studies [3,4,5] have addressed this concern. In [3], the issue of **Minority Collapse** under imbalanced scenarios is discussed, indicating that the Simplex ETF becomes unattainable as the tailed classes are compressed closer together. To address this problem, [4] proposed initializing the classifier as a random Simplex ETF, while [5] introduced a similar constraint to ours.

Reference:

[3] Cong Fang, Hangfeng He, Qi Long, and Weijie J Su. Exploring deep neural networks via layer-peeled model: Minority collapse in imbalanced training. Proceedings of the National Academy of Sciences, 118(43):e2103091118, 2021.

[4] Yibo Yang, Shixiang Chen, Xiangtai Li, Liang Xie, Zhouchen Lin, and Dacheng Tao. Inducing neural collapse in imbalanced learning: Do we really need a learnable classifier at the end of deep neural network? Advances in Neural Information Processing Systems, 35:37991–38002, 2022.

[5] Xuantong Liu, Jianfeng Zhang, Tianyang Hu, He Cao, Yuan Yao, and Lujia Pan. Inducing neural collapse in deep long-tailed learning. In International Conference on Artificial Intelligence and Statistics, pp. 11534–11544. PMLR, 2023.

Reviewer comment (Experiments):
In this paper, multi-classification task is regarded as multiple binary classification tasks. Does the proposed approach yield effective results when applied to imbalanced multi-label classification tasks? The empirical study on the multi-label tasks will benefit a lot to improve the generalizability of the proposed method.

Author response:
We acknowledge that extending the application of PLOT to multi-label tasks would significantly contribute to demonstrating its generalizability. However, conducting such an experiment is beyond the scope of this paper and is unfeasible within the constraints of our limited timeframe. Nevertheless, we are committed to providing supporting evidence for the corresponding results in our future research endeavors.

Reviewer comment (Evaluation):
This paper lacks disjoint analysis on the classification performance wr.t. different groups (many/medium/few).

Author response:
Thanks for your reminder, due to the space limit of the main text, we now provide it in the **Section 8 of the Appendix**.

Reviewer comment (Presentation):
Some results on logit adjustment are missing in Table 2.

Author response:
We would like to clarify that the official code of logit adjustment (LA) solely offers implementation support for CIFAR10/100 datasets when the imbalance ratio is set to 100. It is important to note that the official code includes a specific pre-processing operation, making it infeasible to directly apply changes to the imbalance ratio. Therefore, for the sake of fair comparison, we only augment LA under the same conditions as specified in the official code.

Reviewer comment (Evaluation):
There is no clue that the proposed method solves the gradient conflicts issue depicted in Figure 1.

Author response:
Please refer to gradient similarity examination in the **Section 7.2 of the Appendix**, and compare to those in Figure 2 in the main text. From the comparison, we can observe that PLOT augmented DLTR approaches show a more balanced similarities among individuals (categories), indicating the mitigating of conflict issue.

Reviewer comment (Evaluation):
In figure 8, it's unclear whether the proposed method actually surpass the performance of the baseline method LDAM-DRW in terms of gradient similarities. The gradient similarities of the proposed method exhibit a similar trend in the second stage.

Author response:
We appreciate your concern. The observed similarity in the second stage can be attributed to the strong re-weighting strategy employed by LDAM-DRW, known as DRW. This strategy effectively enhances the dominance of the tail classes, leading to the observed trend.

Reviewer comment (Presentation):
Figure 1 may cause misunderstandings. How is Figure 1 computed? Is it cosine similarities between the mean of the gradient of different classes? Since the diagonal is red (1.0), it seems that the values represent gradient similarities. And what is the connection between "gradient conflicts" and "gradient similarities"?

Author response:
Sorry for unclear description. The Figure 1 in the main text represents the cosine similarities among different classes, with the values indicating the degree of gradient similarity. As defined in **Definition 2.1**, when the cosine similarity between two gradients is less than 0, we classify them as conflicting, which is consistent with the definition in multi-task learning.

Reviewer comment (Experiments):
As you have mentioned MOO is applied during the early stages of representation learning, how to select the applied stage?

Author response:
The selection of the applied stage is determined by a hyper-parameter *E*; however, it is important to note that the selection is limited to a fixed set. For instance, if the long-tailed model is intended to be trained over a total of 200 epochs, we choose the applied stage *E* from the set {50, 80, 120, 140}. As part of our future directions, we aim to develop a strategy that can dynamically determine the applied stage, allowing for greater adaptability and optimization in the training process.

Reviewer comment (Experiments):
The iNaturalist dataset has multiple versions. You should highlight it with "iNaturalist 2018".

Author response:
We thank for your kind reminder, we will fix it in our updated version.

Reviewer comment (Novelty):
(This is not a question, but a suggestion). I found a very similar work that also studies long-tailed learning with multi-objective optimization [1]. Maybe you can publicly release your work considering its timeliness.  

[1] Long-Tailed Learning as Multi-Objective Optimization, in https://arxiv.org/abs/2310.20490

Author response:
We sincerely appreciate your reminder. Upon further investigation, we discovered that this paper was recently published on arXiv on October 31st. It is noteworthy that both our work and this paper share a common objective of integrating MOO and DLTR. However, it is important to highlight that our approaches follow distinct methodologies and regimes. Nonetheless, we genuinely value and appreciate your suggestion.

Reviewer comment (Experiments):
My main concern with this paper is that the authors primarily demonstrate the effectiveness of their approach by augmenting existing methods with PLOT. However, they lack comprehensive performance comparisons by integrating PLOT with state-of-the-art methods like BBN, SADE, PaCo, etc. This omission makes it challenging to assess the method's effectiveness in comparison to the latest advancements in the field, raising questions about its overall impact and general applicability.

Author response:
We appreciate your valuable advice. Following your suggestion, we have incorporated an advanced long-tailed method, i.e., BBN, to further showcase the effectiveness of PLOT on CIFAR10-/100-LT datasets. And the results are presented as below.

CIFAR10-LT:
| imb. ratio | 200        | 100   | 50 |
| ---------- | ---------- | ------- | ----- | 
| BBN        | 73.52      | 77.43 | 80.19 | 
| BBN+PLOT   | 74.34 | 78.49 | 82.44 |

CIFAR100-LT:
| imb. ratio |  200         | 100   | 50|
| ---------- | ------------- | ----- | ------- |
| BBN        | 36.14       | 39.77 | 45.64 |
| BBN+PLOT | 36.21       |40.29 | 46.13 |

As observed, PLOT still benefits BBN under various imbalance ratio scenarios. 
It should be noted that all these implementations are based on their official implementations. For fair comparison, we only show the results of PLOT augmentation on those datasets that the official code has implemented on, and both the vanilla and PLOT augmented version are run on the same three random seeds. 
From above results, we believe the generalization of PLOT is well demonstrated.

Reviewer comment (Writing):
The conclusion of the article should be written in the past tense. Additionally, the conclusions require to be added future work of this paper.

Author response:
Thank you for your reminder, we have fixed it.

Reviewer comment (Writing):
The authors are encouraged to carefully proofread the paper.

Author response:
Thank you for your reminder. We are diligently conducting a thorough proofreading of the entire paper. If you have any further suggestions or advice regarding the writing, we kindly request you to bring them to our attention.