Paper: Exploring Weight Balancing on Long-Tailed Recognition Problem (ICLR-2024)

Reviewer comment (Experiments):
The analysis solely based on one particular model method has certain limitations, as it lacks consideration of other methods.

Author response:
We believe that our research, through the analysis of WB, will contribute more than just understanding one method. Our research has two major strengths: 1. we have proved that WD prevents the cone effect [1], a phenomenon that is undiscovered; 2. we have identified fundamental elements that are useful not only for WB but also for general LTR methods. The cone effect is a recently discovered phenomenon, which is meaningful but little-studied. This effect is an inductive bias that is common in deep neural networks, where the output features tend to have high cosine similarity. It is an obstacle in training classification models since low inter-class cosine similarity is desirable. Theorem 1 mathematically shows that regardless of the distribution of the training data, training with WD and CE prevents the increase in inter-class cosine similarity. This is a significant result that suggests an effective way to train classification models not only for LTRs.

 Our study is also meaningful because it is not only an extension of WB but breaks WB down into the fundamental elements important in LTR methods. WB is a simple but effective method that has attracted attention as a baseline for LTR. We analyze this method both theoretically and experimentally to identify the basic elements that are useful for LTR training. Thus, our study is also significant enough to serve as a basis for future research. Such a simple algorithm with theoretical analysis for the long-tail problem has a useful effect on the entire application field in deep learning because of its simplicity.

Reviewer comment (Novelty):
Exploring why Weight Decay performs exceptionally well indeed raises a thought-provoking question in the long-tail domain. However, the favorable properties of Weight Decay have already been extensively explored in balancing datasets, and its effectiveness can be considered widely recognized.

Author response:
We believe that our research, through the analysis of WB, will contribute more than just understanding one method. Our research has two major strengths: 1. we have proved that WD prevents the cone effect [1], a phenomenon that is undiscovered; 2. we have identified fundamental elements that are useful not only for WB but also for general LTR methods. The cone effect is a recently discovered phenomenon, which is meaningful but little-studied. This effect is an inductive bias that is common in deep neural networks, where the output features tend to have high cosine similarity. It is an obstacle in training classification models since low inter-class cosine similarity is desirable. Theorem 1 mathematically shows that regardless of the distribution of the training data, training with WD and CE prevents the increase in inter-class cosine similarity. This is a significant result that suggests an effective way to train classification models not only for LTRs.

 Our study is also meaningful because it is not only an extension of WB but breaks WB down into the fundamental elements important in LTR methods. WB is a simple but effective method that has attracted attention as a baseline for LTR. We analyze this method both theoretically and experimentally to identify the basic elements that are useful for LTR training. Thus, our study is also significant enough to serve as a basis for future research. Such a simple algorithm with theoretical analysis for the long-tail problem has a useful effect on the entire application field in deep learning because of its simplicity.

Reviewer comment (Evaluation):
Besides the analysis metrics mentioned in the paper, what other commonly used metrics exist? Why was the choice of metrics in the paper considered?

Author response:
We have chosen the evaluation metrics in terms of eliminating arbitrariness as much as possible. We have followed existing studies and selected accuracy. We have adopted the accuracy within each group when classes were categorized by the number of samples and the average accuracy between classes. These metrics are commonly used in LTR studies such as [2] and [3]. We have adopted FDR as a metric for the ease of linear feature separation. Other metrics include the accuracy of unsupervised learning models, e.g., the nearest neighbor search method [2]. However, this metric is arbitrary in terms of which unsupervised learning method we use and the hyperparameters we set. FDR has the advantage of being free from such arbitrariness and relatively simple to compute.

Reviewer comment (Novelty):
If we consider balanced datasets, the analysis in the paper can still hold true. The only difference lies in the performance based on the sota models. What distinguishes this type of analysis from conventional methods when dealing with long-tailed data distributions? What are the innovative aspects of this paper?

Author response:
Our study is innovative in that it demonstrates that LTR is not a special setting. As shown in Table 8 in the Appendix, many existing studies assume that LTR is an unusual setting and improve the accuracy by applying complex techniques. However, the reality is that LTRs are not so extraordinary and that simple methods that can be used to balanced data are sufficient to improve accuracy. The fact that our theory and experiments gave the impression that our analysis is no different from that of the balanced case, which is precisely the innovative point.

Reviewer comment (Novelty):
The paper only discusses the related work of NC and WD but the related work of the long-tail is also necessary.

Author response:
A1: We have added an overview of LTR-related methods in Appendix A.2. We put it in Appendix because of the strict page limit of the main text and because we have already briefly introduced the LTR methods in Introduction. In addition, we have added the following at the end of the first sentence of the second paragraph of Section 1 to make it clear that this section is present.
- see Appendix A.2 for more related research.

Reviewer comment (Theory):
What's the meaning of O in Eq.3 and could the author explain more about Theorem 2?

Author response:
A: This is Bachmann-Landau O-notation, and Theorem 2 indicates that the second stage of WB operates equivalent to multiplicative LA. The notation $O\left(\frac{1}{x}\right)$ indicates that the term is at most a constant multiple of $\frac{1}{x}$ if $x$ is sufficiently large. For example, in the second stage of WB for CIFAR100-LT in [1], $\lambda \rho C = 1000 \gg 1$. Often in the text, we have used a notation that uses O notation and inequality, but since this does not seem very common, we have changed it to a notation of equal signs. Also, to clarify the meaning of $O$, the following sentence is added after Theorem 2.
- where $O$ is a Bachmann-Landau O-notation. For example, the notation in the form $O\left(\frac{1}{x}\right)$ indicates that the term is at most a constant multiple of $\frac{1}{x}$ if $x$ is sufficiently large.

Theorem 2 means that in the second stage of WB, the linear layer is trained so that the logit is larger for tail classes. If $\lambda \rho C$ is large enough, then $\mathbf{w}_{k}^* \sim \frac{\overline{N}}{\lambda N} \boldsymbol{\mu}_k$ is satisfied. Note that the features are trained to have larger norm for tail classes in the first stage of training. Therefore, in the second stage of training, each row vector in the linear layer is oriented in the same direction as the corresponding class feature, and the norm is higher for tail classes. This operation is equivalent to multiplicative LA. See also Section 5.1.

Reviewer comment (Experiments):
Could the author explain why the WD&FR&ETF performs worse than the WD&ETF on the ImageNet-LT dataset in Table 13? And are there any experiments conducted on large-scale datasets, such as iNaturalist 2018?

Author response:
A: FDR is improved by FR also in the results of the ImageNet-LT experiment. The purpose of adding FR is to make it easier to satisfy the assumption of Theorem 1 and to obtain features that are more easily linearly separable. In this respect, the experimental results are consistent with the theory. On the other hand, FR does not significantly degrade accuracy, but fails to improve. We find that FR begins to have an effect when the training samples are almost completely separated. Figure 1 in the supplementary material shows the relationship between the number of training epochs, training accuracy, and FDR for CIFAR100-LT. After 200-epoch training, the difference between training with and without FR increases. At this point, the training accuracy is almost 100%. This suggests that FR alone may fail to have a significant impact on test accuracy itself.


Experiments on other large data sets are considered to be quite difficult due to the lack of an experimental environment in our university. For example, a typical experiment on iNaturalist in LTR requires training with a batch size of 500, which requires a lot of VRAM or GPU hours [1]. In addition, although WB publishes the results of its iNaturalist experiments, it does not disclose the hyperparameters used in the training. We need to explore them for comparison, which requires more computational resources. The cloud servers we use for our experiments have a limited number of GPUs that can be used simultaneously for long periods of time, making the experiments difficult. We also considered cloud computing services such as AWS, but this is also difficult due to its enormous cost. On the basis of our experimental environment for ImageNet-LT, we would need to use a p3dn.24xlarge instance for more than 100 hours on AWS, for example. The on-demand fee for this is about $3100, which is too expensive for this supplemental experimentation. For these reasons, we have determined that the ImageNet-LT results are sufficient.

Reviewer comment (Novelty):
Existing long-tail solutions often rely on expert systems to improve the performance of tail classes, such as RIDE[1] and SADE[2]. Is the proposed method in this paper compatible with them?

Author response:
A: Yes, the proposed combination is compatible with these methods because of its simplicity.

Reviewer comment (Presentation):
The difference between weight balancing (WB) and weight decay (WD) needs to be make clearer. Sec. 3 only reviews WB and overlooks WD. Historically, WD was proposed much earlier than WB. It will be a good idea to include some review of WD in Sec. 3, I think. Note that WD is already present in Table 1 on page 4 (right after Sec. 3).

Author response:
A: We have added the following sentence to the beginning of the paragraph "Regularization Methods" in Section 3.
- We implemented WD as L2 regularization with a hyperparameter $\lambda$ because we used stochastic gradient descent (SGD) for the optimizer. Thus, optimization is written as $\mathbf{\Theta}^* = \arg\min_{\mathbf{\Theta}}F(\mathbf{\Theta}; \mathcal{D}) +  \frac{\lambda}{2}\sum_{k \in \mathcal{Y}}\|\mathbf{w}_{k}\|_2^2$.

In addition, we have deleted the following sentence from the paragraph "Weight Balancing" in Section 3 due to duplication.
-  "Note that we implemented WD as L2 regularization with a hyperparameter λ because we used stochastic gradient descent (SGD) for the optimizer."

Reviewer comment (Writing):
For those who are less familiar with two-stage training of LTR, it might be a good idea to include a concise review of two-stage training methods in the Appendix. Note that CVPR2022 and ICLR2020 have different formulation of two-stage training. Please clarify that the model analyzed in this paper is based on the CVPR2022 work even though it cited the ICLR2020 as the original source of two-stage training.

Author response:
A: We have added an overview of LTR-related methods and two-stage learning in Appendix A.2. For example, the section on two-stage learning is as follows. Please refer to the PDF for details.
- Two-stage learning (Kang et al., 2020) is a method for improving accuracy by dividing LTR training into two stages: feature extractor training and classifier training. It is used in numerous methods (Cao et al., 2019; Ma et al., 2021; 2022; Li et al., 2022; Tian et al., 2022; Liu et al., 2023; Kang et al., 2023), including WB, because of its simple but significant improvement in accuracy. Note that since our work analyzes WB, the formulation of two-stage learning is based on WB. For example, the classifier weights are initialized randomly at the start of the second stage of training in Kang et al. (2020) but not in WB.

Reviewer comment (Writing):
There are many acronyms in this paper. It might be a good idea to provide a table summarizing them in the Appendix A.1 (Notation and Acronym).

Author response:
A: We have added a table of abbreviations in Appendix A.1. Thank you for your constructive suggestion!

Reviewer comment (Presentation):
What do blue and red colors in Table 4 and Table 9 highlight? Some explanations can be added to the caption of those tables.

Author response:
A: Red text indicates that the FDR increases when learned features pass through an additional layer, and blue text indicates that it decreases. In Table 4, when the after FDR is higher than the before, and in Table 9 (Table 10 in the modified version), when the FDR is higher than the FDR with one less additional layer passed, the values are marked in red.
In this section, we examine whether the features learned by each method are more linearly separable when pass through a randomly initialized linear layer and ReLU. Therefore, the difference in FDR before and after passing an additional layer is significant. To explicitly indicate this purpose, we have add the following sentence to each caption.
- Table 4
	- Red text indicates higher values than before, blue text indicates lower values.
- Table 9 (10)
	- Red text indicates an increase compared to the FDR of the feature when the number of layers passed is one less, while blue text indicates a decrease.

Reviewer comment (Evaluation):
In Table 5, the accuracy performance of LA (N/A) is noticeably higher than add/mult for the category of "many". Why does LA only work for the Medium and Few classes?

Author response:
A: Because LA increases the probability of being classified into tail classes. LA is a method to correct classification bias by manipulating the output logit for each class. For example, Additive LA adds larger values for classes with fewer samples, while Multiplicative LA multiplies. As a result, LA increases the probability of being classified into tail classes and the accuracy of the classes.

Reviewer comment (Novelty):
This paper appears to resemble an appendix on Weight Balancing to some extent and the technical innovation is rather limited.

Author response:
A1: Our work differs from [1] in the following points: 1. we compare FDRs; 2. we provide theoretical bases; 3.  we demonstrate and experiment with more effective combinations of methods based on those bases.
Indeed, while the Appendix in [1] compares the accuracy of various combinations of regularization methods, we compared FDRs to quantify the ease of linear separation of features. We also present a theory consistent with these experimental results, confirming a theoretical foundation not presented in [1]. On these grounds, we also present combinations not considered in [1], such as ETF, LA, and FR, and demonstrate their validity. In these respects, our work is more than just an extension of [1].

Reviewer comment (Novelty):
Given that Weight Balancing is not the best-performing method in the field of imbalanced learning, the significance of this paper in the field remains debatable.

Author response:
A2:  WB is an essential method suitable for baselines and that theoretical analysis and improvement of it is meaningful. As shown in [1] and Table 7 (Table 8 in the modified version) in the Appendix of our paper, models that have achieved SOTA, such as [2], are complex methods combining ensembles, etc. However, with minimal effort, WB achieves higher accuracy than other existing methods [1]. Therefore, [1] is considered important as a baseline and has attracted attention, with more than 70 citations. We analyze this method both theoretically and experimentally to identify the basic elements that are useful for LTR training. Thus, our study is also significant enough to serve as a basis for future research.

Our research also reveals significant properties of DNN training in general, not just the analysis of WB. It is also credited with discovering the new importance of WD and CE in training classification models, where WD and CE prevent the cone effect [3]. This phenomenon is an inductive bias that is common in deep neural networks, where the output features tend to have high cosine similarity. It is an obstacle in training classification models since low inter-class cosine similarity is desirable. Theorem 1 mathematically shows that regardless of the distribution of the training data, training with WD and CE prevents the increase in inter-class cosine similarity. This is a significant result that suggests an effective way to train classification models not only for LTRs.

Reviewer comment (Novelty):
Considering that Weight Balancing involves implicit constraints at the parameter level (compared to direct correction in other long-tail classification methods), its extension to address broader distribution shift issues should hold greater value.

Author response:
A3: Problems dealing with shifts to uniform distributions are highly significant. Many studies dealing with shifts to uniform distributions have been published, including [1] and [2], which have received attention. Indeed, addressing general distributional shifts, such as [4], has also attracted interest. However, addressing shifts to uniform distributions has become more significant. For example, [4] trains one neural network of an ensemble using existing methods for shifts to uniform distributions. Thus, the methods for shifts to uniform distributions are the basis of methods for general shifting distributions, and improvements to them are of sufficient importance.

Reviewer comment (Theory):
Sec 5.1 "the second stage of WB is equivalent to multiplicative LA". Why not just use explicit LA?

Author response:
A4: We verify the effectiveness of replacing the second stage in WB with LA in Section 5.2. We analyze the effectiveness of WB both experimentally and theoretically before Section 5.2. In Section 5.2, we explicitly replace WB with essential elements to test the validity of the combination. We also test the case where we explicitly replace the second stage of training with LA. Table 5 shows that the appropriate combination including LA achieves better accuracy than WB. We have changed the final sentence of Section 5.1 as follows to clarify this logic. See the modified PDF for details. Updated sections in the PDF are indicated in red.
- This also suggests that replacing the second stage of training with LA would be more generic and we confirm its validity in the following section.