Paper: Learning Label Shift Correction for Test-Agnostic Long-Tailed Recognition (ICLR-2024)

Reviewer comment (Experiments):
The proposed method was not evaluated on general test label distributions, but rather on three limited types, forward, uniform and backward. How does the performance of the proposed method change as encountering more general label distribution shift?

Author response:
**Response:**
  In this paper, we delve into the test-agnostic long-tailed recognition problem, aligning with previous research efforts such as SADE [NeurIPS 2021], LADE [CVPR 2021], and BalPoe [CVPR 2023]. To the best of our knowledge, no existing long-tail learning method can generalize to arbitrary test label distributions—a limitation inherent in the no-free-lunch theorem. To address this challenge, a pivotal issue is how to effectively simulate diverse label distributions, a facet that has not been extensively explored in the field. By developing the capability to simulate any label distribution, we believe our proposed method can encompass various simulations into the training data, thereby learning to estimate arbitrary label distributions.

Reviewer comment (Experiments):
The effectiveness of neural estimator in correcting the label distribution shift may highly depend on how many ($Q$ in Alg. 1) and diverse class priors have been encountered during the neural estimator. The authors need to explain more about this and the corresponding overhead in training. For example, when the test distribution has imbalance ratio (backward with 200), can the proposed method, which only experienced the imbalance ratio of range [1/100,100], correctly estimate the test label distribution?

Author response:
**Response:** 
  We show that the training of our proposed neural estimator can be done efficiently. We first elaborate on the simulation of training data for the neural estimator. Given a class prior, we determine the corresponding number of samples for each class for simulation. Subsequently, we swiftly sample a subset of training samples for every class by pre-dividing the samples in the entire training set into their respective class pools. In general, the time complexity of the re-sampling process is approximately $O(QN)$, where *$Q$* is the number of times of re-sampling and $N$ is the number of training samples. By default, we set $Q=2\times10^3$ in all experiments and the re-sampling process can be done in seconds for training data of size $N=10^6$ on modern computers. We also monitor the time consumption of the entire training process of the neural estimator, it takes *12.78* seconds and *355.15* seconds on CIFAR100-LT and ImageNet-LT, respectively. To demonstrate the effectiveness of the simulation of different label distributions, we attach the `Python` code for reference.

  ```python
  def multiple_subset_resampling(imbalance_ratio_list, img_idx_each_cls):
    """sample multiple subsets from a list of class priors"""
    subset_list = []
    for imbalance_ratio in imbalance_ratio_list:
      selected_idx_list = subset_resampling(imbalance_ratio, img_idx_each_cls)
      subset_list.append(selected_idx_list)
    return subset_list
  
  def subset_resampling(imbalance_ratio, img_idx_each_cls):
    """sample a subset from a given class prior"""
    img_num_per_cls = produce_num_per_cls(imbalance_ratio)
    selected_idx_list = []
    for the_class, the_img_num in enumerate(img_num_per_cls):
      idx = img_idx_each_cls[the_class]
      selected_idx = torch.multinomial(torch.ones(idx.size(0)), the_img_num, replacement=True)
    	selected_idx_list.append(selected_idx)      
    return selected_idx_list
  ```

Reviewer comment (Theory):
More elaborations on Eq. (5) is needed. First, it is unclear how $\hat{Z}^h_{ij}$ and $\hat{Z}^t_{ij}$ are defined. The authors comment that $\hat{Z}^h$ and $\hat{Z}^t$ are the sum of head and tail parts in $\hat{Z}$. Then, what does index $j$ mean in $\hat{Z}^h$ and $\hat{Z}^t$? Can the authors elaborate why the optimal $k$ maximizing (5) results in a higher value for the backward case and lower for the forward case?

Author response:
**Response:**
  We are sorry for the confusion and correct Eq. (5) as follows:

  $$  k = \arg \max _ {k \in \mathcal{K}} \mathbb{I}(\pi _ 0^{h} > \lambda \pi _ 0^{t}) \widehat{Z}^{h} + \mathbb{I}(\pi _ 0^{h} < \lambda \pi _ 0^{t}) \widehat{Z}^{t} \quad s.t. \quad \widehat{Z}=logitClip({Z}, k)$$

**Response:**
The rationale behind Eq. (5) is twofold.

Firstly, we observe that most long-tail learning methods generate logits that exhibit a bias towards tail classes. Specifically, the model may produce large (positive) logits for tail classes even when the ground truth corresponds to a head class. Conversely, the produced logits for head classes can be small (negative) values, excluding the ground-truth class. Consequently, without a higher value of $k$, it becomes easy to observe the backward label distribution, as tail classes dominate head classes in the logits matrix. However, we require a lower value of $k$ to clip negative logits of head classes, enabling the identification of a forward label distribution. This observation is visually depicted in Figure 2(b) of the paper.

Secondly, we utilize Eq. (5) to determine the value of $k$ because the predicted probabilities (i.e., $\pi _ 0$) can provide a rough sketch of the true label distribution. This observation is illustrated in Figure 2(a) of the paper. Combining these observations, Eq. (5) is capable of automatically searching for a suitable value of $k$.

Reviewer comment (Writing):
**[Writting about the proposed method is confusing.]** In my view, there are several points in Section 3 making me confused. (1) Your proposed method is LSC, but I notice that you use a lot of space to introduce GBBSE. So what is the connection between LSC and GBBSE? (2) What does $g_{\theta}$ refer to in your experiments? Is $g_{\theta}$ equal to NeuralEstimator? (3) Since the training dataset is imbalanced, how can you sample a subset of the training dataset when its smallest class should be the biggest class in the subset?

Author response:
**Concern #1: The connection between LSC and GBBSE?**

**Response:** Sorry for any confusion. This paper presents a general label-shift estimation framework called GBBSE. We then propose a practical implementation under the framework of GBBSE and the implemented method is called LSC. We will improve the writing in the next version soon.


**Concern #2: Is $g _ \theta$ equal to *NerualEstimator*?**

**Response:** Yes, we refer to $g _ \theta$ as the *NerualEstimator* in our paper.


**Concern #3: Since the training dataset is imbalanced, how can you sample a subset of the training dataset when its smallest class should be the biggest class in the subset?**

**Response:**
Given a class prior, we utilize either undersampling or oversampling for simulation. In cases where additional training data is required for the smallest class, we implement resampling of the training data with replacement. To enhance diversity, we incorporate data augmentation strategies, such as RandAugment or AutoAugment, which generate distinct copies of the same image. To elucidate the sampling process, we provide the attached `Python` code for reference:

  ```python
  def multiple_subset_resampling(imbalance_ratio_list, img_idx_each_cls):
    """sample multiple subsets from a list of class priors"""
    subset_list = []
    for imbalance_ratio in imbalance_ratio_list:
      selected_idx_list = subset_resampling(imbalance_ratio, img_idx_each_cls)
      subset_list.append(selected_idx_list)
    return subset_list
  
  def subset_resampling(imbalance_ratio, img_idx_each_cls):
    """sample a subset from a given class prior"""
    img_num_per_cls = produce_num_per_cls(imbalance_ratio)
    selected_idx_list = []
    for the_class, the_img_num in enumerate(img_num_per_cls):
      idx = img_idx_each_cls[the_class]
      selected_idx = torch.multinomial(torch.ones(idx.size(0)), the_img_num, replacement=True)
    	selected_idx_list.append(selected_idx)      
    return selected_idx_list
  ```

Reviewer comment (Writing):
**[The highlight of the proposed method is not clear.]** The goal of LSC is to find an estimator $g$ that aligns the predicted test label distribution with the true. But in Line 7 of the pseudo-code, $g_{\theta}$ aims to align subset label distribution with the prior global label distribution on the training dataset. So can you explain which step achieves this goal?

Author response:
**Concern #4: Can you explain which step achieves the alignment?**

**Response:** As illustrated in the pseudo-code, the training of $g _ \theta$ is conducted on simulated subsets with various label distributions, rather than the global label distribution. The objective is to minimize the discrepancy between the output of $g _ \theta$ and the ground-truth label distribution of our simulated subsets. Ultimately, the well-trained $g _ \theta$ is able to generalize to the test data and estimate the test label distribution. We will modify the pseudo-code to make the notations more clear.

Reviewer comment (Theory):
**[Your theoretical analysis cannot explain why your proposed method can align the discrepancy between the predicted and the true test label distributions.]** You have mentioned two theorems in your paper. However, both of them only certify that a smaller gap between predicted and the true test label distributions leads to a lower generalization error. They do not explain why your proposed method can effectively estimate the true test label distribution.

Author response:
**Concern #5: Can you theoretically analyze why this method can align the predicted and the true test label distribution?**

**Response:**
  Theorem 3.2 in our paper ensures that we can acquire a sufficiently trained base model, i.e., $\epsilon _ {L}(g^{\star})$ is small. Assuming that the error gap $\epsilon _ {L}(g _ {\theta}) - \epsilon _ {L}(g^{\star})$ can be bounded through the training of $g _ {\theta}$ using generated multiple subsets (ensured by the Bayes-risk consistency of the training), we can approximate the test label distribution precisely to adjust the model, especially when the test sample size is sufficiently large.
  More precisely, assuming that the multiple subsets are obtained by *i.i.d* sampling, we can bound the estimation error through classical generalization theories. 

  However, in the context of long-tail learning, generating multiple subsets for training $g _ {\theta}$ involves oversampling for tail classes, disrupting the *i.i.d* properties of sampling. This departure from *i.i.d* conditions complicates the application of traditional machine learning generalization analysis. To the best of our knowledge, analyzing the generalization bound in this scenario without additional assumptions proves to be extremely challenging.

Reviewer comment (Writing):
Can you explain which step achieves the alignment?

Author response:
**Concern #4: Can you explain which step achieves the alignment?**

**Response:** As illustrated in the pseudo-code, the training of $g _ \theta$ is conducted on simulated subsets with various label distributions, rather than the global label distribution. The objective is to minimize the discrepancy between the output of $g _ \theta$ and the ground-truth label distribution of our simulated subsets. Ultimately, the well-trained $g _ \theta$ is able to generalize to the test data and estimate the test label distribution. We will modify the pseudo-code to make the notations more clear.

Reviewer comment (Theory):
Can you theoretically analyze why this method can align the predicted and the true test label distribution?

Author response:
**Concern #5: Can you theoretically analyze why this method can align the predicted and the true test label distribution?**

**Response:**
  Theorem 3.2 in our paper ensures that we can acquire a sufficiently trained base model, i.e., $\epsilon _ {L}(g^{\star})$ is small. Assuming that the error gap $\epsilon _ {L}(g _ {\theta}) - \epsilon _ {L}(g^{\star})$ can be bounded through the training of $g _ {\theta}$ using generated multiple subsets (ensured by the Bayes-risk consistency of the training), we can approximate the test label distribution precisely to adjust the model, especially when the test sample size is sufficiently large.
  More precisely, assuming that the multiple subsets are obtained by *i.i.d* sampling, we can bound the estimation error through classical generalization theories. 

  However, in the context of long-tail learning, generating multiple subsets for training $g _ {\theta}$ involves oversampling for tail classes, disrupting the *i.i.d* properties of sampling. This departure from *i.i.d* conditions complicates the application of traditional machine learning generalization analysis. To the best of our knowledge, analyzing the generalization bound in this scenario without additional assumptions proves to be extremely challenging.

Reviewer comment (Experiments):
What is the effect of $\lambda$ on the final performance? Why do you choose the value of $\lambda$ as 1.5?

Author response:
**Concern #6: What is the effect of $\lambda$ on the final performance?**

**Response:**
  We examine the impact of $\lambda$ on the final performance by selecting values from the set $\\{1, 1.5, 2, 3\\}$. The experiments are conducted on the ImageNet-LT dataset, and the results are presented in the table. Overall, we observe comparable performance across different values of $\lambda$ in the majority of cases.

  |$\lambda$ | Forward50 | Forward5  | Uniform | Backward5  | Backward50 |
  |:---------:|:----:|:----:|:----:|:----:|:----:|
  | 1     | 72.3 | 65.6 | 59.2 | 58.2 | 57.3 |
  | 1.5    | 72.3 | 65.6 | 60.5 | 58.2 | 57.3 |
  | 2     | 72.3 | 65.6 | 60.5 | 58.2 | 57.3 |
  | 3     | 70.9 | 65.6 | 60.5 | 55.0 | 57.3 |

Reviewer comment (Experiments):
Does the number of sampling subsets of the training dataset affect the final performance?

Author response:
**Concern #6: Does the number of sampling subsets of the training dataset affect the final performance?**

**Response:** Thank you for this interesting question. We explore the impact of the number of sampled subsets on the ImageNet-LT dataset. We conduct simulations with subsets ranging from $\\{400, 1000, 2000, 4000\\}$ and analyze the resulting final performance. Generally, we observe an improvement in test accuracy with an increase in the number of simulated subsets. This is attributed to the aim of ensuring that the training set for $g _ \theta$ covers diverse label distributions. However, we only sample 2000 subsets for a balance between effectiveness and efficiency.

  | # of subsets | Forward50 | Forward5  | Uniform | Backward5  | Backward50 |
  |:-----------:|:----:|:----:|:----:|:----:|:----:|
  | 4000    | 72.3 | 65.6 | 60.5 | 58.0 | 57.7 |
  | 2000    | 72.3 | 65.6 | 60.5 | 58.3 | 57.3 |
  | 1000    | 72.2 | 65.6 | 60.5 | 58.2 | 56.8 |
  | 400     | 70.9 | 65.6 | 61.0 | 57.1 | 52.7 |