Paper: Alleviating Label Shift Through Self-trained Intermediate Distribution: Theory and Algorithms (ICLR-2024)

Reviewer comment (Experiments):
The three datasets, MNIST, CIFAR10, and CIFAR100, used to construct the experimental scenarios, are somewhat simplistic. Moreover, certain experiments and metrics are only disclosed for one or two of these datasets.

Author response:
For the first weakness, since our source dataset has the restriction of label distribution, we try to verify the performance of ALS on a larger dataset with a larger number of categories. However, due to time constraints, we did not obtain new results. But, we add MLLS, which is an advanced method for last two years, as a comparison method to illustrate ALS more effectively.

Reviewer comment (Reproducibility):
The disclosure of experimental hyperparameters appears to be incomplete. Algorithm hyperparameters, network hyperparameters, hyperparameters for comparison methods, random seeds, and other relevant details should be fully presented. Experimental results should include the mean and variance of metrics from multiple trials.

Author response:
For the second weakness, we directly adopt the code provided by the authors of the compared methods. Since the comparison method can be embedded in our method, the network architecture and hyperparameters are set according to the compared methods. We illustrate it in the paper for a clear illustration.

Reviewer comment (Experiments):
The two types of distribution changes in the experimental setup are not intuitive, lacking visualization, making it difficult for readers to perceive the extent of label distribution shift in the experimental setup. It is also challenging to determine what the most extreme scenarios included in the experimental setup are like.

Author response:
For the third weakness, we use the parameter $\alpha$ to determine the label distribution. We choose $\alpha=1$ as the upper bound of the shift, because $\alpha=0.5$ may cause some classes to have zero data, which will break the algorithm.

Reviewer comment (Evaluation):
The applicability of the algorithm should be thoroughly discussed. Considerations such as when the algorithm might fail, its compatibility with other algorithms and models, and its performance in extreme conditions (such as extreme imbalance or unseen categories) should be discussed.

Author response:
For the forth weakness, the new class can be seen as a special case of extreme imbalance. One of the directions for future research is how to correct the model in the case of extreme imbalance. This article focuses more on how to use the data in the target domain, and thus does not consider extreme imbalanced cases specially.

Reviewer comment (Writing):
There exist some minor errors, such as the “i” in Eqn.(7) is confusing with different ranges, and in the Appendix, the equation numbers are wrongly used, for example, “Combining Eq.(17) and Eq. (20), we have” and “Combining Eq. (22), Eq. (24), Eq. (25) and Eq. (26), we have”. The method BBSE is wrongly written as BBLS in "which outperforms both BBSL and RLLS across diverse datasets and.......". In Table 1, the combination of BBSE or RLLS and ALS is wrongly written as BBSE-LSC or RLLS-LSC.

Author response:
For the first question, we revise this paper in the following two parts. (1) In view of the grammar and symbol usage problems of the article, we again checked and revised it to ensure that there were no symbol misuse and syntax error problems. (2) We propose a new method to extend ALS with MLLS, which is an advanced method for last two years, and call it MLLS-ALS. Then we compare MLLS-ALS with MLLS, and the experimental results in subsection A.3.4 of the supplementary material show that MLLS-ALS achieves good performance in most cases.

Reviewer comment (Experiments):
In the Introduction, MLLS is mentioned, which is one of the latest methods for label shift. It is better to show the experimental results about MLLS or the combination of MLLS and ALS.

Author response:
For the first question, we revise this paper in the following two parts. (1) In view of the grammar and symbol usage problems of the article, we again checked and revised it to ensure that there were no symbol misuse and syntax error problems. (2) We propose a new method to extend ALS with MLLS, which is an advanced method for last two years, and call it MLLS-ALS. Then we compare MLLS-ALS with MLLS, and the experimental results in subsection A.3.4 of the supplementary material show that MLLS-ALS achieves good performance in most cases.

Reviewer comment (Experiments):
Proposed method is designed to be used in conjunction with methods that are several years old. Empirical evaluation also does not compare against more recent methods like the mentioned MLLS or other cited related works from last 2-3 years.

Author response:
Thank you for your comments. 
For the first weakness, as the reviewers said, ALS can be viewed as an additional trick for improving performance on most label shift methods. Thus we propose a new method to extend ALS with MLLS, which is an advanced method for last two years, and call it MLLS-ALS. Then we compare MLLS-ALS with MLLS, and the experimental results in subsection A.3.4 of the supplementary material show that MLLS-ALS achieves good performance in most cases.

Reviewer comment (Writing):
Naming choices: By proposition 1, "ideal" intermediate distribution is non-unique, why not just call it an "intermediate distribution"? Acronym ALS is never explained (unless I just didn't see it).

Author response:
For the third weakness, we illustrate the meaning of ALS in the part of Introduction, i.e., 'our approach, abbreviated as ALS, learns a self-trained intermediate distribution...'. We used 'ideal' earlier to better distinguish between the true and pseudo-label-guided intermediate distributions. Borrowing from the reviewer's suggestion, we drop the 'ideal' for clearer presentation. In view of the grammar problems of the article, we have checked and revised it again.

Reviewer comment (Writing):
Text should be checked by a proofreader or grammar software; eg "alleviates" in Definition 1 should take a grammatical object, unclear what's meant by "k-th class of label y" in beginning of 3.1 until reading the sentence after equation (8).

Author response:
For the third weakness, we illustrate the meaning of ALS in the part of Introduction, i.e., 'our approach, abbreviated as ALS, learns a self-trained intermediate distribution...'. We used 'ideal' earlier to better distinguish between the true and pseudo-label-guided intermediate distributions. Borrowing from the reviewer's suggestion, we drop the 'ideal' for clearer presentation. In view of the grammar problems of the article, we have checked and revised it again.

Reviewer comment (Experiments):
Because MLLS doesn't have good theoretical guarantees, I understand why you didn't want to do the theoretical analysis of ALS applied to MLLS and used BBSE/RLLS instead. However, can ALS still empirically be applied with MLLS or other more recent and well-performing methods? I could see selling ALS as a method-agnostic, additional trick for improving performance on top of most label shift techniques.

Author response:
For the first question, we compare the proposed MLLS-ALS with MLLS in subsection A.3.4 of the supplementary material to show the effectiveness of ALS.

Reviewer comment (Evaluation):
The experimental section doesn't include the time performance.

Author response:
Thank you for your comments. 
For the second weakness, we analyze the time cost through experiments in subsection A.3.5 of the supplementary material and the experimental results show that our method does not have a particularly high time cost. Although if we pick more pseudo-labels, we will spend more time training the reweighted target classifier. However, we only select samples with the highest confidence that can maintain the label ratio in this paper, so it does not result in too much time cost.

Reviewer comment (Writing):
What's the meaning of those undefined notations?

Author response:
Thank you for your comments. 
For the first question, we reinterpret these notations in the main text.