Paper: Learning Label Refinement and Thresholds for Imbalanced Semi-Supervised Learning (ICLR-2024)

Reviewer comment (Theory):
There is no theoretical ground for the proposed algorithm.

Author response:
Now we rephrase our method section and supply SEVAL with its theoretical motivation. We also derive the principal hypothesis behind the success of SEVAL. Like other SSL algorithms, SEVAL is mostly heuristic. However, we believe it is valuable as our analysis sheds new lights on the problem of pseudo-label refinement and we validate the effectiveness with extensive experiments.

Reviewer comment (Experiments):
For CIFAR-10, the experimental settings are not enough. For example, other LTSSL (long tailed semi supervised learning) studies such as DARP, DASO, SAW conducted experiments on CIFAR-10 with changing both imbalanced ratio of labeled and unlabeled sets.

Author response:
Due to page limitations, we only report key experiment results in main texts. We provide plenty of experiments in section B. For the experiments of CIFAR10-LT, now we provide the results when $\gamma_l \neq \gamma_u$ and $\gamma_l=\gamma_u=150$. Take this into account, our experiments of CIFAR10-LT are equally or more extensive than previous works including DARP, DASO and SAW.

Reviewer comment (Experiments):
The proposed algorithm appears to require extensive hyperparameter tuning.

Author response:
SEVAL requires three core hyperparameters, including $t$ and two moments for curriculum learning ($\rho_\pi $ and $ \rho_\tau$). Other parameters, such as the length of the acquired curriculum ($L$), predominantly affect SEVAL's learning speed with minimal influence on the outcomes. Based on empirical observations, we reduce $L$ as the number of classes increases to enhance training efficiency. 

When compared with FixMatch, we only add two hyper-parameters because $t$ is also required for FixMatch. Considering that it is reasonable to keep $\rho_\pi $ and $\rho_\pi $ the same, we believe that SEVAL does not require extensive hyper-parameter tuning.

Now we conduct sensitivity analysis of these parameters in section B.6 and we find SEVAL is not sensitive to hyper-parameters. We summarize the detailed hyper-parameter in section C.4 to facilitate reproducibility.

Reviewer comment (Experiments):
The proposed algorithm and baseline algorithms are combined with only FixMatch, without being combined with ReMixMatch (other LTSSL studies such as DARP, ABC, SAW and CoSSL conducted experiments with combining their algorithms with both FixMatch and ReMixMatch).

Author response:
We think there might be some misunderstandings. We integrate SEVAL with Mean Teacher, MixMatch and ReMixMatch and report experimental results in Figure 4(a), showing better performance than other methods. Now we report more detailed and complete results in section B.4 and summarize implementation details in section C.4.

Reviewer comment (Experiments):
The proposed algorithm makes a class balanced validation set from the train set, which seems not realistic. For example, for the iNaturalist dataset, the most minor class has only 2 samples, and in this case, it would be hard to make a validation set from the 2 samples.

Author response:
We also think there might be some misunderstandings. As we demonstrate in the beginning of section 4, we make no assumption regarding the distribution of the validation set. To achieve so, we make our optimization function class-balanced, c.f. Eq.3 and Eq. 4. In fact, as we describe in the method section, we partition the training dataset into two subset with the same number of samples. Therefore, in most of our experiments, the validation set is also (highly) imbalanced.

Regarding sample efficiency, we validate our algorithms using Semi-Aves, derived from the mentioned iNaturelist dataset. We find SEVAL works well under such settings and summarize the results in section B.2. SEVAL achieves this with frequency normalization and group-wise optimization, as mentioned in section 4.2 and described in details in section C. Now we also supply extreme experiment settings when very few labelled samples are available (tailed class contains 2 labelled samples or 40 labelled samples in total) in section B.3. We find SEVAL works well under such scenarios, demonstrating that SEVAL is labelled sample efficient.

Reviewer comment (Experiments):
Lack of literature review. This work missed a couple of important baselines to compare with. For example, the methods presented in [4,5,6,7, 8].

Author response:
> Q2. Lack of literature review. This work missed a couple of important baselines to compare with. For example, the methods presented in [4,5,6,7, 8].
 
We think there might be some misunderstandings. We compare with Adsh [4], ACR [7], DARP [8] and report the results under the same setting mainly in Table 2 and also other tables. We discuss their differences with SEVAL in related work and method sections. We also mention [5] in related work, and compare with a very similar work (CreST) along most tables. Now we add a more detailed discussion of SAW [7] in section 2, which focuses on reweighting of unlabelled samples, therefore is orthogonal to our studies which focus on pseudo-label refinement. We think SEVAL and SAW can work well together and leave the investigation for future works.

Reviewer comment (Experiments):
Lack of experimental design. This work only evaluated it on FixMatch. However, in other pieces of literature [3,5,6,8], they evaluated on various SSL algorithms to show the effectiveness of the method, such as MixMatch and ReMixMatch. It would be helpful if the authors can present the results of these SSL algorithms. Moreover, FixMatch was proposed three years ago. It would be interesting to apply SEVAL to more recent algorithms, such as FlexMatch, CoMatch, or other SSL algorithms.

Author response:
> Q3. Lack of experimental design. This work only evaluated it on FixMatch. However, in other pieces of literature [3,5,6,8], they evaluated on various SSL algorithms to show the effectiveness of the method, such as MixMatch and ReMixMatch. It would be helpful if the authors can present the results of these SSL algorithms. Moreover, FixMatch was proposed three years ago. It would be interesting to apply SEVAL to more recent algorithms, such as FlexMatch, CoMatch, or other SSL algorithms.
 
We think there might be some misunderstandings. We integrate SEVAL with Mean Teacher, MixMatch, and ReMixMatch, reporting experimental results in Figure 4(a), demonstrating superior performance compared to other methods. Now we report more detailed and complete results in section B.4 and summarize implementation details in section C.4.
 
Our method cannot apply to FlexMatch as they both modify the thresholding. Instead, we conduct a thorough comparison between SEVAL and FlexMatch throughout this manuscript. CoMatch enhances the pseudo-label through feature regularization, a technique akin to the semantic alignment regularization introduced by DASO. We report our method and work well with semantic alignment regularization in section 5.2.1. We anticipate SEVAL can also work well with CoMatch, and we would like to leave these investigations for the future.

Reviewer comment (Experiments):
The motivation is not clear to me. The main goal of SSL is to reduce the labeling effort. However, the experimental setting shows they still use a large amount of labeled data in the training data and a labeled validation set. FixMatch can achieve ~90% accuracy when only using 40 labeled data. It would be helpful if the authors could do experiments where the labeled data are scarce, e.g., 40 labeled data in total, while the unlabeled data can be highly imbalanced.

Author response:
> Q4. The motivation is not clear to me. The main goal of SSL is to reduce the labeling effort. However, the experimental setting shows they still use a large amount of labeled data in the training data and a labeled validation set. FixMatch can achieve ~90% accuracy when only using 40 labeled data. It would be helpful if the authors could do experiments where the labeled data are scarce, e.g., 40 labeled data in total, while the unlabeled data can be highly imbalanced.
 
We utilize labelled validation dataset to learn the curriculum parameters because we find this can lead to better label refinement and thresholding, further improving current SSL algorithms. Existing methods cannot attain this without optimization on the validation dataset, as we elucidate in Section 4.
 
Following the suggestion of the reviewers, now we provide experiment results under extreme settings with very few labeled samples (2 labeled samples in the tailed class or 40/100 labeled samples in total) in Section B.3. Our findings indicate that SEVAL performs effectively in such scenarios, highlighting its efficiency with limited labeled samples.

Reviewer comment (Experiments):
The parameter is tuned on the validation set, which is not practical in my view. How do we prepare a labeled validation set in real-world scenario? If we have such labeling budget, why don't we choose other learning algorithms but stick to SSL?

Author response:
> Q5. The parameter is tuned on the validation set, which is not practical in my view.
 
Tuning the hyper-parameter based on the performance of the validation dataset is a common practice to develop machine learning models.
 
SEVAL necessitates three key hyper-parameters, which include $t$ and two moments for curriculum learning ($\rho_\pi$ and $\rho_\tau$). In comparison to FixMatch, we only introduce two additional hyper-parameters because $t$ is also required for FixMatch. Given the reasonable assumption of maintaining consistency between $\rho_\pi$ and $\rho_\pi$, we assert that SEVAL does not demand extensive hyper-parameter tuning.
 
In Section B.6, now we conduct a sensitivity analysis of these parameters and observe that SEVAL exhibits robustness to hyper-parameter variations. Detailed hyper-parameter information is summarized in Section C.4 to facilitate reproducibility.
 
> Q6. How do we prepare a labeled validation set in real-world scenario?
 
We think there might be some misunderstandings. As we demonstrate in section 4.3, in practice SEVAL does not require additional labelled validation sets. Instead, we learn the curriculum based on a partition of the training dataset. Specifically, we partition the training dataset into two equal subsets and utilize one of them for SEVAL parameter optimization. We do not utilize any additional labelled samples compared with other SSL algorithms.
 
> Q7. If we have such labeling budget, why don't we choose other learning algorithms but stick to SSL?
 
SEVAL is labelled data efficient and does not require more labelled samples than other SSL algorithms. In all the experimental settings we investigate in, we observe that SEVAL consistently outperforms state-of-the-art SSL algorithms with the same labeled budgets, even when the number of labeled samples is extremely limited (e.g., 40 in total). 
 
*We kindly ask the reviewer to reconsider the value of our work within the context of decomposing and advancing the design of pseudo-label based SSL rather than mainly an alternative method for pseudo-label refinement.*

Reviewer comment (Novelty):
The discussion of relevant work was not sufficient, so that the progressiveness and novelty of the article could not be highlighted. As mentioned in the experimental section, some existing methods also include both pseudo label refinement (PLR) and threshold adjustment (THA) modules. A detailed discussion is needed on the differences between the proposed method and these methods.

Author response:
We have rephrased our method section and highlighted the differences between SEVAL and other methods focusing on PLR and THA. We demonstrate distinct variations from both a theoretical standpoint and through easily comprehensible toy examples. In the context of PLR, SEVAL can yield more optimal results by leveraging Bayes' theorem, as elaborated in section 4.1. In the case of THA, SEVAL stands out as the sole method that estimates thresholds based on precision, offering a more reliable approach compared to existing methods, as discussed in section 4.2.

Reviewer comment (Theory):
Relevant theoretical analysis is needed to ensure the rationality of the proposed method. Otherwise, the proposed method is more like a combination of techniques.

Author response:
We have enhanced the method section, providing SEVAL with a robust theoretical foundation. Additionally, we have elucidated the core hypothesis underpinning SEVAL's effectiveness. Although SEVAL shares the heuristic nature typical of many SSL algorithms, its value lies in our analysis, offering novel insights into the pseudo-label refinement challenge. We substantiate SEVAL's efficacy through comprehensive experimental validation.

Reviewer comment (Writing):
The expression of the article needs to be improved, such as (a) grammar error, e.g., in Page 4, “It which comprise two...” (b) spelling error, e.g., in Page 9, “We compared with the ... inf Figure 8.” (c) Figure 9 is not referenced in the main text. (d) inconsistent expression, e.g., $n_i $ and $n1$ are both appeared. (e) improper use of terminology, e.g., in Page 3, the meaning of "data loader" is unclear.

Author response:
We thank the reviewer for detailed suggestions. All the identified typos have been rectified.

Reviewer comment (Evaluation):
In experimental section, why not record specific evaluation criteria for imbalanced classification, such as precision, recall, and F1 score?

Author response:
We follow the literature and utilize accuracy as a main metric to evaluate the effectiveness of SSL algorithms. Of note, as the test data has a uniform distribution, the reported accuracy is equivalent to average recall of all classes and can represent the overall classification performance of all classes.

We agree with the reviewers on the importance of comprehensive evaluation metrics. Due to page limitations, we summarize the class-wise precision, recall and F1 score in appendix section E. From the detailed results, we find that SEVAL improves the classification model by being more sensitive to the minority classes.

Reviewer comment (Evaluation):
Is the accuracy recorded in the experiment evaluated on the unlabeled samples used during the training phase? Is it evaluated on a test set that is not used during the training phase?

Author response:
We report the accuracy on a test set that is not used during the training phase. We do not evaluate the performance on the unlabeled data as it is out of scope for the problem setting of SSL.

Reviewer comment (Experiments):
In the experimental section, is the class distribution on the data set used in the test phase the same as the labeled sample set used in the training phase? In real applications, how to ensure that the class distribution on a small number of labeled samples is the same as the real class distribution?

Author response:
Following the literature, the distribution of test data is different from that of training data. In particular, the test data is assumed to follow a uniform distribution, because we care more about the overall classification performance.

Reviewer comment (Experiments):
In Table 1, why is the imbalanced ratio of unlabeled data set unknown on the STL10-LT?

Author response:
Our algorithms and experimental setting assume that the training data follows the long-tailed distributions, which is common for real world datasets. We make no assumption on the distribution of the validation dataset, as we declare in the beginning of section 4. This is achieved class-balanced optimization function, c.f. Eq.3 and Eq. 4.

Reviewer comment (Presentation):
In Table 1, what does $n_1$ mean? Is it the number of labeled samples from majority class or the number of labeled samples from minority class?

Author response:
The purpose of STL10 is to make use of the unlabelled data, which comes from a similar but different distribution from the labelled data, to learn better representation. Therefore, this dataset does not provide any class prior of unlabelled  samples, making it impossible for us to control the imbalanced ratio.

Reviewer comment (Reproducibility):
On line 16 of Algorithm 2, why not use the final $\pi$ and $\tau$ learned in the previous stage?

Author response:
$n_{1}$ represents the samples from the class that contains the most samples as $n_{c}$ is sorted in a descending order. Now we further clarify this in section 3.

Reviewer comment (Theory):
This paper mandates the strong assumption that there must exist a sufficient amount of labeled validation data per class (at least 10 per class), which could easily be violated under semi-supervised learning settings.

Author response:
SEVAL does not request more labelled samples than other SSL algorithms. Our ablation study demonstrates that SEVAL remains stable with varying amounts of validation data, ranging from 10 to 500 per class.
In order to further demonstrate the data efficiency of SEVAL, now we also supply extreme experiment settings when very few labelled samples are available (tailed class contains 2 labelled samples or 40 labelled samples in total) in section B.3. The experimental results demonstrate that SEVAL is labelled sample efficient and works well under such scenarios.

Reviewer comment (Experiments):
Highly relevant and overlapping method such as Adsh [1] is not included in the experiments.

Author response:
Following the request of the reviewers, now we add the comparison with Adsh in the experiments. We now update the results in Table.2. We find that Adsh improves model performance in most cases. Our algorithm is consistently better than Adsh in all the settings.

Reviewer comment (Theory):
The motivation is not strong enough, in the abstract, the authors mentioned that existing dynamic threshold methods can cause bias in the pseudo-labeling. However, there seems a lack of detailed justification to substantiate how that method can cause bias. In addition, approaches such as FlexMatch and FreeMatch use class-dependent thresholds, shouldn't this approach reduce the bias caused by class imbalance?

Author response:
The bias in pseudo-labeling arises from disparities in class distributions between training and test data. As discussed in Theorem 1, the optimal decision boundary for test data should consider differences in marginal distributions. Current solutions fall short, either by neglecting the class prior of test data or lacking calibrated probabilities, as illustrated in section 4.1. 
We have emphasized the reason for class bias and the problems of current methods in section 4.1.

Reviewer comment (Presentation):
The presentation could be further improved, specifically on the layout of Figures 3-5 and figure 7-9.

Author response:
Current dynamic threshold methods (including FlexMatch and FreeMatch) mostly adjust thresholding based on maximum class probability (MCP), which is an approximation of recall per class. However, as we analyze in Section 4.2, the optimal thresholds should correlate with precision rather than recall. Therefore, as we demonstrate in a two-moon toy example, existing MCP-based methods would fail two out of the four threshold adjustment scenarios, therefore not optimal. In contrast, SEVAL can accommodate all the four cases. We also validate this in CIFAR10-LT experiments in section D. Moreover, we  show that the selected pseudo-labels are more correct than FlexMatch and FreeMatch in Figure 3(a).
We have rephrased the method section to emphasize these differences with more analysis and evidence.

Reviewer comment (Theory):
This paper is overall heuristic, the proposed solution lacks theoretical justification.

Author response:
We thank the reviewers for the suggestions. We now polish the layout of all the figures, including the mentioned two (now Figure 3 and Figure 4).

Reviewer comment (Experiments):
The authors are encouraged to improve the proposed method in situations where the labeled set is not big enough.

Author response:
We have revised the method section and provided SEVAL with its theoretical foundation. Additionally, we have formulated the key hypothesis underpinning the success of SEVAL. While SEVAL, like other SSL algorithms, is primarily heuristic, we consider it valuable as our analysis offers new insights into the pseudo-label refinement problem, and we support the effectiveness of SEVAL through comprehensive experiments.

Reviewer comment (Theory):
I failed to conceive how curriculum learning reconciles with the threshold adjustment, is there a process where SEVAL reordering the data? Or the curriculum in this context is different from the commonly referred curriculum learning [1]?

Author response:
As also discussed in Q1, SEVAL works well when the labelled set is small. In particular, SEVAL accomplishes this through frequency normalization and group-wise optimization, as described in section 4.2 and detailed in Section C.