Paper: Unleashing the Power of Annotation: Enhancing Semi-Supervised Learning through Unsupervised Sample Selection (ICLR-2024)

Reviewer comment (Experiments):
limited comparison to other sampling methods (only random sampling)

Author response:
We are currently in the process of replicating some other methods, and we will provide a response with the results once this task is completed.

Reviewer comment (Evaluation):
marginal improvement over random sampling (Tab. 1 overlapping confidence intervals)

Author response:
Current semi-supervised algorithms have already achieved relatively superior performance, making further enhancements a challenging and complex endeavor. Our methodology demonstrates limited improvements when the annotation budget is substantial. However, it shows considerable advancements in scenarios with constrained budgets. As shown in Table 1 of the manuscript, when employing UPA, Flexmatch achieves an accuracy improvement of 4.4% on the CIFAR-10 dataset with an annotation budget of 40, a 2.5% increase on the CIFAR-100 dataset with a budget of 400, and a 3.6% rise on the SVHN with a budget of 250. These findings align with intuitive conjecture: as the annotation budget increases, regardless of the sampling method employed, there is a greater likelihood of acquiring a more comprehensive set of samples. Consequently, the disparity between different sampling methods diminishes. Therefore, the superiority of a sampling method becomes more pronounced with a smaller annotation budget.

Reviewer comment (Experiments):
Uncertainty about whether the chosen application is the right one; suggestion to focus on the selection step to approximate the distribution (estimated from the large dataset) with few examples.

Author response:
The process of sample selection in AL is inherently coupled with model training. Thus, the selection procedure iterates in tandem with the model training, where a subset of samples is chosen in each iteration for labeling and training until the budget is exhausted. However, our approach entails a one-time selection of all samples for labeling. This results in our sample selection algorithm being inapplicable to the AL paradigm, thereby posing challenges in direct comparison with other AL methods.

In order to provide a more intuitive comparison between our sampling method and AL sampling methods in Related Work, while considering that AL sampling methods only utilize labeled data for supervised training, we devised a new comparative experiment under the same labeling budget constraint. First, we employed our method to select samples for annotation, followed by using these annotated samples for supervised classifier training. This step was undertaken to demonstrate the performance contrast between different sampling methods in supervised learning. The table below presents the experimental results.

| Methods | Budget | Acc (%) |
| --- | --- | --- |
| *Active Learning* |  |  |
| VAAL | 7500 | 86.8 |
| UncertainGCN | 7500 | 86.8 |
| CoreGCN | 7500 | 86.5 |
| MCDAL | 7500 | 87.2 |
| *Supervised Learning* |  |  |
| Ours | 7500 | 86.6 |
||||

It is worth noting that within AL framework, sample selection is tightly coupled with model training, where model training is dynamic, and subsequent sample selection becomes more precise as performance improves. This is an iterative process that continues until the labeling budget is exhausted. In contrast to our sampling method, all samples that require annotation are determined before training (*i.e.,* in the absence of knowledge about the training data). The challenge with this approach is that all samples must be selected initially, and there is no flexibility to adjust sample selection based on intermediate training results. Although theoretically, this may lead to lower performance compared to dynamic sample selection in AL methods, experimental results demonstrate that even under a static selection framework, our method can achieve performance similar to AL. This indirectly reflects the advantages of our method.

Reviewer comment (Novelty):
Lack of comparison of the proposed approach to related work, and insufficient discussion of pros and cons.

Author response:
The process of sample selection in AL is inherently coupled with model training. Thus, the selection procedure iterates in tandem with the model training, where a subset of samples is chosen in each iteration for labeling and training until the budget is exhausted. However, our approach entails a one-time selection of all samples for labeling. This results in our sample selection algorithm being inapplicable to the AL paradigm, thereby posing challenges in direct comparison with other AL methods.

In order to provide a more intuitive comparison between our sampling method and AL sampling methods in Related Work, while considering that AL sampling methods only utilize labeled data for supervised training, we devised a new comparative experiment under the same labeling budget constraint. First, we employed our method to select samples for annotation, followed by using these annotated samples for supervised classifier training. This step was undertaken to demonstrate the performance contrast between different sampling methods in supervised learning. The table below presents the experimental results.

| Methods | Budget | Acc (%) |
| --- | --- | --- |
| *Active Learning* |  |  |
| VAAL | 7500 | 86.8 |
| UncertainGCN | 7500 | 86.8 |
| CoreGCN | 7500 | 86.5 |
| MCDAL | 7500 | 87.2 |
| *Supervised Learning* |  |  |
| Ours | 7500 | 86.6 |
||||

It is worth noting that within AL framework, sample selection is tightly coupled with model training, where model training is dynamic, and subsequent sample selection becomes more precise as performance improves. This is an iterative process that continues until the labeling budget is exhausted. In contrast to our sampling method, all samples that require annotation are determined before training (*i.e.,* in the absence of knowledge about the training data). The challenge with this approach is that all samples must be selected initially, and there is no flexibility to adjust sample selection based on intermediate training results. Although theoretically, this may lead to lower performance compared to dynamic sample selection in AL methods, experimental results demonstrate that even under a static selection framework, our method can achieve performance similar to AL. This indirectly reflects the advantages of our method.

Reviewer comment (Presentation):
Missing brief sketch of various applications (including SSL) that would benefit from the approach.

Author response:
The process of sample selection in AL is inherently coupled with model training. Thus, the selection procedure iterates in tandem with the model training, where a subset of samples is chosen in each iteration for labeling and training until the budget is exhausted. However, our approach entails a one-time selection of all samples for labeling. This results in our sample selection algorithm being inapplicable to the AL paradigm, thereby posing challenges in direct comparison with other AL methods.

In order to provide a more intuitive comparison between our sampling method and AL sampling methods in Related Work, while considering that AL sampling methods only utilize labeled data for supervised training, we devised a new comparative experiment under the same labeling budget constraint. First, we employed our method to select samples for annotation, followed by using these annotated samples for supervised classifier training. This step was undertaken to demonstrate the performance contrast between different sampling methods in supervised learning. The table below presents the experimental results.

| Methods | Budget | Acc (%) |
| --- | --- | --- |
| *Active Learning* |  |  |
| VAAL | 7500 | 86.8 |
| UncertainGCN | 7500 | 86.8 |
| CoreGCN | 7500 | 86.5 |
| MCDAL | 7500 | 87.2 |
| *Supervised Learning* |  |  |
| Ours | 7500 | 86.6 |
||||

It is worth noting that within AL framework, sample selection is tightly coupled with model training, where model training is dynamic, and subsequent sample selection becomes more precise as performance improves. This is an iterative process that continues until the labeling budget is exhausted. In contrast to our sampling method, all samples that require annotation are determined before training (*i.e.,* in the absence of knowledge about the training data). The challenge with this approach is that all samples must be selected initially, and there is no flexibility to adjust sample selection based on intermediate training results. Although theoretically, this may lead to lower performance compared to dynamic sample selection in AL methods, experimental results demonstrate that even under a static selection framework, our method can achieve performance similar to AL. This indirectly reflects the advantages of our method.

Reviewer comment (Experiments):
Unclear how the approach copes with low and high dimensions.

Author response:
a) Using MMD to measure the representativeness is not influenced by the dimension of feature space theoretically, since MMD is actually the distance between kernel mean embeddings of full data and selected samples in a RKHS. Similarly, $\alpha$-MMD also enjoys such property. Therefore, the approach is not sensitive to the dimension of feature space.

Reviewer comment (Experiments):
Unclear how the approach copes with different distributions (overlapping, clusters well separated, etc.).

Author response:
b) The "optimal choice" is actually a recommended choice, and derived only from theoretical aspect. However, in practice, this choice seems to be effective. For well separated clusters, the algorithm is encouraged to select more data points lying on the boundaries of the clusters, which are more informative than the data points that lie aside the center of clusters. For overlapped distributions, since in the assumptions of SSL, few points are close to each other but in different classes, diversification is still a good idea for selecting informative samples. Consider that diversification prefer to select boundary points, low-density-region points and points from small clusters, the representativeness ensures that the proportion of selected points from each cluster is not too different from the proportion of full data. Nevertheless, if the cluster are sufficiently far from each other, we could encourage more diversity via decreasing the value of $\alpha$ so that more boundary points are selected.

Reviewer comment (Evaluation):
Unspecified limitations of the approach.

Author response:
c) A limitation of our method is that there is no certain criteria for choosing $\alpha$ that consider the distribution of full data. In b) we find that for overlapped distributions and clusters-well-separated distributions, choosing $\alpha$ could possibly follow different strategies. In other words, there could be a better choice (such as some distribution-dependent ones) for $\alpha$ than that we recommend.

Reviewer comment (Experiments):
The empirical validation is lacking additional datasets to support the advantage of the method.

Author response:
Your point regarding the importance of additional datasets for validating the strengths of our method is very pertinent. In this paper, due to time and resource constraints, we selected CIFAR-10, CIFAR-100, and SVHN as our primary datasets for experiments. These datasets were chosen because of their wide acceptance and representativeness in the SSL domain. Our method demonstrated significant improvements on these datasets, especially in scenarios with a limited labeling budget. However, we acknowledge that applying our method to a broader and more diverse set of datasets is necessary for a more comprehensive validation of its strengths. To this end, we have initiated preliminary experiments on the STL-10 and ImageNet datasets, which is expected to take some time. We will provide a detailed response with the results upon the completion of this task.

Reviewer comment (Evaluation):
The improvement over random baseline is too modest, with only 0.5 percent improvement when the standard deviation is at 0.5, which is not convincing.

Author response:
Current semi-supervised algorithms have already achieved relatively superior performance, making further enhancements a challenging and complex endeavor. Our methodology demonstrates limited improvements when the annotation budget is substantial. However, it shows considerable advancements in scenarios with constrained budgets. As shown in Table 1 of the manuscript, when employing UPA, Flexmatch achieves an accuracy improvement of 4.4% on the CIFAR-10 dataset with an annotation budget of 40, a 2.5% increase on the CIFAR-100 dataset with a budget of 400, and a 3.6% rise on the SVHN with a budget of 250. These findings align with intuitive conjecture: as the annotation budget increases, regardless of the sampling method employed, there is a greater likelihood of acquiring a more comprehensive set of samples. Consequently, the disparity between different sampling methods diminishes. Therefore, the superiority of a sampling method becomes more pronounced with a smaller annotation budget.

Reviewer comment (Experiments):
A simple ablation study is missing — if results of FlexMatch+UPA are shown, the results of just FlexMatch should also be presented, otherwise it is unclear if the advantage is due to FlexMatch or the addition of UPA.

Author response:
In the paradigm of Semi-Supervised Learning (SSL), a large amount of unlabeled data is initially provided, from which a subset is selected for labeling. The SSL algorithms, such as FlexMatch/FreeMatch used in our study, then leverage both the labeled and remaining unlabeled data for training. Therefore, any SSL algorithm cannot function independently without a sample selection method. The most widely used sample selection strategies currently are random sampling and stratified sampling. Stratified sampling necessitates prior knowledge of each sample's category for performing random sampling within each category, which is impractical in real-world scenarios. Consequently, random sampling serves as a feasible baseline method for comparison with our approach.

Reviewer comment (Experiments):
Only 3 independent runs are conducted, which is not enough.

Author response:
Thank you for your comment. We will increase the number of experiments conducted to reduce the randomness in the experimental results.

Reviewer comment (Experiments):
The selection of particular m values for each dataset is unclear. Why is it different for each dataset? A graph showing performance over a range of m values would be preferable.

Author response:
In the domain of SSL, most of the popular algorithms adopt such experimental settings, as referenced in [1]-[3].

[1] https://proceedings.neurips.cc/paper/2020/file/06964dce9addb1c5cb5d6e3d9838f733-Paper.pdf  
[2] https://proceedings.neurips.cc/paper/2021/file/995693c15f439e3d189b06e89d145dd5-Paper.pdf  
[3] https://openreview.net/pdf?id=PDrUPTXJI_A

Reviewer comment (Experiments):
The requirement that m/n <= 0.2 is quite weak and unrealistic. Sometimes 0.2 can be a huge dataset to annotate, which poses challenges in an Active Learning setting where the budget is limited.

Author response:
We conducted comparative experiments under different labeling budgets, where m/n=0.2 represents the scenario with the maximum labeling budget. In most comparative experiments, the value of m/n is much less than 0.2. For instance, on the CIFAR-10 dataset with a budget of 40, the ratio is m/n=40/50000=0.0008; similarly, on the SVHN dataset with a budget of 250, it is m/n=250/73257=0.0034.

Reviewer comment (Reproducibility):
It is unclear what the input space over which the method is used is. Is it the actual data? For coreset, is it the penultimate layer representation that is being used?

Author response:
The input of the sampling algorithm is the output of the image feature extractor. The specific process is as follows. First, we leverage the pre-trained image feature extraction capabilities of CLIP, which is a vision transformer architecture, to extract features. Subsequently, the [CLS] token features produced by the model’s final output are employed for sample selection.

Reviewer comment (Theory):
Remark 1 is confusing; it presents Theorem 3.1 and then claims that Theorem 3.1 doesn’t always work.

Author response:
Thm 3.1. claims that if the model (or hypothesis class) belongs to a certain RKHS, then the expected risk of the trained model (or hypothesis) is controlled. If not, then the theory is not valid. Since the methodology is designed to be model-free, we think it would be better if claim that Thm 3.1 does not work for every model.

Reviewer comment (Writing):
In Equation 6, f_I_p(x_i) appears to be defined only for x; this is unclear.

Author response:
This is definitely a typo. We are really sorry for that. Substitute $x_i$ by $x$ would be right. We will correct that in the revised version. Thank you for your carefulness.

Reviewer comment (Writing):
In Equation (8), it is unclear what K is in B=2K.

Author response:
K is defined in Thm 3.2. It is the upper bound of k(x,x) over $\mathcal{X}$.

Reviewer comment (Theory):
The computational complexity of O(mn) should be better explained. Wouldn’t the kernel construction cost more?

Author response:
The result of computational complexity almost directly follows from the result of https://arxiv.org/pdf/2101.07564.pdf. Please refer to Section 3.1. However, after numerical experiments, we find that the kernel construction indeed cost more time in high-dimensional cases. Therefore, we may revise the computation complexity by O(mnd) where d is the dimension of feature space. Thanks.

Reviewer comment (Theory):
There is skepticism about the approach of diversification and representativeness being optimal. Other important trade-offs in active learning, such as exploration-exploitation, could outperform the UPA approach.

Author response:
Diversity as a criteria of selecting samples in semi-supervised learning is theoretically studied by https://doi.org/10.1080/01621459.2023.2282644, and the simulation results in this paper shows that diversification indeed works. Representativeness is a straightforward idea to measure the sampling quality. Thus our criteria for the unsupervised sample selection task could be proper. As for optimality, we think that there does not exists optimal strategy for such unsupervised task. The exploration-exploitation framework in active learning is indeed effective and ingenious, but our unsupervised sample selection task does not require feedbacks from the model, which makes some active learning framework improper for semi-supervised learning.

Reviewer comment (Evaluation):
The reviewer questions why diversification and representativeness is considered the best approach for active learning.

Author response:
In this paper we mainly focus on the methodology of unsupervised sample selection and its application in semi-supervised learning tasks, nevertheless, we did not discuss its application in AL tasks. Maybe diversification and representativeness could be a good choice for AL, as a extension of core-set approach (only representativeness considered) proposed in https://arxiv.org/abs/1708.00489, which is a well-recognise work in the community.

Reviewer comment (Experiments):
The basic methodology essentially selects a representative and diverse subset. There are many methodologies for such a diverse subset selection, e.g., [1*]-[3*]. None of such methods have been discussed nor have been empirically compared with. This makes it difficult to evaluate the significance of the proposal.

Author response:
We appreciate your suggestion regarding the comparative methods. We are currently in the process of replicating these methods, which is expected to take some time. We will provide a response with the results once this task is completed.

Reviewer comment (Experiments):
Regarding table 2, since UPA is employed above SSL (flexmatch/freematch), it may be important to compare against baseline/SOTA AL criteria when employed with flexmatch/freematch. Then the advantage of the proposed criteria would be explicit. Now it is not clear whether the improvement is because of flexmatch/freematch or because of the proposed criteria.

Author response:
The improvement is related to both aspects. As shown in Table 1 of the manuscript, Flexmatch/Freematch already outperforms the AL/SSAL methods when our method is not used. Furthermore, with the adoption of our method, there is a further enhancement in performance.

In order to provide a more intuitive comparison between our sampling method and existing AL sampling methods, while considering that AL sampling methods only utilize labeled data for supervised training, we devised a new comparative experiment under the same labeling budget constraint. First, we employed our method to select samples for annotation, followed by using these annotated samples for supervised classifier training. This step was undertaken to demonstrate the performance contrast between different sampling methods in supervised learning. The table below presents the experimental results.

| Methods | Budget | Acc (%) |
| --- | --- | --- |
| *Active Learning* |  |  |
| VAAL | 7500 | 86.8 |
| UncertainGCN | 7500 | 86.8 |
| CoreGCN | 7500 | 86.5 |
| MCDAL | 7500 | 87.2 |
| *Supervised Learning* |  |  |
| Ours | 7500 | 86.6 |
||||

It is worth noting that within AL framework, sample selection is tightly coupled with model training, where model training is dynamic, and subsequent sample selection becomes more precise as performance improves. This is an iterative process that continues until the labeling budget is exhausted. In contrast to our sampling method, all samples that require annotation are determined before training (*i.e.,* in the absence of knowledge about the training data). The challenge with this approach is that all samples must be selected initially, and there is no flexibility to adjust sample selection based on intermediate training results. Although theoretically, this may lead to lower performance compared to dynamic sample selection in AL methods, experimental results demonstrate that even under a static selection framework, our method can achieve performance similar to AL. This indirectly reflects the advantages of our method.

Furthermore, our research aims to enhance the performance of SSL algorithms. We acknowledge that in the SSL domain, the sample selection problem has been a long-neglected and under-researched area. Consequently, there are limited existing methods available for comparison. In our paper, particularly in Table 1, we have conducted a detailed comparison of two commonly used methods in SSL algorithms. The comparative results showcase the superiority of our method, indicating that our approach can improve SSL performance, especially when the annotation budget is limited.

Simultaneously, considering practical application scenarios, especially when dealing with a large amount of unlabeled data and constrained labeling budgets, both AL, SSAL and SSL are viable options. In our paper, Table 2 compares the performance of these different methods in practical applications, aiming to illustrate the variations in results brought about by different choices. We hope this comparison can provide valuable reference and guidance to future researchers facing similar situations.

We hope that our response above addresses your concerns. If you have any further questions, we are open to conducting additional experiments and analyses to provide further clarification.