Paper: Realistic Evaluation of Semi-supervised Learning Algorithms in Open Environments (ICLR-2024)

Reviewer comment (Writing):
The manuscript lacks clarity in some of its detail descriptions, such as the explanation of the variable 't', which is not very comprehensible. The author could provide a detailed example to help readers better understand the meaning and function of 't'.

Author response:
Regarding the weakness 1. Thank you for your suggestion. In the revised version, we have added illustrative examples for the variable 't'. Please refer to the third line on the second page.

Reviewer comment (Presentation):
In Figure 2, there are too many methods presented, making it difficult to read. It would be sufficient to showcase a comparison of some representative methods instead.

Author response:
Regarding the weakness 3. Thank you for your suggestion. We have addressed our formatting issues in the new version.

Reviewer comment (Evaluation):
The manuscript introduces numerous evaluation metrics; however, the inherent relationships among these metrics are not very clear. The individual motivations for proposing each of these metrics could be further elucidated.

Author response:
Regarding the weakness 4. We discussed the motivation for each metric at the end of the third section of the paper. In the revised version, we have further supplemented the introduction of motivations.

Reviewer comment (Theory):
The current theoretical results appear to be applicable only when the degrees of inconsistency across the three types of variations are the same. It raises the question of whether the theory would still hold if the degrees of inconsistency were different.

Author response:
Regarding the question. This is a good question, and we appreciate the insight you provided. In fact, in scenarios where multiple inconsistencies exist, a variable 't' can only be used when the degrees of inconsistency are the same. For situations with varying degrees of inconsistency, we just need to set multiple corresponding variables 't'. In such cases, the theory will still hold by some simple variable substitutions.

Reviewer comment (Experiments):
It appears that some algorithms were only evaluated on tabular and image datasets, with no evaluation on text datasets.

Author response:
Dear reviewer:

Thank you for the feedback on our paper. We appreciate the time and effort you have put into reviewing our work. In this rebuttal, we respond to the concerns raised in the reviews.

Regarding the question and the weaknesses. Certain methods have not been employed in the evaluation of text data, as they were initially designed for image data. For instance, numerous algorithms utilize Mixup operations designed for image data. While these algorithms can be adapted for tabular data to some extent, their applicability to text data lacks generality and rationale. As a result, we did not evaluate them in experiments involving textual data. We will clarify this point in the new version of the paper.

Many thanks.

Reviewer comment (Experiments):
Some algorithms have been evaluated on image and tabular datasets but not on text datasets. What could be the reason for this difference?

Author response:
Dear reviewer:

Thank you for the feedback on our paper. We appreciate the time and effort you have put into reviewing our work. In this rebuttal, we respond to the concerns raised in the reviews.

Regarding the question and the weaknesses. Certain methods have not been employed in the evaluation of text data, as they were initially designed for image data. For instance, numerous algorithms utilize Mixup operations designed for image data. While these algorithms can be adapted for tabular data to some extent, their applicability to text data lacks generality and rationale. As a result, we did not evaluate them in experiments involving textual data. We will clarify this point in the new version of the paper.

Many thanks.

Reviewer comment (Presentation):
Although the content of this paper is sufficient and informative (up to 46 pages), the main paper contains excessive text and lacks tables and figures to present the experiment results and conclusions.

Author response:
Regarding the weaknesses 1. Thank you very much for your suggestions. We will address formatting issues in the revised version of the paper, addressing the concerns of excessive text in the main body.

Reviewer comment (Evaluation):
This paper proposes various metrics to evaluate the robustness of existing semi-supervised learning methods in different aspects. Is it possible to propose a unified method for ranking existing methods, as there are many experiment results but we cannot know which algorithm is better?

Author response:
Regarding the question 1. It is possible to rank algorithms based on any metric, but the obtained results only represent a specific aspect. For instance, the algorithm with the best average accuracy, the one with the best worst-case accuracy, or the one with the smallest performance variation. In practical applications, it is essential to establish ranking criteria according to the user's requirements for the algorithm. If necessary, comparisons can also involve weighted considerations for different metrics. We summarized the robustness performance of the compared algorithms in Tables 3 and 4. However, these rankings based on metrics only capture a partial aspect of the algorithm's characteristics.

Reviewer comment (Evaluation):
The main weakness, as discussed in the limitations section by the authors, lies in the fact that the proposed evaluation framework cannot assess highly complex real-world scenarios with low calculation complexity.

Author response:
Regarding the weakness. Indeed, evaluation complexity rises with the environment's complexity. In complex environments, we must strike a balance between high precision and low complexity in evaluations.

Reviewer comment (Evaluation):
In practical applications, how can the metrics for measuring real-world inconsistency be correlated with the inconsistency in the evaluation framework?

Author response:
Regarding the question. In practical applications, specific metrics may be necessary to quantify inconsistencies. Considering the scenario where all unlabeled samples are inconsistent samples: in the case of inconsistency in data distributions, the degree of inconsistency can be directly measured by numerous metrics in the field of transfer learning such as $\mathcal{H}\Delta\mathcal{H}$ distance; in the case of inconsistency in label or feature spaces, the degree of inconsistency can be directly measured by the proportion of shared classes or features between labeled and unlabeled data compared to the total number of classes or features. Considering the scenario where unlabeled samples containing not only inconsistent samples but also consistent samples: the proportion of inconsistent samples can also be used as the metric to measure the inconsistency. In the end, these inconsistency metrics can be normalized and linked to the variable 't'.