Paper: Rethinking pseudo-labeling: Data-centric insights improve semi-supervised learning (ICLR-2024)

Reviewer comment (Theory):
This paper improves the robustness of SSL against noisy labeled data through a simple data selection method. However, further analysis is not provided regarding the reasons for the success of this data selection. Is there a theoretical connection between the statistical features of historical predictions (average confidence and aleatoric uncertainty in this paper) and the improvement in pseudo-label quality in SSL? Under what conditions does this method work effectively for the overall quality of labeled data, such as the proportion of label noise? More in-depth analysis can further improve this paper.

Author response:
### (B) Analysis of conditions of success
We thank the reviewer for the question on conditions of success of DIPS. To answer it, we conduct a synthetic experiment following a similar setup as in Sec. 5.1 in our manuscript. Our setup considers the dependencies between the amount of label noise and the amount of labeled data. Note that the experiment is synthetic in order to be able to control the amount of label noise.
We considered the same list of label noise proportions as in Sec. 5.1, ranging from 0. to 0.45. For each label noise proportion, we consider $n_{\mathrm{lab}} \in$ {50,100,1000}, and fix $n_{\mathrm{unlab}} = 1000$.
For each configuration, we conduct the experiment $40$ times. 
We report the results in the following plots available at: **https://i.imgur.com/rzFeFsl.png**

As we can see on the plots, PL+DIPS consistently outperforms the supervised baselines and PL in almost all the configurations. The performance gap between DIPS and PL is remarkably noticeable for $n_{\mathrm{lab}}=1000$, that is, when the amount of labeled samples is high, which hints at the fact that curation is made easier with more samples, and hence leads to improved performance.
When the amount of labeled data is very low ($n_{lab} = 50$) and the proportion of corrupted samples is high ($0.45$), we can see that the different baselines have closer accuracies. We note, though, that DIPS consistently improves the PL baseline for reasonable amounts of label noise which we could expect in real-world settings (e.g. 0.1).

**UPDATE:** We have included these results in Appendix C.5 of the revised manuscript.


The reviewer also asks about any "theoretical connection" involving "statistical features of historical predictions". Some previous works in the *supervised learning setting* have shown that learning dynamics emit a strong signal about the nature of samples ([R3],[R4]). For example, Theorem 3 in [R5] (Appendix A.4 in the corresponding paper) gives some intuition about mislabeled samples in the supervised setting, by stating that the probability that mislabeled examples are classified with their given (incorrect) labels tends to $0$ in a mixture setting, as the model is trained until convergence. As such, this phenomenon directly impacts the agreement between the predictions of the model and the given label of a mislabeled sample, the agreement being worse as the model converges. This agreement is directly the quantity that we capture with the learning dynamics (confidence and aleatoric uncertainty) and DIPS' selection mechanism $s$.

Reviewer comment (Evaluation):
Can the author provide more discussion and analysis to demonstrate the conditions under which this proposal is successful?

Author response:
### (B) Analysis of conditions of success
We thank the reviewer for the question on conditions of success of DIPS. To answer it, we conduct a synthetic experiment following a similar setup as in Sec. 5.1 in our manuscript. Our setup considers the dependencies between the amount of label noise and the amount of labeled data. Note that the experiment is synthetic in order to be able to control the amount of label noise.
We considered the same list of label noise proportions as in Sec. 5.1, ranging from 0. to 0.45. For each label noise proportion, we consider $n_{\mathrm{lab}} \in$ {50,100,1000}, and fix $n_{\mathrm{unlab}} = 1000$.
For each configuration, we conduct the experiment $40$ times. 
We report the results in the following plots available at: **https://i.imgur.com/rzFeFsl.png**

As we can see on the plots, PL+DIPS consistently outperforms the supervised baselines and PL in almost all the configurations. The performance gap between DIPS and PL is remarkably noticeable for $n_{\mathrm{lab}}=1000$, that is, when the amount of labeled samples is high, which hints at the fact that curation is made easier with more samples, and hence leads to improved performance.
When the amount of labeled data is very low ($n_{lab} = 50$) and the proportion of corrupted samples is high ($0.45$), we can see that the different baselines have closer accuracies. We note, though, that DIPS consistently improves the PL baseline for reasonable amounts of label noise which we could expect in real-world settings (e.g. 0.1).

**UPDATE:** We have included these results in Appendix C.5 of the revised manuscript.


The reviewer also asks about any "theoretical connection" involving "statistical features of historical predictions". Some previous works in the *supervised learning setting* have shown that learning dynamics emit a strong signal about the nature of samples ([R3],[R4]). For example, Theorem 3 in [R5] (Appendix A.4 in the corresponding paper) gives some intuition about mislabeled samples in the supervised setting, by stating that the probability that mislabeled examples are classified with their given (incorrect) labels tends to $0$ in a mixture setting, as the model is trained until convergence. As such, this phenomenon directly impacts the agreement between the predictions of the model and the given label of a mislabeled sample, the agreement being worse as the model converges. This agreement is directly the quantity that we capture with the learning dynamics (confidence and aleatoric uncertainty) and DIPS' selection mechanism $s$.

Reviewer comment (Experiments):
While the authors assert that DIPS is intended to be a versatile tool that can be seamlessly merged with current pseudo-labeling strategies, the experiments are mainly limited to tabular datasets. It would be beneficial to extend testing to the USB[1] benchmark, encompassing datasets from computer vision, natural language processing, and speech to better demonstrate DIPS's generalizability. I will reconsider my score if more experiments are conducted.

Author response:
### (A) Additional experiments with the USB benchmark 
We thank the reviewer for this good suggestion to include datasets from the suggested USB benchmark, which, we believe, will strengthen our submission. To address this point, we have conducted experiments on $4$ additional datasets: satellite images from Eurosat [R1] and medical images from TissueMNIST which form part of the USB benchmark [R3]. Additionally, we include the related OrganAMNIST, and PathMNIST, which are part of the MedMNIST collection [R2].
Given that these datasets are well-curated, we consider a proportion of $0.2$ of symmetric label noise added to these datasets.

We report the results in the figure available at **https://i.imgur.com/NYmYGHs.png**


As is shown, DIPS consistently improves the FixMatch baseline, demonstrating its generalizability beyond tabular datasets.

Additionally, we wish to reiterate that beyond CIFAR-10N, we have also conducted additional experiments on CIFAR-100N in Appendix C.4.1.

**UPDATE:** we have included these additional results in Appendix C.4.4 in our revised manuscript.

Reviewer comment (Presentation):
The sections on Notation and Methodology are challenging to interpret. Transferring the pseudo-code to the Methodology section could enhance clarity and comprehension.

Author response:
### (B) Placement of pseudo-code
Thank you for your suggestion to improve the presentation of our work. We agree with you and have moved the pseudo-code presented in Algorithm 1 to Section 4.4 in the updated manuscript.

**UPDATE:** Algorithm 1 moved to Section 4.4 in the revised manuscript.

We hope this answers your points, please let us know if there are any remaining concerns.

Reviewer comment (Novelty):
It seems that this paper is not the first to concern the inherent label noise in the labeled data and take the data-centric approach. It is necessary to discuss and compare with [L. Schmarje et al.,2022] to more clarify the conceptual novelty.

Author response:
Thank you for highlighting the paper [L. Schmarje et al.,2022]. While both DIPS and DC3 handle the data-centric issue of issues in data and share similarities in their titles, they tackle different data-centric problems which might arise in semi-supervised learning. The main differences are along 4 different dimensions.

(i) **Problem setup/Type of data-centric issue**: DIPS tackles the problems of hard noisy labels where each sample has a single label assigned in the labeled set which could be incorrect. In contrast, DC3 deals with the problem of soft labeling where each sample might have multiple annotations from different annotators which may be variable.

(ii) **Label noise modeling**: DIPS aims to identify the noisy labels, whereas DC3 models the inter-annotator variability to estimate label ambiguity.

(iii) **Integration into SSL**: DIPS is a plug-in on top of any pseudo-labeling pipeline, selecting the labeled and pseudo-labeled data. DC3 on the other hand uses its ambiguity model (learned on the multiple annotations) to either keep the pseudo-label or use a cluster assignment.

(iv) **Dataset applicability**: DIPS has lower dataset requirements as it can be applied to any dataset with labeled and unlabeled samples, even if there is only a single label per sample. It does not require multiple annotations. DC3 has higher dataset requirements as it relies on having multiple annotations per sample to estimate inter-annotator variability and label ambiguity. Without multiple labels per sample, it cannot estimate ambiguity and perform joint classification and clustering. Consequently, DIPS is applicable to the standard semi-supervised learning setup of limited labeled data and abundant unlabeled data, whereas DC3 targets the specific problem of ambiguity across multiple annotators.

**UPDATE:** we have added this discussion in realtion to Schmarje et al.,2022 to Appendix A.4 in our revised manuscript.

Reviewer comment (Experiments):
Connection to the active learning literature is missing. Selection metric for the ‘useful’ data samples among the unlabeled data pool is of central interest in active learning. The term ‘usefulness’ can include various criterions, such as confidence and uncertainty (as in this work), coverage, diversity, and etc. In that sense, more comprehensive discussions and comparisons on selection metrics are expected. Related to this issue, how the quality and diversity (e.g., class distributions) of the selected training samples change for every generation?

Author response:
We wish to clarify the difference in setting of DIPS vs active learning. We agree that the concept of 'usefulness' in the selection of data samples is a significant aspect of both active learning and our work. However, it is crucial to highlight the distinct contexts in which this term is used in both settings.

Active learning primarily focuses on the iterative process of selecting data samples that, when labeled, are expected to most significantly improve the model's performance. This selection is typically based on criteria such as uncertainty sampling which focuses on **epistemic uncertainty** [R1-R4]. The primary objective is to minimize labeling effort while maximizing the model's learning efficiency.

In contrast, DIPS does both labeled and pseudo-labeled selection and employs the term 'useful' in a different sense. Here, 'usefulness' refers to the capacity of a data sample to contribute positively to the learning process based on its likelihood of being correctly labeled. Our approach, which leverages training dynamics based on **aleatoric uncertainty** and confidence, is designed to flag and exclude mislabeled data. This distinction is critical in our methodology as it directly addresses the challenge of data quality, particularly in scenarios where large volumes of unlabeled data are integrated into the training process.

In active learning, these metrics are used to identify data points that, if labeled, would yield the most significant insights for model training. In our approach, they serve to identify and exclude data points that could potentially deteriorate the model's performance due to incorrect labeling. 

**UPDATE:** We have included this discussion in Appendix A.5 in our revised manuscript.

Reviewer comment (Experiments):
Choice of the threshold parameters $\tau_{\text{conf}}$ and $\tau_{\text{al}}$ seems to be very important. For example, highly tight thresholds (i.e., high $\tau_{\text{conf}}$ and low $\tau_{\text{al}}$) will remain only a few samples for training likely to be correct and abundant samples yet include noise for the vice versa. As the proposed algorithm is not designed to make a correction on the mislabeled samples but filter potentially harmful examples, exploring such trade-offs between remaining data proportion and performance depending on the thresholds will be valuable.

Author response:
We thank the reviewer for the question on the choice of threshold parameters. To address this point, we conducted an experiment in the synthetic setup where we varied the thresholds used for both the confidence and the aleatoric uncertainty. In addition to our choice used in our manuscript (confidence threshold = 0.8, and adaptive threshold on the aleatoric uncertainty), we consider two baselines:
1) confidence threshold = 0.9 and uncertainty threshold = 0.1 (aggressive filtering)
2) confidence threshold = 0.5 and uncertainty threshold = 0.2 (permissive filtering)

We show the test accuracy for these baselines in the following plot available at: https://i.imgur.com/QCYEgzm.png.

As we can see, our configuration outperforms both the aggressive filtering configuration (red line) and a permissive one (blue line), which is why we adopt it for the rest of the experiments. We empirically notice in Section 5.2 that it performs well on the 12 real-world datasets we used.

**UPDATE:** we have included these experimental results in Appendix C.8 in our revised manuscript.

Reviewer comment (Experiments):
Considering the computational aspects, it could be nearly free when the scale of unlabeled data is relatively small (i.e., in vision domain, CIFAR-10/100). In other words, when the scale of the unlabeled dataset grows large (i.e., million-scale samples such as ImageNet), computational overheads caused by evaluating labeled and pseudo-labeled examples with all checkpoints in previous rounds cannot be ignored.

Author response:
Thank you for asking a question about the computational overheads in our method, particularly in relation to large-scale datasets. While evaluating labeled and pseudo-labeled examples to compute the learning dynamics might incur additional overhead, we want to highlight that our selection mechanism yields a direct improvement in the quality of data used for training. This effect leads to a quicker model convergence. We refer to Fig. 8) a) in our manuscript, which gives the time efficiency of DIPS for the dataset CIFAR-10N. It shows that DIPS accelerates convergence by a factor 1.5-4X, despite the computation of learning dynamics.

Reviewer comment (Theory):
The proposed DIPS framework follows the iteration-based self-training scheme, while a typical pseudo-label-based SSL algorithm such as FixMatch doesn’t. FixMatch takes both labeled and unlabeled data and learns from both supervision signals (i.e., labels and online pseudo-labels). But the self-training scheme makes offline pseudo-labels after an iteration phase and some selected pseudo-labeled examples considered labeled data in the self-training next iteration. It seems to be a conflict between two different mechanisms. Hence, an illustrative example of how the non-iteration-based SSL algorithms can be incorporated into the DIPS framework is expected.

Author response:
We thank the reviewer for asking this question on the integration of DIPS in non-iteration-based SSL.
The DIPS framework necessitates two ingredients: a list of model checkpoints, and a label for each sample for which we want to compute the learning dynamics. In the case of non-iteration-based SSL methods, DIPS updates the learning dynamics at every training step and performs its selection of labeled and unlabeled data every $k$ training steps (for example, a training step could denote an epoch). For the unlabeled data, DIPS uses the pseudo-labels computed by the model at the beginning of every learning dynamics cycle. Then, after $k$ training steps, DIPS curates the labeled and unlabeled data using its selector function and then updates the pseudo-labels used for the computation of learning dynamics on unlabeled data for the next $k$ training steps.

Reviewer comment (Theory):
"PL methods do not update the pseudo-labels of unlabeled samples once they are incorporated in one of the $\mathcal{D}_{train}^{(i)}$ -- authors incorrectly (check out e.g. Xie, Qizhe, et al. "Self-training with noisy student improves imagenet classification." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020) formulate teacher-student pseudo-labeling widely used and thus overstated issues in second part of sec 3.2. On the next iteration of teacher-student training all unlabeled data are relabeled, some selection based on confidence or/and uncertainty is applied to pseudo-labels and then labeled data and these new pseudo-labeled data are combined (or labeled data can be skipped) to train new student model. I never saw in prior works on any teacher-student training (when new student is trained from scratch on new data) for images, text and speech input data that old pseudo-labels (from older teacher) are used along with new ones (with latest teacher).

Author response:
### (E) Terminology of pseudo-labeling

We thank the reviewer for the comment about the terminology of teacher-student pseudo-labeling and apologize if our description of this paradigm was not clear enough in our manuscript.
We decided to describe the common teacher-student pseudo-labeling methodology adopted in the **tabular setting**. As a consequence, we used the implementation provided by [R1], which grows the training set with pseudo-labels generated at the current iteration, thus keeping old pseudo-labels in the training dataset in subsequent iterations. 

In addition to adopting this practice, we investigated this choice experimentally, by comparing between two versions of confidence-based pseudo-labeling: 
- Version 1): with a growing set of pseudo-labels (as followed by the implementation of [R1] and our paper) 
- Version 2): without keeping old pseudo-labels, as suggested by the reviewer.

We evaluate these two methods in the synthetic setup described in Section 5.1 of our manuscript, and report the test accuracy in the plot available at: https://i.imgur.com/VSPg1mD.png.



The red line corresponds to Version 1) (the implementation we used in the manuscript), while the green line corresponds to Version 2) (suggested by the reviewer).
As we can see, in this tabular setting, growing a training set by keeping the pseudo-labels generated at each iteration leads to the best results, motivating our adoption of this pseudo-labeling methodology used in the tabular setting.

**UPDATE:** we have included this experiment in Appendix C.6 of our revised manuscript.

Reviewer comment (Experiments):
Absence of empirical analysis showing that confidence filtering of labeled data is not enough and aleatoric uncertainty is necessary. Also there are no baselines with widely used uncertainty-based filtering for pseudo-labeled data to be used for labeled data -- motivating usage of aleatoric uncertainty.

Author response:
### (C) Importance of aleatoric uncertainty

We thank the reviewer for the suggestion. We conducted an ablation study where we removed the aleatoric uncertainty and only kept a confidence-based selection (with threshold = 0.8). We term this confidence ablation to highlight if there is indeed value to the aleatoric uncertainty component of DIPS.

We report results in the following table, for the $12$ tabular datasets used in Section 5.2, which shows the benefit of the two-dimensional selection criterion of DIPS. Of course, in some cases there might not be a large difference with respect to our confidence ablation --- however we see that DIPS provides a statistically significant improvement in most of the datasets. Hence, since the computation is negligible, it is reasonable to use the 2-D approach given the benefit obtained on the noisier datasets.



|                   | DIPS          |  Confidence Ablation |
|-------------------|---------------------|---------------------|
| adult             | **82.66 ± 0.10**    | 82.13 ± 0.16        |
| agaricus-lepiota  | **65.03 ± 0.25**        | 64.38 ± 0.23   |
| blog              | **80.58 ± 0.10**    | 80.22 ± 0.33        |
| credit            | **81.39 ± 0.07**    | 79.76 ± 0.15        |
| covid             | **69.97 ± 0.30**    | 69.28 ± 0.40        |
| compas            | **65.34 ± 0.25**    | 64.69 ± 0.25         |
| cutract           | **68.60 ± 0.31**    | 66.32 ± 0.12         |
| drug              | **78.16 ± 0.26**    | 75.37 ± 0.71        |
| higgs             | **81.99 ± 0.07**    | 81.42 ± 0.16        |
| maggic            | **67.60 ± 0.08**    | 66.26 ± 0.18          |
| seer              | **82.74 ± 0.08**    | 82.02 ± 0.15       |


**UPDATE:** we have included these results in Appendix C.10 of our revised manuscript.



While DIPS builds on a notion of uncertainty, we highlight that this uncertainty differs in nature from the uncertainty metrics used in the pseudo-labeling literature.
One key difference is that DIPS computes the aleatoric uncertainty based on some given labels (provided labels for $D_{lab}$ or pseudo-label for $D_{unlab}$), and then computes the agreement between the prediction of a model and these given labels as is captured by the quantity $[f_{e}(x)]_{y}$. This is very different in nature from the uncertainty metrics used in pseudo-labeling, which traditionally do not rely on a given label as they are applied to *unlabeled data* (as is done when using an ensemble of models and computing a standard deviation to define an uncertainty metric). This explains why these uncertainty based filtering methods only focus on unlabeled data, while ours can handle both labeled and pseudo-labeled data.

Reviewer comment (Experiments):
Absence of empirical justification of the proposed selection method for unlabeled data only, as then we also have mislabeling and thus it should be effective there too assuming all labeled data are correctly labeled. It could be also that maybe proposed selection is only needed for labeled data while any prior selection methods could be used for unlabeled data.

Author response:
### (D) Selection of unlabeled data

To answer the Reviewer's question regarding the selection of unlabeled data in DIPS, we refer to Figure 10 in our submitted manuscript (Appendix C.1). This figure shows results when our selection mechanism is only applied to the labeled data and we only use prior selection methods for the unlabeled data (this baseline is denoted as "A1"). The results show DIPS outperforms this baseline by a large margin. This justifies why we apply our selection method both to labeled and unlabeled data. We have updated our manuscript (Section 4.1) to more clearly highlight this result - thank you for the suggestion!

Reviewer comment (Experiments):
Training of "greedy PL" in sec. 5.1 (which is the latter type of PL methods) is out of initial formulation of PL in sec 3.1. Moreover, there is no results in Fig 3 for the zero corrupted samples, it seems PL itself it not improving upon supervised training in this toy example which is strange (looking at x=0.1)

Author response:
Finally, to address the query about Fig 3 with zero corrupted samples, we add to the results in Fig.3 the zero corruption setting in the updated plot available at: https://i.imgur.com/EBUYDkY.png.

In the zero corruption setting, all methods almost achieve perfect test performance. 

**UPDATE:** we have updated Fig. 3)c) in our revised manuscript.


We also highlight that pseudo-labeling does always increase performance over the purely supervised approach, which is natural since PL has access to unlabeled data. However, we are interested in a setting where the labeled data is subject to **mislabeling**. Hence, this explains why the margin between PL and Supervised in Fig. 3)c) is not bigger: noise propagates to the unlabeled data, which limits the effectiveness of PL and justifies the data-centric lens that DIPS adopts, hence giving a big performance gap with the baselines.

Reviewer comment (Experiments):
Baselines in sec 5.2 are a mixture of both methods, which is not aligned again with formulation in sec 3.1.

Author response:
**Baseline implementations:** The implementation of the baselines in our tabular experiments (Section 5.2) trains the models from scratch, while FixMatch in our images experiment (Section 5.5) is continuous model training. Hence, the methods in Section 5.2 are not implemented as a mixture of these two types of methods, they are implemented as teacher-student methods. We used their implementation provided by the authors of [R1] at the following link: https://github.com/amzn/confident-sinkhorn-allocation. We will clarify this distinction between teacher-student training and continuous model training in our revised manuscript.

Reviewer comment (Theory):
I do not agree entirely with statement "labeled data are noisy" as if labeled data are very limited in a lot of applications we could ask for the golden (correct) labels, as we need only small amount.

Author response:
- *"I do not agree entirely with statement "labeled data are noisy \& if small amounts we can request the gold standard"*

The issue of labeled data errors are prevalent in many industries from healthcare to finance. This is even the case in seemingly curated datasets as shown by Northcutt et al., where label error rates of widely-used benchmark datasets can reach up to 10\%. As mentioned in Section 1, while it might appear possible to manually inspect the data to identify errors in the labeled set, we note that this requires domain expertise and is human-intensive, especially in modalities such as tabular data where inspecting rows in a spreadsheet can be much more challenging than reviewing an image. In other cases, updating labels is actually infeasible due to rerunning costly experiments in domains such as biology and physics, or indeed impossible due to lack of access to either the underlying sample or equipment.

Reviewer comment (Novelty):
"application: these works use pseudo-labeling as a tool for supervised learning, whereas DIPS extends the machinery of pseudo-labeling itself." I don't see really huge difference here, as noisy labels means - we don't know the correct label, so mathematically it is very close tasks and solutions.

Author response:
- *"application: these works use pseudo-labeling as a tool for supervised learning, whereas DIPS extends the machinery of pseudo-labeling itself." I don't see really huge difference here, as noisy labels means - we don't know the correct label, so mathematically it is very close tasks and solutions.*

We agree that mathematically, the auditing task is similar. The main difference is how semi-supervised learning is used. In the first setting SSL is a solution to noisy labels, whereas we study and resolve the effect of noisy labels on SSL.

Reviewer comment (Experiments):
Why do we use checkpoints for the learning dynamic being epoch and not some parameter based on number of iterations? For very large data we will never do 1 epoch, or only 1 epoch, or only few and thus measure based on epochs will be weak. From appendix info it is not clear even what checkpoints are selected and how this selection important.

Author response:
- *"why do we use checkpoints for the learning dynamic being epoch and not some parameter based on number of iterations? For very large data we will never do 1 epoch, or only 1 epoch, or only few and thus measure based on epochs will be weak."*

We first note that DIPS only needs models which are trained in an iterative manner. For example, in our tabular experiments in Section 5.2, we use an XGBoost backbone, where the learning dynamics are computed over the different boosting iterations. Hence what we call a checkpoint in our manuscript (Section 4.2) needs not be an epoch. For neural networks, we do not necessarily need to define checkpoints as epochs, when the dataset is large; we can instead define checkpoints after every $k$ gradient steps.

Reviewer comment (Experiments):
Did authors try to exclude some first epochs from confidence and uncertainty definition as they could be not very informative?

Author response:
- *"did authors try to exclude some first epochs from confidence and uncertainty definition as they could be not very informative?"*

We thank the reviewer for this suggestion. We have conducted an additional experiment to investigate the choice of the range of iterations used to compute the learning dynamics. We consider ignoring the first $25$%/$50$%/$75$% iterations and use the remaining iterations to compute the learning dynamics.

The plot available at https://i.imgur.com/QzNnHba.png

It shows the mean performance difference (on test accuracy) by using the truncated iteration windows versus using all the iterations, and averages the results over the 12 datasets used in Section 5.2. As we can see, it is better to use all the iterations window, as the initial iterations carry some informative signal about the hardness of samples.
This motivates our choice of computing the learning dynamics over the whole optimization trajectory, a choice which we adopt for all of our experiments.

**UPDATE:** we have added these experimental results in Appendix C.7 of our revised manuscript.

Reviewer comment (Theory):
I don't understand this statement "Recall aleatoric uncertainty captures the inherent data uncertainty, hence is a principled way to capture issues such as mislabeling". Why does aleatoric uncertainty capture mislabeling? Why def. 4.2 is aleatoric as we consider variation across checkpoints (= over learning process)?

Author response:
- *"Why does aleatoric uncertainty capture mislabeling? Why def. 4.2 is aleatoric as we consider variation across checkpoints (= over learning process)?"*

Def. 4.2 of aleatoric uncertainty builds on the variability that originates from the inability to predict the correct label with high confidence over the different checkpoints. Intuitively, a sample with high aleatoric uncertainty would be a sample for which we are not confident in predicting the ground-truth label at every checkpoint (e.g. $[f_{e}(x)]_{y} = 1/2$ for all iteration $e$ in the case of binary classification). This would mean that the model is not capable of learning the given label, which happens for mislabeled points.

Reviewer comment (Evaluation):
What is the upper bound (when all data are labeled and when all data are labeled and correct / w/o mislabeling) in Fig. 4?

Author response:
- *"What is the upper bound (when all data are labeled and when all data are labeled and correct / w/o mislabeling) in Fig. 4?"*

We thank the reviewer for the question and we report in the following plot, available at https://i.imgur.com/gsLrb7r.png, the test accuracy when all the data are labeled in Section 5.2. Note that we cannot control the amount of mislabeling in this setting, as we only have access to the training labels in the datasets.


Training a model on all the labeled data often yields the best results, which is intuitive, as we consider the pseudo-labeling setting which uses only $10\%$ of labeled data. We notice that in some datasets (e.g. cutract, compas, drug), DIPS improves the fully supervised baseline, which may hint at greater inherent noise in the datasets. We can't quantify exactly the level of noise as we don't have the ground-truth, but the performance gap obtained by using DIPS could be seen as a proxy for it.

Reviewer comment (Reproducibility):
How many iterations are done for teacher-student baselines in Fig. 4?

Author response:
- *"How many iterations are done for teacher-student baselines in Fig. 4?"*
*"I see that aleatoric uncertainty is adopted per dataset."*

We used $T=5$ iterations for the experiment in Fig. 4. This follows the guideline of the sensitivity experiment conducted in [R1] (Appendix B.2 in the corresponding paper) and explains why we kept this parameter fixed for the datasets used in Section 5.2 in our manuscript. 
Regarding the aleatoric uncertainty threshold, it is chosen to be adaptive, and is not hand-crafted for each dataset. Furthermore, we stress that DIPS is applied on top of the baselines.

Reviewer comment (Writing):
This statement is incorrect "Note that s is solely used to select pseudo-labeled samples, among those which have not already been pseudo-labeled at a previous iteration." in both types of algorithms like greedy PL and UPS we relabel and reselect pseudo-labeled data.

Author response:
* On teacher-student:
For the teacher-student baselines, we do not repeat the training examples and do not assign _different_ pseudo-labels to them. After the selection, a pseudo-label is assigned to samples from the unlabeled set and added to the labeled set. Once the samples are in the labeled set, we do not assign them alternative labels, as they are discarded from the unlabeled set.


* On FixMatch:
You are right, since FixMatch is a continuous-training method, we do not have this cache mechanism, as the model updates the PL at every batch before doing the gradient step. Note though that to compute learning dynamics we need to freeze pseudo-labels at the beginning of each learning dynamics cycle.

_We hope these clarifications are useful! Thanks again for all the engagement._

Reviewer comment (Experiments):
How the parameters of the baselines in Appendix B1 are found? Why it is not adopted per dataset? I see that aleatoric uncertainty is adopted per dataset, which could be unfair parameters selection for the baselines.

Author response:
*"I see that aleatoric uncertainty is adopted per dataset."*

We used $T=5$ iterations for the experiment in Fig. 4. This follows the guideline of the sensitivity experiment conducted in [R1] (Appendix B.2 in the corresponding paper) and explains why we kept this parameter fixed for the datasets used in Section 5.2 in our manuscript. 
Regarding the aleatoric uncertainty threshold, it is chosen to be adaptive, and is not hand-crafted for each dataset. Furthermore, we stress that DIPS is applied on top of the baselines.

Reviewer comment (Reproducibility):
B 3.1, 3.2 what does it mean T=5 for greedy PL and for UPS? Could authors describe exactly how they do PL for both algo with 5 iterations? Does it mean that for UPS it is 5 teacher-student trainings and for greedy PL 5 teacher-student trainings with each student training based on the original prior work where we continuously train model?

Author response:
- *"How many iterations are done for teacher-student baselines in Fig. 4?"*

We used $T=5$ iterations for the experiment in Fig. 4. This follows the guideline of the sensitivity experiment conducted in [R1] (Appendix B.2 in the corresponding paper) and explains why we kept this parameter fixed for the datasets used in Section 5.2 in our manuscript.

Reviewer comment (Experiments):
What will happen if ablation in C2 is done only for labeled data selection and unlabeled data are selected based on PL prior works?

Author response:
### (D) Selection of unlabeled data

To answer the Reviewer's question regarding the selection of unlabeled data in DIPS, we refer to Figure 10 in our submitted manuscript (Appendix C.1). This figure shows results when our selection mechanism is only applied to the labeled data and we only use prior selection methods for the unlabeled data (this baseline is denoted as "A1"). The results show DIPS outperforms this baseline by a large margin. This justifies why we apply our selection method both to labeled and unlabeled data. We have updated our manuscript (Section 4.1) to more clearly highlight this result - thank you for the suggestion!