Paper: Confident Sinkhorn Allocation for Pseudo-Labeling (ICLR-2024)

Reviewer comment (Presentation):
Errors are present in the tables and figures. In Table 1, it is noted that in the comparison of related approaches, FlexMatch should be characterized as non-greedy based on the provided content. Regarding Figure 4, the top red square on the left fails to adequately illustrate the distinctions in assignments.

Author response:
Question: In Table 1, FlexMatch should be characterized as non-greedy 

Answer: Using OT, each datapoint will be jointly assigned labels in the presence of other assignments. In contrast, using FlexMatch (and other PL approaches), each data point assignment will purely depend on the given threshold and does not necessarily depend on the other assignments. FlexMatch still falls into this greedy category.
Nevertheless, in comparison to the other PL approaches, FlexMatch is less greedy in the sense that it defines the different thresholds for each class.

---

Question: Figure 4, the top red square on the left fails to adequately illustrate the distinctions in assignments.

Answer: We can improve this figure in the final version to make it clearer. In Figure 4, we use color (yellow is 1 and dark blue is 0) to indicate the numerical value of the assignment. The top red square actually has the value of 0.3 in brighter blue, although it is not visually clear enough to distinguish the dark blue (value =0), comparing to the other red squares in the bottom.

Reviewer comment (Reproducibility):
In Algorithm 1, Confident Sinkhorn Label Allocation (simplified), the derivation of b_{-} and b_{+} when setting marginal distributions is not explicitly stated. Based on the content, it is inferred that they are empirically estimated from the class label frequency in the training data or from prior knowledge. This should be explicitly mentioned in the algorithm; otherwise, it leads to unknowns and incompleteness in the algorithm.

Author response:
Question: In Algorithm 1, CSA (simplified), the derivation of $b_{-}$ and $b_{+}$ ...it is inferred that they are empirically estimated from the class label frequency in the training data or from prior knowledge.

Answer: Yes, they are empirically estimated from the class label frequency in the training data or from prior knowledge. We will mention them in the algorithm for clarity. Note that, $b_{-}$ and $b_{+}$ should be correctly written as $w_{-}$ and $w_{+}$.

Reviewer comment (Experiments):
In Section 2.2, you outlined two challenges associated with assigning pseudo-labels. The proposed resolution entails employing an ensemble learning framework along with Welch's T-test to discern and exclude less confident samples. Are there alternative, more dependable methods for comparing the most probable class with the second most probable class? (Except those compared in the appendix)

Author response:
Question: Are there alternative methods for statistical testing, more dependable methods for comparing the most probable class with the second most probable class?

Answer: Other alternatives for statistical testing include: Student-t test and Mann–Whitney U test.
However, according to the literature [1,2,3], we should use Welch’s t-test by default, instead of Student’s t-test, because Welch's t-test performs better whenever sample sizes and variances are unequal between groups, and gives the same result when sample sizes and variances are equal. 
In addition, Ruxton (2006) [3] stated “...the unequal variance t-test (or Welch’s t-test should always be used in preference to the Student’s t-test or Mann–Whitney U test”

[1] Delacre, M. et al (2017). Why Psychologists Should by Default Use Welch’s t-test Instead of Student’s t-test. International Review of Social Psychology

[2] https://daniellakens.blogspot.com/2015/01/always-use-welchs-t-test-instead-of.html 

[3] Ruxton, G. D. (2006). The unequal variance t-test is an underused alternative to Student's t-test and the Mann–Whitney U test. Behavioral Ecology.

We, therefore, opt for the proposed Welch’s t-test as the default choice in our framework. However, we can bring the above discussion in the paper to make it clearer for the readers. We thank the Reviewer for raising this useful question.

Reviewer comment (Experiments):
In the appendix, it is noted that the computational time of XGBoost increases with each iteration. Could this escalation in computational time become substantial when applied on a larger scale, potentially resulting in inefficiencies that outweigh its benefits?

Author response:
Question: the computational time of XGBoost increases with each iteration

Answer: First of all, most of the pseudo-labeling and SSL-based methods will suffer the same issue that the labeled dataset will grow over time after augmenting the unlabeled data. Such a growing dataset size will lead to more computational time.
One potential solution is to consider online learning to incrementally update the model for XGB, without retraining from the scratch [1]. Note that this is beyond the scope of our paper, but we can mention it in the updated version of the paper.

[1] Montiel, et al. "Adaptive xgboost for evolving data streams. IJCNN 2020

Reviewer comment (Experiments):
I've noted that in the algorithm when ρ equals 1, it indicates full allocation. Additionally, I observed a limited elucidation regarding this parameter. In the appendix, ρ is configured to allocate more data points in the earlier iterations and fewer in the later ones. I am intrigued by the potential impact of varying ρ. While this obviates the necessity to predefine a suitable threshold γ, it introduces a new variable, ρ, necessitating a predefined value. Is the outcome sensitive to the choice of ρ? Is there a universally applicable ρ that ensures the algorithm's effectiveness across diverse tasks? The role of ρ in the algorithm appears somewhat ambiguous, and I am particularly keen on gaining a deeper understanding of it.

Author response:
Question: $\rho$ obviates the necessity to predefine a suitable threshold $\gamma$. But it introduces a new variable, $\rho$, necessitating a predefined value. 

Answer: This is a very insightful comment. We follow the existing setting from SLA [44] to define the allocation schedule $\rho$ controlling the fraction of examples to be assigned labels in each iteration. The role of $\rho$ is to avoid assigning the entire 100% allocation in the first iteration for the OT approaches (both CSA and SLA). On the other hand, the threshold will stop this behavior from happening for other PL approaches.

A single assignment in PL can be sensitive to the value of $\gamma$ in being assigned or not. On the contrary with $\rho$ in OT, the assignment will be less sensitive to $\rho$ because we optimize the objective function to learn the assignment, i.e., minimizing the cost matrix to satisfy the row marginal distribution and column marginal distribution given the schedule allocation. There exist other factors (defined in the objective function) that influence the optimal assignment than the value of $\rho$ itself.

Reviewer comment (Novelty):
The paper's contribution appears somewhat vague. While it introduces a new pseudo labeling (PL) method, it lacks a clear probabilistic formulation. However, it's worth noting that the author does provide a PAC-Bayes generalization bound in Section 2.4, particularly for ensembling multiple classifiers. It would enhance clarity to explicitly state the individual contributions of various sections within the methodology.

Author response:
Question: It would enhance clarity to explicitly state the individual contributions of various sections within the methodology. The novelty of the optimal transport assignment in Section 2.3 appears somewhat limited. The primary concept seems to be derived from the original SLA [44]. 

Answer: The OT assignment plays an important role in our algorithm. However, we don't claim the novelty and contribution toward this OT part (we have clearly stated in the abstract, the last paragraph of the introduction, and Sec 2.3). 

We summarize the novelty and significance as follows:

First, we are the first (to the best of our knowledge) to theoretically study the role of uncertainty in PL.
The theoretical finding reveals that less uncertainty is more helpful. More unlabeled data is useful for a good estimation. Less number of classes and less number of input dimensions will make the estimation easier. The analysis takes a step further to show that both aleatoric uncertainty and epistemic uncertainty can reduce the probability of obtaining a good estimation.
We note that the result presented in SLA [44] has only analyzed the first property (more number of unlabeled data is useful for a good estimation). The rest of the theoretical finding is novel and original.

Second, we propose to use Welch’s T-test to ascertain whether the most likely class holds statistically greater significance than the second-most likely class. This T-test (to be used with OT)  eliminates the need to predefine the heuristic thresholds used in existing pseudo-labeling methods. 
Third, we extend PAC-Bayes bounds of ensemble models which provides a guarantee of generalization performance when trained on labeled data and tested on unlabeled data. Our bound strictly improves existing PAC-Bayes bounds for ensemble models and is of extended independent interest.

Note that  both of our theoretical results contribute to subsequent research in this domain. These are identified and  highlighted by Reviewer jWfa  and Reviewer 33FF.

Reviewer comment (Presentation):
There seems to be an inconsistency in the citation format used in the main text. The citation style in the introduction relies on numbers, but corresponding numbers in the reference list are missing. This inconsistency makes it challenging to match citations in the main text to the references.

Author response:
Question: There seems to be an inconsistency in the citation format used in the main text. The citation style in the introduction relies on numbers, but corresponding numbers in the reference list are missing. This inconsistency makes it challenging to match citations in the main text to the references.

Answer: We thank the reviewer for pointing out this citation format. In the current version, when clicking to the number citation in the main text using PDF, it will automatically link to the correct citation in the Reference. We will fix this formatting and update the paper.

Reviewer comment (Novelty):
The novelty of the optimal transport assignment in Section 2.3 appears somewhat limited. The primary concept seems to be derived from the original SLA [44]. Clarifying the extent of novelty and how the proposed method builds upon or diverges from previous work would be beneficial for the reader's understanding.

Author response:
Question: It would enhance clarity to explicitly state the individual contributions of various sections within the methodology. The novelty of the optimal transport assignment in Section 2.3 appears somewhat limited. The primary concept seems to be derived from the original SLA [44]. 

Answer: The OT assignment plays an important role in our algorithm. However, we don't claim the novelty and contribution toward this OT part (we have clearly stated in the abstract, the last paragraph of the introduction, and Sec 2.3). 

We summarize the novelty and significance as follows:

First, we are the first (to the best of our knowledge) to theoretically study the role of uncertainty in PL.
The theoretical finding reveals that less uncertainty is more helpful. More unlabeled data is useful for a good estimation. Less number of classes and less number of input dimensions will make the estimation easier. The analysis takes a step further to show that both aleatoric uncertainty and epistemic uncertainty can reduce the probability of obtaining a good estimation.
We note that the result presented in SLA [44] has only analyzed the first property (more number of unlabeled data is useful for a good estimation). The rest of the theoretical finding is novel and original.

Second, we propose to use Welch’s T-test to ascertain whether the most likely class holds statistically greater significance than the second-most likely class. This T-test (to be used with OT)  eliminates the need to predefine the heuristic thresholds used in existing pseudo-labeling methods. 
Third, we extend PAC-Bayes bounds of ensemble models which provides a guarantee of generalization performance when trained on labeled data and tested on unlabeled data. Our bound strictly improves existing PAC-Bayes bounds for ensemble models and is of extended independent interest.

Note that  both of our theoretical results contribute to subsequent research in this domain. These are identified and  highlighted by Reviewer jWfa  and Reviewer 33FF.

Reviewer comment (Experiments):
The method is primarily evaluated within the context of semi-supervised learning tasks, where sample selection is a general aspect of the approach. It raises the question of whether this technique can be extended to other settings, such as active learning or learning with noisy labels. Exploring the adaptability of this method to these different scenarios and discussing potential challenges or advantages would provide valuable insights into its broader applicability and limitations.

Author response:
Questions: The method is primarily evaluated within the context of semi-supervised learning tasks, where sample selection is a general aspect of the approach. It raises the question of whether this technique can be extended to other settings, such as active learning or learning with noisy labels. Exploring the adaptability of this method to these different scenarios and discussing potential challenges or advantages would provide valuable insights into its broader applicability and limitations.

Answer: we would like to thank the reviewer for the suggestion on the possible research extension from our proposed method, beyond semi-supervised learning tasks. Indeed, active learning or learning with noisy labels are the promising research directions that can leverage pseudo-labeling. Within the scope of our paper, we focus on the main task of SSL in which most of the pseudo-labeling methods have been evaluated. We will consider active learning and learning with noisy labels as the future work.

Reviewer comment (Novelty):
The choice to employ optimal transport methods for pseudo-labeling is not immediately clear, especially given the existence of the method detailed in section 2.2. It would be beneficial if the authors could elucidate on the rationale behind selecting optimal transport over the direct pseudo-labeling approach from section 2.2.

Author response:
Q: the rationale behind selecting OT over the direct PL approach (Sec 2.2)

A: The Reviewer mentions pseudo-labeling approach presented from Sec 2.2. However, Sec 2.2. presents Welch's T-test for identifying high confidence samples while the PL approaches are described in the Related Work subsection of the Introduction. Please correct us if we miss anything.

OT is the non-greedy algorithm which can learn the best (non-trivial) label assignments. On the other hand, most of the existing PL methods are greedy which will rely on the predefined threshold to assign the labels. Specifically, using OT, each data point will be jointly assigned labels in the presence of others. In contrast, using the PL, each data point assignment will purely depend on the given threshold and does not necessarily depend on the other assignments.

We show in Sec 3.3 and Fig. 4 that OT assignment cannot be achieved simply by changing the threshold value (also mentioned by the Reviewer 33FF).

Reviewer comment (Experiments):
The paper employs an ensemble of M models, but it is ambiguous whether the observed improvement in performance is attributed to the ensemble effect or the proposed algorithm itself. The absence of an ablation study leaves this point unclarified.

Author response:
Q: ablation study to discern the individual contributions of the ensemble effect and the proposed algorithm to the overall performance improvement?

A: Thanks for the great suggestion. We have made (below) the ablation study to learn the effect of ensembling. We show that the gain from the ensemble toward the final performance is not that significant to the overall performance improvement. In particular, ensembling XGBs will improve 2/5 datasets (Analcatdata, Digits) while degrade slightly for the other 3/5 datasets.


| Dataset | PL (single XGB) | PL (ensemble XGBs) | CSA |
| ------ | ------| ------| ------|
| Wdbc | 91.23±3 | 90.22±3 | 91.83±3 |
| Analcatdata  | 90.95±2 | 91.84±3 | 96.60±2 |
| German-credit | 70.72±3 | 70.15±2 | 71.47±3 |
| Breast cancer | 92.89±2 | 92.63±2 | 93.55±2 |
| Digits | 81.67±3 | 82.91±3 | 88.10±2 |

Reviewer comment (Theory):
The optimization objectives for optimal transport presented in section 2.3 lack clear explanations. It would be beneficial if the authors provide detailed interpretations for each constraint, elucidating on their roles and significance within the context of the problem.

Author response:
Q: The optimization objectives for OT in Sec 2.3 lack clear explanations

A: Given the space constraints, we have presented additional details of the OT derivation and optimization in Appendix A.2 and A.3.
There are three key components in the OT setting: the cost matrix, the marginal distributions for row and for column. We minimize the cost matrix to learn the optimal assignment, satisfying both marginal distributions. Specifically, the column marginal is a vector of one, i.e., each data point should not be assigned to more than one label, Eq. (6) and Eq. (11). In addition, each label should receive the amount of assignment ranging from lower bound and upper bound label frequency $[w_-, w_+]$, Eq. (7,8,12,13). We will highlight this interpretation in the final version.

Reviewer comment (Theory):
In section 2.1 of the article, there appears to be some inconsistency and potential oversight regarding notation. Firstly, the representation of the unlabeled data set as $\{\tilde{X_i^k}\}$ seems non-standard. Given that $X_i^k$ denotes an individual data point, it would be more appropriate to use lowercase notation for clarity. Secondly, the expression for the probabilistic classifiers, specifically $f_k(x_i)$, is stated to produce a scalar value indicating the likelihood of $x_i$ being labeled as $k$. However, the provided formulation $f_k(x_i):=\mathcal{N}(x_i|\hat\theta_k,\Lambda)$ suggests it's a function mapping to a normal distribution parameterized by $x_i\vert\hat\theta_k$ and $\Lambda$.

Author response:
Q: In Sec 2.1. the representation of the unlabeled data set as $\tilde{ X^k_i }$ seems non-standard. Given that $X^k_i$ denotes an individual data point, it would be more appropriate to use lowercase notation for clarity. 

A: we follow the same notation in [Yang and Xu (2020), Sec 2.1] to denote the labeled vs unlabeled data set. The difference is that the original paper considers the binary setting with $X^+$ and $X^-$ while our setting considers multi-classification with k classes. In addition, we (also by Yang and Xu) use $X^k_i$ to define a data point which takes a class k. $X^k_i$ is equivalent to write $x_i \in X^k$ and $\tilde{X}^k_i$ is $x_i \in \tilde{X^k}$.

---

Q: the expression for the probabilistic classifiers $f_k(x_i)$ stated to produce a scalar value indicating the LLK of $x_i$ being labeled as $k$. However, the provided formulation $f_k(x_i) = N( x_i | \hat{ \theta_k} , \Lambda)$ suggests it's a function mapping to a normal distribution parameterized by $x_i |  \hat{ \theta_k}$ and $\Lambda$.

A: $f_k(x_i)$ produces a scalar value indicating the LLK of $x_i$ being labeled as $k$. Then, we particularly consider the simple case of the Gaussian classifier, $f_k(x_i) := p(x_i | y_i = k) = N( x_i | \hat{ \theta_k} , \Lambda)$. This is the common notation used in [1].

[1] https://www.cs.ubc.ca/~murphyk/Teaching/CS340-Fall07/gaussClassif.pdf (see slide 7)

---

We are glad that we have resolved most of the concerns as acknowledged by the reviewer.

We thank the reviewer for mentioning the remaining concern left.

In our paper, we define $f_k$ produces a scalar value indicating the likelihood of being labelled as $k$. In particular, in Line 2, Page 4 in the paper, we have defined $f_k(x_i) := N( x_i | \hat{\theta}_k, \Lambda ) $  which is correct.

However, during our response, we have written $f_k(x_i) := \textcolor{red}{p(x_i | y_i = k)} = N( x_i | \hat{\theta}_k, \Lambda ) $ which is confusing as the reviewer pointed out. This should be corrected (by the reviewer) as $f_k(x_i) := \textcolor{blue}{p(y_i = k | x_i)} = N( x_i | \hat{\theta}_k, \Lambda ) $.

Nevertheless, we want to call out that this typo $ \textcolor{red}{p(x_i | y_i = k)}$ is from our rebuttal response while the equation presented in paper is still correct (Line 2, Page 4). 

Given (1) the above correctness and (2) that we have addressed all other concerns, we would greatly appreciate if you can reconsider the score or please let us know what is your suggestion to fix this typo?

Thank you!

Reviewer comment (Writing):
In Theorem 1, the definition of $\mu_{\backslash k}=\mu_j\vert\exist j\in {1,...K}\backslash k$ is ambiguous. It would be beneficial for the authors to provide a more explicit definition or clarification regarding the intended meaning of $\mu_{\backslash k}$ in the context of the theorem.

Author response:
Q: In Theorem 1, $\mu_{\\k} = \mu_j | \exists j \in {1…K} \ k$ is ambiguous

A: Intuitively,  k \ {1,...K}  indicates the correct class index. We denote the incorrect class being \k = \mu_j \in {1…K} \ k. This refers to one of the remaining classes excluding k. $\exists j$ can be optionally dropped to make it concise. We hope it clear and will clarify them in the context of the theorem

Reviewer comment (Experiments):
Why did the authors choose to use optimal transport methods for pseudo-labeling instead of the direct approach described in section 2.2?

Author response:
Q: the rationale behind selecting OT over the direct PL approach (Sec 2.2)

A: The Reviewer mentions pseudo-labeling approach presented from Sec 2.2. However, Sec 2.2. presents Welch's T-test for identifying high confidence samples while the PL approaches are described in the Related Work subsection of the Introduction. Please correct us if we miss anything.

OT is the non-greedy algorithm which can learn the best (non-trivial) label assignments. On the other hand, most of the existing PL methods are greedy which will rely on the predefined threshold to assign the labels. Specifically, using OT, each data point will be jointly assigned labels in the presence of others. In contrast, using the PL, each data point assignment will purely depend on the given threshold and does not necessarily depend on the other assignments.

We show in Sec 3.3 and Fig. 4 that OT assignment cannot be achieved simply by changing the threshold value (also mentioned by the Reviewer 33FF).

Reviewer comment (Experiments):
Have the authors considered conducting an ablation study to discern the individual contributions of the ensemble effect and the proposed algorithm to the overall performance improvement?

Author response:
Q: ablation study to discern the individual contributions of the ensemble effect and the proposed algorithm to the overall performance improvement?

A: Thanks for the great suggestion. We have made (below) the ablation study to learn the effect of ensembling. We show that the gain from the ensemble toward the final performance is not that significant to the overall performance improvement. In particular, ensembling XGBs will improve 2/5 datasets (Analcatdata, Digits) while degrade slightly for the other 3/5 datasets.


| Dataset | PL (single XGB) | PL (ensemble XGBs) | CSA |
| ------ | ------| ------| ------|
| Wdbc | 91.23±3 | 90.22±3 | 91.83±3 |
| Analcatdata  | 90.95±2 | 91.84±3 | 96.60±2 |
| German-credit | 70.72±3 | 70.15±2 | 71.47±3 |
| Breast cancer | 92.89±2 | 92.63±2 | 93.55±2 |
| Digits | 81.67±3 | 82.91±3 | 88.10±2 |