Paper: SEAL: Simultaneous Label Hierarchy Exploration And Learning (ICLR-2024)

Reviewer comment (Writing):
It is challenging to grasp the underlying meaning of 'latent labels' in the explored hierarchical structure. For instance, 'red' and 'green' appear to be related to both 'apple' and 'paint,' even though the author provided Example 1.

Author response:
Thank you for your feedback. The latent labels can be seen as "superclasses" or "higher order abstraction of classes". The latent labels can be seen as capturing the underlying relationships among classes. In our example, 'red' and 'green' indeed to be related to both 'apple' and 'paint,'.  **These two latent labels capture the relationship that apples (or paints) may have both colors green and red.**

Reviewer comment (Evaluation):
Following the previous question, it is important to note that in the current method, which is based on a hierarchical, the calibration of predictions at each level becomes crucial. However, the authors currently seem to neglect the calibration at latent levels.

Author response:
Thank you for your question. When you have a fixed hierarchy, you can calibrate the prediction at each level. But our adaptively updates the hierarchy to make the tree structure more suitable for the data distribution. **As the hierarchical level may change in our SEAL method, we define the total prediction error on the entire hierarchy to deal with this problem.**

Reviewer comment (Theory):
The theoretical analysis presented lacks valuable insights for the proposed approach. Furthermore, the author fails to clearly and concisely describe the integration of theory with their proposed approach, as mentioned in the Weaknesses 1.

Author response:
Thank you for your feedback. The theoretical analysis shows that our new loss can be interpreted as an optimal transport loss. As cross-entropy loss can be seen as a metric measures the similarity between probability distributions, **thus the theoretical analysis makes adding our new loss as a regularizer to cross-entropy loss in our SEAL method is more natural.**

Reviewer comment (Experiments):
The datasets used in the paper are very small and trivial, and the comparative methods employed are not up to date with the SOTA. The experiments are not convincing enough to demonstrate the effectiveness of the proposed approach.

Author response:
Thank you for your question. Firstly, we want to point out that the benchmarks we used are widely adopted in SSL literatures, so conduct experiments on these benchmarks may give a fair comparision. **Secondly, our method is easy to implement, fast, easy to plug in existing SOTA to further increase accuracy. So we think this demonstrates its effectiveness.** Lastly, as the reviewer questions "the comparative methods employed are not up to date with the SOTA". We conduct additional experiments on a recently proposed SOTA FreeMatch and results are summarized in the following Table (in error rate).

|  | CIFAR-10: 40 | CIFAR-10: 250 | CIFAR-100: 400 | CIFAR-100: 2500 |
| --- | --- | --- | --- | --- |
| FreeMatch | 4.9 | 4.88 | 37.98 | 26.47 |
| FreeMatch + SEAL | **4.8** | **4.71** | **37.12** | **26** |



FreeMatch: Self-adaptive Thresholding for Semi-supervised Learning, ICLR 2023

Reviewer comment (Experiments):
In Eq. (2), the dimension of A is $N$, so the dimension of the latent labels is $N-K$. Does this mean that we need to specify the number of categories of hidden labels in advance? How to set this hyper-parameter for different datasets, and do different values of this parameter affect the performance for one single dataset? Besides, how do we know the number of levels of label hierarchies? In other words, could the proposed method learn a multi-level class hierarchy? This is unclear in the manuscript.

Author response:
Thank you for your question. Yes, you need to specify the number of hidden labels in advance. This hyper-parameter should be set based on the specific dataset used and the prior knowledge. We will conduct ablation studies on the influence of this hyper-parameter and we empirically set this parameter to be close to the number of observed labels. The number of levels can also be seen as a hyper-parameter. As it is determined by $A_1$, it also should be set based on the specific dataset used and the prior knowledge. **Our method can indeed learn a multi-level class hierarchy, please see Figure 2.**

Reviewer comment (Writing):
In Theorem 2, is $\mathbf 1_N$ a N-dim vector whose elements are all 1? The authors should provide detailed notation for readers.

Author response:
Thank you for your question. Your understanding is correct. We have made this more clear in the revised manuscript.

Reviewer comment (Writing):
It’s hard to understand Definition 1 since there are so many unclear descriptions. For example, what’s the meaning of $\mu_s$ and $\mu_r$. As mentioned above, $\alpha_{sr}$ is a matrix, but in Eq. (3) it seems a scalar. Those unclear words make make it hard to follow this Section.

Author response:
Thank you for your question. $\mu$ a probability distribution on the total labels. We have added its definition more clearly in the revised version of the manuscript. $\mu_{s}$ and $\mu_{r}$ are the probability value of $\mu$ on coordinate $r$ and $s$. We have defined $\alpha=[\alpha_{sr}]_{s,r}$ in the revised manuscript. Thank you again for your help!

Reviewer comment (Reproducibility):
Is $w_s$ in Eq. (4) a hyper-parameter or a learnable parameters?

Author response:
Thank you for your question. In this paper, we set it as a fixed hyper-parameter, where all elements of $w$ is 1. **The reason is that we do not want to incorporate additional prior on the weight of edges.**

Reviewer comment (Writing):
In Section 3.3, does the Task (a) says that one can pre-define $\mathbf A_1$ rather than learn it by the proposed method? If so, how to perform it? The sentence “we choose A1 to be a trivial binary tree or trees derived from prior knowledge such as a part of a knowledge graph or a decision tree.” is quite unclear.

Author response:
Thank you for your question. Yes, you need to pre-define a $A_1$. If one does not have much prior knowledge, one may just initialize a random tree's adjacency matrix as $A_1$. If one has more prior knowledge, one can initialize one based on their prior knowledge. We have discussed the details of different tree structures in Appendix A.2.1. The ablation there includes a random initialized one and a tree constructed from prior knowledge.

Reviewer comment (Experiments):
The authors should compare the proposed method with other methods for learning label hierarchy in order to demonstrate its superiority.

Author response:
Thank you for your question. **As our method adaptively update the hierarchy and can start from random initialized hierarchy, comparing with methods with a good pre-defined hierarchy (which is used in supervised learning approaches, as we discussed in the related work) may be unfair.**  Additionally, we have not come across any other hierarchy exploitation method specifically designed for SSL. Instead, we conducted an ablation study by modifying our prior on tree structures in section A.2.1. It was observed that a more effective prior leads to improved performance.

Reviewer comment (Experiments):
Experiments do not compare with any competing SSL baselines that leverage label hierarchies. As the related work section notes, several hierarchical schemes have already been proposed, some of them specifically for SSL. But their relative performance w.r.t SEAL has not been compared or discussed. Note that the standalone gains due to SEAL alone are not impressive - only when combined with other orthogonal schemes does it work best. Finally, a good qualitative discussion of results is also lacking. Due to these reasons, the true utility of SEAL has not been demonstrated clearly.

Author response:
Thank you for your feedback. **Firstly, in our experiments,  for fair comparison we initialize the tree randomly, which does not incorporate prior knowledge and the works in related work part all used prior knowledge.** Secondly, our method improves the baseline's accuracy with minimal effort and the improvements are significant on STL-10 and CIFAR-10. And we think it is an advantage to consistently improve upon new techniques, which means that SEAL may be incorporated into every new SOTA methods. Lastly, the hierarchy-aware methods in related work are all based on supervised earning. We haven't found any hierarchical-aware method tackles SSL, but we may reproduce some as future work. **Lastly, as our method adaptively update the hierarchy and can start from random initialized hierarchy, comparing with methods with a good pre-defined hierarchy (which is used in supervised learning approaches, as we discussed in the related work) may be unfair.**

Reviewer comment (Experiments):
The set of baselines in Tables 2 and 3 are not the same and no explanation has been provided for this mismatch.

Author response:
Thank you for your inquiry. In our initial paper, we did not include SEAL (Debiased) as a comprehensive comparison due to the absence of CIFAR-100 and STL-10 experiments in the initial paper.

Reviewer comment (Reproducibility):
The optimization algorithm for minimizing the final SSL objective (that incorporates SEAL regularizer) has not been fleshed out in full detail.

Author response:
Thank you for your feedback. The update formula for SSL is consistent with the formula (16) used in supervised learning. The update rule for the hierarchy is given in Appendix A.6. We first conduct a SGD step, then perform an addition projection operation to project each column of $A_2$ onto the probability simplex.

Reviewer comment (Theory):
Theoretical results are not critical. They are more about the sanity of choices made and less about the generalization ability of SEAL technique etc.

Author response:
Thank you for your feedback. As we show in the paper, SEAL can be seen as Wasserstein distance on a posterior tree. **It can have generalization bound given in paper (https://arxiv.org/pdf/2306.04375v1.pdf).**

Reviewer comment (Experiments):
How does SEAL compare with the best of prior label hierarchy (and label correlation) exploiting approaches for SSL? Does (SEAL+Curriculum) outperform (best label hierarchy baseline for SSL+Curriculum)?

Author response:
Thank you for your question. We do not find any other hierarchy exploiting designed for SSL. **Instead, we do ablation study by changing our prior on tree structures in A.2.1. It is observed that better prior will indeed result in better performance.**

Reviewer comment (Experiments):
Why are the set of baselines in Tables 2 and 3 not identical? Please provide full set of results.

Author response:
Thank you for your question. We did not include SEAL (Debiased) as a full comparison because the original paper did not conduct experiments on CIFAR-100 and STL-10 datasets. **Furthermore, the main point is to demonstrate that SEAL can be effectively combined with advanced techniques to enhance performance further.**

Reviewer comment (Theory):
Give more details for (1) hierarchy that is assumed over latent variables, (2) optimization algorithm for minimizing eqn 16.

Author response:
Thank you for your question. (1) We have discussed the details of different tree structures in Appendix A.2.1. The random hierarchy we initialized in the experiment is a depth-4 tree with 21 nodes, where the latent nodes are organized as depicted in Figure 2 and the other nodes are connected following a random initialization.  (2) The update rule is given in Appendix A.6. We first conduct a SGD step, then perform an addition projection operation to project each column of $A_2$ onto the probability simplex.