Paper: Sparse Labels Node Classification: Unsupervised Learning for Mentoring Supervised Learning in Sparse Label Settings (ICLR-2024)

Reviewer comment (Writing):
The writing is not standardized, with too many formatting, grammatical, and presentation errors. For example, the tenses in the introduction part of the experiment are not uniform, and symbols are repeatedly used.

Author response:
We thank the reviewer for this concern. We sincerely apologize for this and will address all the formatting issues in the revision.

Reviewer comment (Theory):
The solutions to the given field problem are: 1) Capture the label distribution information in the graph, and 2) Pseudo-label the representative nodes. For the former, the paper does not give an explanation based on intuition or related theory but only introduces existing strategies. For the latter, this is a common method in semi-supervised scenarios. At the same time, the introduction to the selection of representative nodes is not clear.

Author response:
We apologize that in the current version the motivation for 1) ELI is not clear enough (as other reviewers pointed out as well).  ELI is proposed due to the points below:
- - (a) In the real world, there exist settings in which it is prohibitive to have more labeled nodes (consider the Facebook graph for example, or the YouTube graph), and in such scenarios, one will be in a sparse labeled node classification (SLNC) setting.
- - (b) We observe from the demo datasets we used that existing GNNs (given the baselines we used) cannot handle such a scenario. Since (1) in such a scenario, randomly selected nodes whether on a per-class basis or randomly over all classes may not be representative of the classes in the graph, and (b) The message-passing framework of GNNs may not be well-adapted given that the label distribution over the entire graph may not be well captured by the adjacency matrix of the graph.
- - (c) As such one will need to carefully select key nodes as well as estimate the label distribution of the graph to hope to perform well.
- - (d) Following this we propose ELI. We show in the appendix that the critical component in ELI is not the key labeled nodes but the label distribution over the graph (see the Ablation in figure 4 in section 5.7 that pink plots which have no Key labeled nodes (no KL) but have the KNN graph (KG) perform better than the yellow plots which have the KL but no KG). We will try to improve this in the paper revision during the coming week
- To select the key labeled nodes, we select $l_H$ nodes that have the smallest clustering loss (i.e., the nodes that capture best the pseudo label classes, we select 1 for each pseudo-class where a pseudo-class in our work is a cluster).

Reviewer comment (Experiments):
The paper proposes a semi-supervised learning scenario under sparse labels, but the experimental part is not clear about the setting of this scenario.

Author response:
We apologize if the experiment section is not clear enough, we will clarify this in the revision. From the x-axis of the plots in Figures 1, 3, 4, 6, and 7 one can see that the number of labeled nodes used for training increases gradually, and we plot this against the models' accuracy to show the effectiveness of ELI when only very few labeled nodes are used at training time. Moreover, we conduct ablation studies on ELI to prove that ELI as a whole (i.e., with the label distribution (KG) and with the key labels (KL)) performs best, while the model with the KG only performs second best, thus showing the importance of the label distribution in SLNC setting.

Reviewer comment (Novelty):
As far as I know, there are related works that consider graph learning under limited labels, such as CGPN [1], but this article does not mention and compare.

Author response:
We sincerely apologize that we were not aware of this work or others for the SLNC setting.  
- At first glance, our intuition on this work is that:
- - (a) Indeed it is a great work and we will include it in our main paper in citation and some comparisons.
- - (b) it is interesting to see that they too try to estimate a posteriori distribution of labels just like we did in our ELI framework, which we are happy to know as it comforts our idea and approach as well
- - (c) when nodes are selected randomly on a per-class basis, from the results in the paper both on Citeseer and Cora for 1 label per class to around 4 labels per class (which are shown in the paper), our simple SGC-ELI will perform better than it but we will post more accurate results here and in the paper after experiments (as this is a preliminary response). Our observation is that

Reviewer comment (Theory):
The paper proposes that the label distribution information in the graph needs to be captured to assist classification. What is the intuition behind this approach? A new filter is introduced in the approach, which can make the large eigenvalues in the Laplacian matrix smaller. Why do we do this? At the same time, how is the parameter k set?

Author response:
We apologize that in the current version the motivation for 1) ELI is not clear enough (as other reviewers pointed out as well). ELI is proposed due to the points below:
- -(a) In the real world, there exist settings in which it is prohibitive to have more labeled nodes (consider the Facebook graph for example, or the YouTube graph), and in such scenarios, one will be in a sparse labeled node classification (SLNC) setting.
- - (b) We observe from the demo datasets we used that existing GNNs (given the baselines we used) cannot handle such a scenario. Since (1) in such a scenario, randomly selected nodes whether on a per-class basis or randomly over all classes may not be representative of the classes in the graph, and (b) The message-passing framework of GNNs may not be well-adapted given that the label distribution over the entire graph may not be well captured by the adjacency matrix of the graph.
- - (c) As such one will need to carefully select key nodes as well as estimate the label distribution of the graph to hope to perform well.
- - (d) Following this we propose ELI. We show in the appendix that the critical component in ELI is not the key labeled nodes but the label distribution over the graph (see the Ablation in figure 4 in section 5.7 that pink plots which have no Key labeled nodes (no KL) but have the KNN graph (KG) perform better than the yellow plots which have the KL but no KG). We will try to improve this in the paper revision during the coming week
- To select the key-labeled nodes, we select $l_H$ nodes that have the smallest clustering loss (i.e., the nodes that capture best the pseudo-label classes, we select 1 for each pseudo-class where a pseudo-class in our work is a cluster).
- Once the Label distribution is obtained, we use it to build a new composite laplacian from three views (the original graph adjacency view, the label distribution adjacency view, and the view of the connectivity ground truth training labeled), we then use this new laplacian to build a new lowpass filter as has been proven to be useful for feature denoising\message propagation\label propagation in GNNs and use this lowpass filer as to aggregate our features before feeding them to a neural network or using a traditional heuristic such as label propagation.

Reviewer comment (Theory):
The paper proposes that pseudo-labels should be given to representative nodes. Where is the representativeness reflected here?

Author response:
In our ELI framework,  (1) we estimate the label distribution over the entire graph by estimating pseudo labels for each node (where in our current framework pseudo labels are cluster assignments), and then use this distribution to build the composite laplacian, and we use this label distribution to get representative node where the representative nodes are nodes that represent each pseudo-lass (cluster well), for this we selected the nod per cluster with the smallest distance from the centroid as explained in section 4.2

Reviewer comment (Presentation):
Why is the introduction of related work not divided into chapters? How relevant is it to your question?

Author response:
We apologize that we do not understand the concern well, from our understanding, the reviewer is asking what the main point of our related work section is and as such why is it organized the way it is? 
- - if yes, then the main point of our related work section is to introduce related works from different GNN domains, as well as discuss why they are not suited to the SLNC setting. We will read this section again since it seems the reviewer is suggesting that it may not be very clear.

Reviewer comment (Experiments):
The paper gives a set of experimental hyper-parameters in 5.5.1, such as learning rate, hidden layer size, etc. Does this mean that the settings are the same for each data set?

Author response:
Yes the hyperparameters of ELI-enhanced models and all models in section 5.5.1 are fixed for all datasets as we aimed to make ELI as hyperparameter-free as possible

Reviewer comment (Experiments):
It is recommended that the author compares the method with semi-supervised learning models and advanced methods. Meanwhile, the experimental results need to be clearly presented.

Author response:
We will definitely add at least two newer models in the updated version. 
- We will add tables for the shorter experiments from 1-4 labels per class (random and random-per-class) to the appendix by next Tuesday Eve or Wednesday Eve.

Reviewer comment (Experiments):
The proposed approach should be compared to more recent approaches.

Author response:
Q2) In the experiment, this paper compared the proposed approach to LP, SGC, DGI, GMI in which GMI is the most recent approach published in 2020. However, the compared approaches are not state-of-the-art. Since graph neural network is one of the popular research topics in machine learning, the proposed approach should be compared to more recent approaches. Could you compare the proposed approach to more recent approaches to show its effectiveness? The proposed approach should be compared to the approaches listed in Section 2.
- We will include two newer models in the updated paper by Tuesday (should the reviewer have a specific model in mind, please do suggest here). Due to space, we will add this to the appendix.

Reviewer comment (Experiments):
The parameter setting of the proposed approach is not justified in the paper.

Author response:
Q3) As described in Section 4.4, beta_1, beta_2, and beta_3 are set to 1/3 due to the simplicity. However, it is unclear why this parameter setting is recommended in the paper. The proposed approach can handle other parameter settings. Why is this parameter setting recommended in the paper?
- There is no reason for this suggestion besides it being an assumption that we consider all three information sources of equal value given that we were attempting to make the model less cumbersome (in terms of number of hyperparameters). In fact if one is confident enough of the Label Distribution Estimation model (i.e., the model from which the KNN graph is obtained), one may increase the weight for its normalized adjacency \hat{\mathbf{A}}_{\mathcal{G}_H}, as well as increase the weight for the normalized ground truth label adjacency  \hat{\mathbf{A}}_{\mathcal{G}_Y} as the number of training labels increase (since in this case the label adjacency indeed will correspond more and more to the ground truth distribution of labels). We will  include this to the appendix as well

Reviewer comment (Theory):
The theoretical properties of the proposed approach should be discussed in the paper.

Author response:
Q4) The paper should reveal the theoretical property of the proposed approach. As described in Section 4.3, it needs a high computation cost to compute Equation (4) directly. Therefore, the proposed approach approximately computes it using an iterative solution, as shown in Equation (5). I am interested in the theoretical difference between direct and approximate computations. Similarly, the proposed approach approximately computes the adjacency A_GH, as shown in Section 4.4. Please discuss the theoretical properties of the approximate computations.
- First, for the close-form solution in (4)  vs the approximation in (5): 
- - (a) the closed-form in (4) needs $O(n^3)$ for the matrix inverse and an additional $O(cn^2)$ where c is the number of classes, giving a total of $O(n^3 + cn^2)$ time complexity,
- - (b) assuming that \hat{\mathbf{A}}_{\mathbf{A}} has $N$ nonzero elements, the closed-form in (5) has a complexity of $O(tNc)$ where $t$ is the number of iterations and $c$ the number of classes ($t$ was set to $60 << n$) in the paper.
- Second computing \hat{\mathbf{A}}_{\mathcal{G}_H} from \mathbf{HH^{T}} has a complexity of $O(cn^2)$ if \mathbf{H} is dense using while using the KNN graph built from the soft cluster matrix \mathbf{U} as an approximation to \mathbf{HH^{T}} has a complexity of $O(nd + kn)$, where $k$ is the number of neighbors, and $d$ the number of node features, and in this case, the matrix \hat{\mathbf{A}}_{\mathcal{G}_H}, will be sparse since we can control the number of nodes to connect as shown in figure 2, and in addition since we are not using hard clusters for the graph, there will be some edges between close clusters. We will include these to the appendix as well

Reviewer comment (Experiments):
As described in Section 4.4, beta_1, beta_2, and beta_3 are set to 1/3 due to simplicity, but the reason why this parameter setting is recommended is unclear. While the proposed approach can handle other parameter settings, no justification is provided for this particular choice.

Author response:
Q3) As described in Section 4.4, beta_1, beta_2, and beta_3 are set to 1/3 due to the simplicity. However, it is unclear why this parameter setting is recommended in the paper. The proposed approach can handle other parameter settings. Why is this parameter setting recommended in the paper?
- There is no reason for this suggestion besides it being an assumption that we consider all three information sources of equal value given that we were attempting to make the model less cumbersome (in terms of number of hyperparameters). In fact if one is confident enough of the Label Distribution Estimation model (i.e., the model from which the KNN graph is obtained), one may increase the weight for its normalized adjacency \hat{\mathbf{A}}_{\mathcal{G}_H}, as well as increase the weight for the normalized ground truth label adjacency  \hat{\mathbf{A}}_{\mathcal{G}_Y} as the number of training labels increase (since in this case the label adjacency indeed will correspond more and more to the ground truth distribution of labels). We will  include this to the appendix as well

Reviewer comment (Evaluation):
The reviewer is interested in the theoretical computational cost and empirical processing time of the proposed approach, and whether the proposed approach is more efficient than the previous approaches.

Author response:
Q5) In addition, I am interested in the theoretical computational cost and empirical processing time of the proposed approach. The proposed approach is more efficient than the previous approaches?
- the total time complexity for LP-ELI is $O(cN + cnd + knd + dn^2 + tcnd)$, where $cN$ is for equation (5), $O(nd + kn)$ is for the KNN graph, $O(dn^2)$ is for the SVD, and $O(tcnd) = O(nd)$ is for kmeans since t and c are extemely small
- the total time complexity for SGC-ELI is  $O(Nd + ndc + cnd + knd + dn^2 + tcnd)$, where Nd if for the filtering operation $\hat{\mathbf{X}} = \mathcal{N}(\mathbf{L}_{A})\mathbf{X}$ defined in section~4.5, ndc for multipling the filtered features and the neurons of size d by c i.e, $\hat{\mathbf{X}}\mathbf{W}$,  and the rest being similar to those defined for the LP-ELI model. Giving a simplified complexity of $O(dn^2 + ndc +knd)$, with $c<<d<<N << n^2$ and $t$ being a constant. We will add this to the appendix as well

Reviewer comment (Novelty):
Overall, the proposed method is just a very heuristic combination of an existing unsupervised method for clustering attributed graphs and label propagation. Therefore, the novelty of this work is very limited. Moreover, I think that the performance of the proposed method will be heavily affected by the performance of the label estimation step. There is no principled guarantees for the proposed method to work well in general, especially when there are more labels than what the authors considered in the experiments. More on this in the next point.

Author response:
$Q1)$ Overall, the proposed method is just a very heuristic combination of an existing unsupervised method for clustering attributed graphs and label propagation. Therefore, the novelty of this work is very limited. Moreover, I think that the performance of the proposed method will be heavily affected by the performance of the label estimation step. There is no principled guarantees for the proposed method to work well in general, especially when there are more labels than what the authors considered in the experiments. More on this in the next point. 
- We thank the reviewer for the concern on the motivation of our work. We indeed will clarify this in the revision. As the reviewer seems concerned about our main contributions, we find it necessary to restate them here and elaborate on them:
- - (a) In the real world, there indeed exists extremely large graphs on which it is prohibitive to have more labeled nodes (consider the Facebook graph for example, or the YouTube graph), and in such scenarios, one will be in a sparse labeled node classification setting.
- - (b) We observe from the demo datasets we used that existing GNNs (given the baselines we used) cannot handle such a scenario. Since (1) in such a scenario, randomly selected nodes whether on a per-class basis or randomly over all classes may not be representative of the classes in the graph, and (b) The message-passing framework of GNNs may not be well-adapted given that the label distribution over the entire graph may not be well captured by the adjacency matrix of the graph. 
- - (c) As such one will need to carefully select key nodes as well as estimate the label distribution of the graph to hope to perform well.
- - (d) Following this we propose ELI. We show in the appendix that the critical component in ELI is not the key labeled nodes but the label distribution over the graph (see the Ablation in figure 4 in section 5.7 that pink plots which have no Key labeled nodes (no KL) but have the KNN graph (KG) perform better than the yellow plots which have the KL but no KG). We will try to move the ablation to the appendix for a clearer and more extensive discussion of this by Tuesday evening.

Reviewer comment (Experiments):
I am not really convinced by the empirical section that the sparse labels setting is important. (The sparse labels setting may be a relevant and important problem for many practical settings, but the empirical section fails to demonstrate that.) The datasets used in the experiments are standard benchmarks for node classification on graphs. In the standard train/test splits, these datasets already have a very low label rate, ranging from ~1.5% on Computers to ~5% on Cora and CiteSeer. Existing methods already perform very well at this label rate. The authors considered scenarios where there are only 3 to 60 labels in these datasets. This looks like a made-up setting to me. I would recommend the authors find alternative datasets where it is genuinely hard to obtain many ground-truth node labels, and thus we can only have access to a few dozen labels.

Author response:
$Q2)$ I am not really convinced by the empirical section that the sparse labels setting is important. (The sparse labels setting may be a relevant and important problem for many practical settings, but the empirical section fails to demonstrate that. ) The datasets used in the experiments are standard benchmarks for node classification on graphs. In the standard train/test splits, these datasets already have a very low label rate, ranging from ~1.5% on Computers to ~5% on Cora and CiteSeer. Existing methods already perform very well at this label rate. The authors considered scenarios where there are only 3 to 60 labels in these datasets. This looks like a made-up setting to me. I would recommend the authors find alternative datasets where it is genuinely hard to obtain many ground-truth node labels, and thus we can only have access to a few dozen labels.
- Indeed we agree with the reviewer that some GNNs perform well with label rates of ~1.5% on Computers to ~5% on Cora and CiteSeer. However, our point was to prove that when the label rate is even lower, i.e.,  the SLNC setting, most existing GNNs will not perform well as shown on the same datasets on which they performed well in the semi-supervised setting. However, as the reviewer wisely suggested for completeness we will some larger graphs to demonstrate this fact. Please if the reviewer can suggest one or two such graphs it will be well appreciated.

Reviewer comment (Experiments):
Based on the description given in Section 4.4, in your experiments, did you simply use $\mathbf{A}_{\mathcal{G}_H}$ as the adjacency matrix of the KNN graph, as opposed to $HH^T$? In that case, is the information in $H$ being used anywhere?

Author response:
Q4) Based on the description given in Section 4.4, in your experiments, did you simply use \hat{\mathbf{A}}{\mathcal{G}_H}as the adjacency matrix of the KNN graph, as opposed to \mathbf{HH^{T}}? In that case, is the information in  \mathbf{H} being used anywhere?
- Yes we simply used \hat{\mathbf{A}}{\mathcal{G}_H} rather \mathbf{HH^{T}}, as this will be sparse as we can control the number of nodes to connect  (figure 2).  Also, since we are not using hard clusters for the graph, there will be some edges between close clusters.