Paper: Label-free Node Classification on Graphs with Large Language Models (LLMs) (ICLR-2024)

Reviewer comment (Experiments):
Since LLMs generate annotations without access to ground truth labels, there is a risk of noisy annotations. It would be better to investigate the robustness of LLM-GNN to noisy annotations and potential strategies for mitigating their effects.

Author response:
Q1: Since LLMs generate annotations without access to ground truth labels, there is a risk of noisy annotations. It would be better to investigate the robustness of LLM-GNN to noisy annotations and potential strategies for mitigating their effects. 

**Response** Thanks for your great comments. We'll first summarize our findings and show details in the next paragraph. (1) We find that normal cross-entropy loss has a smaller overfitting on LLMs' annotations compared to random noisy labels, which is one advantage of using LLMs as the annotators; (2) We find that (graph) noisy label learning methods do not show clear improvements compared to normal cross-entropy loss in our setting. **Simple normal and weighted cross-entropy loss works better**. 

(1) To show the characteristics of LLMs' annotations, we train GNN models with ground truth labels, LLMs' annotations, and synthetic noisy labels with the same annotation quality (for example, suppose LLMs correctly label 80% of the points, we will randomly replace the labels of 20% of the points with incorrect labels based on the ground truth labels. ) We put the training curves in Appendix J and show the maximum test accuracies and final test accuracies in the following table. We observe that, unlike synthetic noisy labels, the gap between maximum accuracy and final accuracy for LLMs is very small. On one hand, it shows one good point of LLMs' annotations that we don't use a validation set to find the stopping point. On the other, it implies that it may be difficult to design a loss function to improve the performance. 

|                        	|      Cora      	|       Cora       	|    CiteSeer    	|     CiteSeer     	|
|----------------------|--------------|----------------|--------------|----------------|
|                        	| Final accuracy 	| Maximum accuracy 	| Final accuracy 	| Maximum accuracy 	|
|   Ground truth labels  	|      0.82      	|       0.825      	|       0.7      	|       0.73       	|
|   LLMs's annotations   	|      0.705     	|       0.71       	|      0.65      	|       0.68       	|
| Synthetic noisy labels 	|      0.58      	|       0.76       	|      0.51      	|       0.62       	|

(2) We further compare the differences among different loss functions. We compare the following four baselines: random selection and normal cross-entropy loss; random selection and NCERCE loss (design for noisy label learning) [1]; RIM with normal cross-entropy loss; random selection and weighted cross entropy loss; original RIM with weighted loss. It should be noted that RIM is an active learning method designed to process synthetic noisy labels. From the table, we find that loss designed for noisy label learning doesn't present an advantage over normal and weighted cross-entropy loss. One possible reason for this behavior is that the noisy pattern of LLMs' annotations is much more complex than synthetic noisy labels (see Appendix G.1), while noisy label learning methods are usually evaluated on synthetic noisy labels. Another reason is that the number of training samples is limited so it's hard to capture the pattern of noisy annotations. Compared to noisy label learning loss, we find that a simple weighted cross entropy loss ("WE" in the table) works well in most cases. 

|                	|  Cora 	| CiteSeer 	|
|--------------|-----|--------|
|    Random+CE   	| 70.48 	|   65.11  	|
| Random+NCE+RCE 	| 61.18 	|   67.75  	|
|     RIM+CE     	| 67.39 	|   64.37  	|
|    Random+WE	    | 71.85	    |   66.72   |
|       RIM      	| 68.28 	|   63.06  	|


[1] Ma X, Huang H, Wang Y, et al. Normalized loss functions for deep learning with noisy labels[C]//International conference on machine learning. PMLR, 2020: 6543-6553.

Reviewer comment (Experiments):
LLM-GNN's performance is demonstrated on a specific dataset (OGBN-PRODUCTS), and while it achieves impressive results, its generalizability to other datasets or domains is not thoroughly explored in the paper. The effectiveness of the approach in different scenarios and with various types of graphs should be investigated to assess its broader applicability.

Author response:
Q2: LLM-GNN's performance is demonstrated on a specific dataset (OGBN-PRODUCTS), and while it achieves impressive results, its generalizability to other datasets or domains is not thoroughly explored in the paper. The effectiveness of the approach in different scenarios and with various types of graphs should be investigated to assess its broader applicability.

**Response** Thanks for your great comments. We are not quite sure whether we exactly understand some details of your remarks. We will first retell the key points and answer them accordingly. 

We think you make the following key points: (1) Only the results of OGBN-Products are demonstrated. (2) Whether the proposed pipeline can be extended to more tasks like graph classification and more types of graphs beyond text-attributed graphs needs to be explored. 

For the first point, we admit we focus on the performance of OGBN-Products in the abstract and introduction part, which may lead to confusion. **We also test five other datasets in the experiment part.** For the second question, we think our pipelines have the potential to be applied to **more types of graphs and more types of tasks**. 


(1) Other than the performance on OGBN-PRODUCTS, we also show the performance of our methods on other datasets like Cora, CiteSeer, Pubmed, WikiCS, and Arxiv in the paper, and we will show them in the following table. Specifically, Cora, CiteSeer, and Arxiv are about papers from the computer science domain. Pubmed is about the papers from the medical domain. WikiCS is about the Wikipedia page. We have included multiple datasets from different domains to show the general applicability of our methods. Moreover, our pipeline shows consistent effectiveness across these datasets. In the following table, we demonstrate that our pipelines can get good performance with a much lower cost (both money and time) and scale to large graphs. 

| Performance        	| Cora  	| CiteSeer 	| PubMed 	| WikiCS 	| Arxiv 	| Products 	|
|--------------------|------|----------|--------|--------|-------|----------|
| LLMGNN             	| 75.54 	| 69.06    	| 81.95  	| 66.09  	| 66.14 	| 74.91    	|
| LLMs-as-Predictors 	| 67.33 	| 66.33    	| 87.33  	| 71     	| 73.67 	| 75.33    	|

| Costs              	| Cora  	| CiteSeer 	| PubMed 	| WikiCS 	| Arxiv 	| Products 	|
|--------------------|-------|----------|--------|--------|-------|----------|
| LLMGNN             	| 0.11  	| 0.1      	| 0.25   	| 0.16   	| 0.63  	| 0.74     	|
| LLMs-as-Predictors 	| 1.26  	| 1.5      	| 9.2    	| 5.46   	| 79    	| 1952     	|

(2) We admit that our paper focuses on the node classification tasks on the text-attributed graphs. However, we think our pipeline has the potential to be extended to more types of tasks and graphs. 

For different tasks like graph classification and knowledge graph QA, [1] does some exploration, and LLMs also show promising zero-shot capabilities. As a result, we may extend our pipelines to these tasks with some small modifications to the architectures. 

For more types of graphs, in a recent paper [2], the authors introduced a prompt-based method that aligns all types of node features through natural language, allowing large language models to naturally process problems in the textual domain with notable success. For discrete features like bag of words, [3] also propose a potential way to let large language models handle them by converting the original features into clustering centers. These methods may be integrated with our work and extend our methods to more types of graphs. We think you make two valuable points for future directions. 

[1] Guo J, Du L, Liu H. GPT4Graph: Can Large Language Models Understand Graph Structured Data? An Empirical Evaluation and Benchmarking[J]. arXiv preprint arXiv:2305.15066, 2023.

[2] Liu H, Feng J, Kong L, et al. One for All: Towards Training One Graph Model for All Classification Tasks[J]. arXiv preprint arXiv:2310.00149, 2023.

[3] Zhao J, Zhuo L, Shen Y, et al. Graphtext: Graph reasoning in text space[J]. arXiv preprint arXiv:2310.01089, 2023.

Reviewer comment (Evaluation):
There could be better with a detired comparison on the economic perspective.

Author response:
Q3: There could be better with a detailed comparison from the economic perspective.

**Response** Thanks for your great comments, which inspire us to further compare both the economy and performance of our pipelines. Compared to other baselines, our methods can **achieve promising performance with very low costs**.  


We use the following data to validate our claims. The first table demonstrates the performance of each pipeline. For LLMGNN and GNN, we deliberately let them produce similar performances to conduct a fair cost comparison. For Pubmed*, we demonstrate the performance of LLMGNN with a larger budget since the performance of LLMs-as-Predictors is superior on this dataset. We can improve the performance of LLMGNN by slightly increasing the costs.
|     (Accuracy)     	|  Cora 	| CiteSeer 	| Pubmed 	| Pubmed* 	| WikiCS 	| Arxiv 	| Products 	|
|------------------|-----|--------|------|-------|------|-----|--------|
|       LLMGNN       	| 75.54 	|   69.06  	|  74.98 	|  81.95  	|  66.09 	| 66.14 	|   74.9   	|
| LLMs-as-Predictors 	| 68.33 	|   66.33  	|  87.33 	|  87.33  	|   71   	| 73.33 	|   75.33  	|
|         GNN        	| 75.54 	|   69.06  	|  74.98 	|  81.95  	|  66.09 	| 66.14 	|   74.9   	|

The second table compares the number of training samples needed to achieve the performance in the first table. Here, PL stands for pseudo labels generated by LLMs. GT stands for ground truth labels.
| (Number of   annotations) 	| Cora 	| CiteSeer 	| Pubmed 	| Pubmed* 	| WikiCS 	|  Arxiv 	| Products 	|
|-------------------------|----|--------|------|-------|------|------|--------|
|         LLMGNN (PL)        	|  140 	|    120   	|   60   	|   300   	|   200  	|   800  	|    940   	|
|   LLMs-as-Predictors (PL)  	| 2708 	|   3186   	|  19717 	|  19717  	|  11701 	| 169343 	|  2449029 	|
|          GNN (GT)         	|  50  	|    50    	|   42   	|   210   	|   40   	|   560  	|    400   	|


From these two tables, we can see that **LLMGNN significantly reduces the costs compared to LLMs-as-Predictors**. Comparing GNN with LLMGNN, the costs of ground truth labels are hard to estimate. From a recent study [1], it shows that the annotations generated by ChatGPT are more than 20 times cheaper than human annotators. If we use this ratio, we can see that although LLMGNN requires more pseudo labels, the overall costs are still much cheaper than traditional GNN-based pipelines. 

[1] Gilardi F, Alizadeh M, Kubli M. Chatgpt outperforms crowd-workers for text-annotation tasks[J]. arXiv preprint arXiv:2303.15056, 2023.

Reviewer comment (Novelty):
Though I acknowledge that the proposed method is a valid solution, the technical contribution of the paper is somewhat limited, especially considering that the three major components are largely based on heuristic observations, and the rest are based on existing LLMs and GNNs. It would make the paper stronger if some theoretical analyses could be provided for the proposed components.

Author response:
**Response** Thanks for your great comments. We first summarize our responses and then provide more details. (1) We agree that theoretical analysis is important to support the effectiveness of our proposed methods. We use theoretical analysis to show why difficulty-aware selection can get high-quality annotations. We also add the following part into Appendix L. (2) We would like to strengthen our contributions: 1. We propose a flexible pipeline LLMGNN which demonstrates effectiveness with low costs; 2. We propose three effective strategies: difficulty-aware selection, confidence-aware annotations, and post-filtering to further enhance the effectiveness of LLMGNN.

Q1 (1) One important heuristic used in our pipeline is difficulty-aware selection. Here, we analyze why it can enhance the annotation quality of LLMs. 



Given the parameter distribution of LLMs $\mathcal{Q}$, the parameter distribution of encoder $\mathcal{P}$ (SBERT), we assume ground truth $Y\_L\in \mathcal{R}^{N \times M}$, pseudo label $Y \in \mathcal{R}^{N \times M}$,  and node features encoded by the encoder $X\in \mathcal{R}^{N \times d}$. Here, $M$ denotes the number of classes and $d$ denotes the hidden dimension. Our objective is to maximize the accuracy of annotations, which is thus to minimize the discrepancies between $Y$ and $Y\_{L}$. 

$$
\\min\_{(n\_{1},...,n\_{k})}{\\mathbb{E}\_{\\theta\\sim Q}{f(Y)}} = \\ell (Y,Y\_{L})=\\sum\_{i=1}^{k} \\ell({y}^{n\_{i}} , y\_{L}^{n\_{i}})
$$

where $(n\_{1},...,n\_{k})$ is the index of the $k$ selected nodes, and $f(Y)$ is defined as follows, where ${x}^{n\_{i}}$ represents the feature of node ${n\_{i}}$, while $x\_{L}^{n\_{i}}$ represents the unknown latent embedding of node ${n\_{i}}$. 
$$
f(Y)=\\ell(Y,Y\_{L})=\\sum\_{i=1}^{k} \\ell({y}^{n\_{i}} , y\_{L}^{n\_{i}})= \\ln(1+\\sum\_{i=1}^{k}\\|{x}^{n\_{i}} - x\_{L}^{n\_{i}}\\|^{2})= \\ln(1+\\|X-X\_{L}\\|^{2})
$$


Since we only have access to the node features $X$ generated by encoder $\theta$, we need to make a connection between $\mathcal{Q}$ and $\mathcal{P}$. 

Lemma 1. For any annotation $y$ generated by LLMs, $\mathbb{E}\_{\theta\sim \mathcal{Q}}f(y)\leq \log \mathbb{E}\_{\theta\sim \mathcal{P}} \exp(f(y))+KL( \mathcal{Q} \| \mathcal{P})$

Proof:
$$
\\begin{aligned}
&\\\\
&{\\mathbb{E}\_{\\theta^{\\prime}\\sim \\mathcal{P}}{f(y)}}=\\int f(y) p(y) d y=\\int f(y) \\frac{p(y)}{q(y)} q(y) d y=\\mathbb{E}\_{\\theta\\sim \\mathcal{Q}} f(y) \\frac{p(y)}{q(y)}\\\\
&\\text{Since}\\quad KL(\\mathcal{Q} \\| \\mathcal{P})=\\mathbb{E}\_{\\theta\\sim Q} \\log \\frac{q(y)}{p(y)}\\\\
&\\log \\mathbb{E}\_{\\theta^{\\prime}\\sim \\mathcal{P}} f(y)= \\log\\mathbb{E}\_{\\theta\\sim \\mathcal{Q}} f(y) \\frac{p(y)}{q(y)} \\geq\\mathbb{E}\_{\\theta\\sim \\mathcal{Q}}\\log f(y) \\frac{p(y)}{q(y)}=\\mathbb{E}\_{\\theta\\sim \\mathcal{Q}}\\log f(y)-KL( \\mathcal{Q} \\| \\mathcal{P}) \\\\
\\end{aligned}
$$

Then, we transform the objective into 
$$
\\min\_{(n\_{1},...,n\_{k})} \\mathbb{E}\_{\\theta^{\\prime}\\sim \\mathcal{P}} \\exp(f(Y)) 
$$

Assuming the label distribution $\\mathcal{H}$, we further have
$$
\\min\_{(n\_{1},...,n\_{k})} \\mathbb{E}\_{Y\\sim \\mathcal{H}} \\mathbb{E}\_{\\theta^{\\prime}\\sim \\mathcal{P}} \\exp(f(Y)) = \\min\_{(n\_{1},...,n\_{k})} \\mathbb{E}\_{Y\\sim \\mathcal{H}} \\mathbb{E}\_{\\theta\\sim \\mathcal{P}}\\|X-X\_{L}\\|^{2} \\\\
$$

Assuming $X$ and $X\_{L}$ follows gaussian distribution, where $X\sim N(\mu\_{i},\sigma\_{i});X\_L\sim N(\mu\_{j},\sigma\_{j})$, then

$$
\\begin{aligned}
\\mathrm{E}\\left(\\|X-X\_{L}\\|^{2}\\right) & =\\mathrm{E}\\left(\\left\\|\\left(X-\\mu\_{i}\\right)-\\left(X\_{L}-\\mu\_{j}\\right)+\\left(\\mu\_{i}-\\mu\_{j}\\right)\\right\\|^{2}\\right) \\\\
& =\\mathrm{E}\\left(\\left\\|X-\\mu\_{i}\\right\\|^{2}\\right)+\\mathrm{E}\\left(\\left\\|X\_{L}-\\mu\_{j}\\right\\|^{2}\\right)+\\left\\|\\mu\_{i}-\\mu\_{j}\\right\\|^{2} \\\\
& =n \\sigma\_{i}^{2}+n \\sigma\_{j}^{2}+\\left\\|\\mu\_{i}-\\mu\_{j}\\right\\|^{2}
\\end{aligned}
$$

Since $(\mu\_{j},\sigma\_{j})$ is unknown, $\mu\_{i}$ is fixed, thus we want to minimize $\sigma\_{i}$.  Given an arbitrary node, $\sigma\_{i}$ can be viewed as the distance to the clustering centers. The smaller $\sigma\_{i}$ is, the smaller the corresponding $\mathrm{E}\left(\|X-X\_{L}\|^{2}\right)$ also becomes, indicating that the minimum value of $\mathbb{E}\_{\theta\sim \mathcal{Q}}f(y)$ is attained when $\sigma\_{i}$ is at its smallest. This demonstrates why nodes closer to the clustering centers are "preferred" by LLMs and thus achieve better annotation quality.

Reviewer comment (Writing):
The authors should more explicitly mention that their proposed method only works for text-attributed graphs rather than any general graph, e.g., in the abstract and introduction. Otherwise, the paper may have overclaiming issues.

Author response:
**Response** Thanks for your great comments. We agree that our pipelines focus more on text-attributed graphs, while our methods have the potential to be applied to more types of graphs. We add discussions on this problem in the revision. 

Currently, for the task of node classification, most datasets feature text-type nodes, and thus our method is well-suited for the majority of datasets and tasks. Furthermore, our approach has the potential to be extended to more graphs with different types of node features. In a recent paper [1], the authors introduced a prompt-based method that aligns all types of node features through natural language, allowing large language models to naturally process problems in the textual domain with notable success. For discrete features like bag of words, [2] also propose a potential way to let large language models handle them by converting the original features into clustering centers. These methods may be integrated with our work and extend our methods to more types of graphs. Your point is incredibly insightful and valuable, and we consider it an excellent extension of our work. In our revised version, we have included a discussion about this issue in both the abstract, introduction, and preliminaries.

[1] Liu H, Feng J, Kong L, et al. One for All: Towards Training One Graph Model for All Classification Tasks[J]. arXiv preprint arXiv:2310.00149, 2023.

[2] Zhao J, Zhuo L, Shen Y, et al. Graphtext: Graph reasoning in text space[J]. arXiv preprint arXiv:2310.01089, 2023.

Reviewer comment (Experiments):
In generating the initial node labels using LLMs, it seems that only the feature information is utilized and no structure is considered. Since it is well-known in the graph machine learning literature that both features and structures greatly affect the node labels, there exists a large room for improvement.

Author response:
**Response** Thanks for your great comments, which inspire us to have a further look at structure-aware prompts. We find that incorporating structural information in the annotation prompt **can not largely improve performance** with the current prompt design. 


We demonstrate both the accuracy of LLMs' annotations (the ratio of LLMs' annotations matching ground truth labels) and the accuracy of LLM-GNN in the following table. "No struct" means the "hybrid" (combining both "TopK" and "Most Voting") prompt strategy used in our papers. "Struct" is the new version that combines the "neighbor summarization" strategy in [1] and the "hybrid" prompt strategy.

|       (Annotation)            	|         Cora    	|       Cora   	|      CiteSeer 	|     CiteSeer    	|
|-----------------|---------------|------------|-------------|---------------|
|                   	|      No Struct  	|      Struct  	|     No Struct 	|      Struct     	|
|        FeatProp   	|   70.71 ± 0.00  	| 64.52 ± 0.34 	|  66.39 ± 2.83 	|  67.50  ± 0.65  	|
|     FeatProp + PS 	|    73.81 ± 1.52 	| 63.69 ± 1.11 	|  78.12 ± 1.07 	|   78.12 ± 1.45  	|

|         (LLMGNN)          	|         Cora    	|        Cora   	|      CiteSeer 	|     CiteSeer    	|
|-----------------|---------------|-------------|-------------|---------------|
|                   	|      No Struct  	|       Struct  	|     No Struct 	|      Struct     	|
|        FeatProp   	|    72.82 ± 0.08 	|  69.08 ± 0.39 	| 66.61 ± 0.55  	|   65.92 ± 0.43  	|
|     FeatProp + PS 	|    75.54 ± 0.34 	|  67.55 ± 0.70 	|  69.06 ± 0.32 	|   67.80 ± 0.45  	|

We observe that:
1. Incorporating structural information in the annotation prompts can not improve the performance, and may affect the effectiveness of post-selection. For example, the annotation quality even drops after adding structural information. 
2. Incorporating structural information will increase the costs of prompts since the length of prompts becomes longer. 

Based on these points, we think that using text-only annotation prompts for LLMs is appropriate in the current stage. However, we also admit that how incorporating structural information in LLMs is a valuable future direction.

[1] Chen Z, Mao H, Li H, et al. Exploring the potential of large language models (llms) in learning on graphs[J]. arXiv preprint arXiv:2307.03393, 2023.

Reviewer comment (Novelty):
There are some missing related works regarding zero-shot node classification such as [1-2], which should be added.

Author response:
**Response** Thanks for your great comments which inspire us to further explore the effectiveness of our paper under the **zero-shot node classification** setting. We first summarize our responses and explain the details. (1) We've cited those related works and add a discussion section. However, **there are some differences between the setting** of label-free node classification and zero-shot node classification; (2) For zero-shot node classification, our methods can still achieve promising performance. 


(1) The setting for label-free node classification involves training a GNN using annotations generated by LLMs, without any pre-existing labels. In contrast, the zero-shot node classification setting in these two papers involves training a model with training data for training classes (for example, A, B, C), which can then be generalized to new classes in the testing phase (for example, D, E, F). In practice, our label-free methods can be used to solve zero-shot node classification by directly annotating the new classes. However, those zero-shot node classification methods can not be directly applied to label-free node classification. 

(2) In the following table, we compare the performance of LLMGNN to zero-shot node classification baselines DGPN [1], DBiGCN [2], and GraphCEN [3] in zero-shot node classification settings. For LLMGNN, we use the featprop + Post Selection with 20% pruning ratio and 140 annotation budget. We take the experimental settings from [3]. 

|          	| Cora  	| CiteSeer 	|
|----------|-------|----------|
| DGPN     	| 33.76 	| 37.74    	|
| DBiGCN   	| 45.08 	| 38.57    	|
| GraphCEN 	| 48.43 	| 40.77    	|
| LLMGNN   	| 79.45 	| 66.67    	|

We can see that our methods can consistently beat all those baselines under the zero-shot node classification setting. We appreciate your comments which inspire us to demonstrate the strong capability of our methods in traditional zero-shot node classification problems. 

[1] Wang Z, Wang J, Guo Y, et al. Zero-shot node classification with decomposed graph prototype network[C]//Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 2021: 1769-1779.

[2] Yue Q, Liang J, Cui J, et al. Dual Bidirectional Graph Convolutional Networks for Zero-shot Node Classification[C]//Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2022: 2408-2417.

[3] Ju W, Qin Y, Yi S, et al. Zero-shot Node Classification with Graph Contrastive Embedding Network[J]. Transactions on Machine Learning Research, 2023.

Reviewer comment (Experiments):
I also wonder how different LLMs affect the model (the reported results are all based on GPT-3.5-turbo).

Author response:
**Response** Thanks for your great comments. We have added experiments using other large language models like GPT-4. Surprisingly, **we find that GPT4 brings no clear improvement compared to the results of GPT-3.5-turbo.**

We show the results in the following table. We can see that GPT4 only achieves clear improvements in **DA-GraphPart** on Cora. We appreciate your questions to help us have an exploration for this interesting phenomenon. 


|   (LLMGNN)   	|     Cora     	|     Cora     	|    CiteSeer    	|   CiteSeer   	|
|------------|------------|------------|--------------|------------|
|              	|    GPT3.5    	|     GPT4     	|     GPT3.5     	|     GPT4     	|
|    Random    	| 70.48 ± 0.73 	| 70.03 ± 1.42 	| 65.11   ± 1.12 	| 67.00 ± 1.21 	|
|   GraphPart  	| 69.54 ± 2.18 	| 70.65 ± 0.98 	|  66.59 ± 1.34  	| 65.66 ± 0.74 	|
| DA-GraphPart 	| 69.64 ± 0.30 	| 75.06 ± 0.08 	|  69.88 ± 1.81  	| 69.92 ± 1.10 	|
|   FeatProp   	| 72.82 ± 0.08 	| 69.46 ± 0.69 	|  66.61 ± 0.55  	| 66.36 ± 0.77 	|
|  PS-FeatProp 	| 75.54 ± 0.34 	| 71.76 ± 0.33 	|  69.06 ± 0.32  	| 69.15 ± 1.06 	|

One possible reason for this phenomenon is that the primary area where GPT-4 surpasses GPT-3.5 is in complex reasoning capabilities. However, in tasks such as text classification, the difference between the two is not very significant. This phenomenon has also been observed in text classification tasks in other fields, such as financial text classification [1]. 

[1] Loukas L, Stogiannidis I, Malakasiotis P, et al. Breaking the bank with chatgpt: Few-shot text classification for finance[J]. arXiv preprint arXiv:2308.14634, 2023.

Reviewer comment (Evaluation):
Difficulty aware (DA) selection: According to the paper, LLMs annotation quality degrades when they have to annotate nodes which are away from the centers. It implies that the LLMs annotation quality would suffer in case of the diverse nodes (away from center). However, GNNs accuracy will only improve if the nodes are diverse. Hence, difficulty aware selection i.e. use of c-density might not always help and, in fact, it may hinder in some cases. This is also evident from the results shown in Table 2 : Active_Selection_Methods and the corresponding DA-Active_Selection_Methods show similar performance on average across different techniques (i.e., selection methods). Moreover, for at least 50% of the cases, the DA-method (row 2) performs poorly compared to the corresponding active learning method (row 1).

Author response:
**Response** Thanks for your great comments. We want to first summarize our responses and then explain them in details. Applying the original DA may lead to clear performance degradation, which is due to the class imbalance problem by a small clustering center number $K$ on datasets like Pubmed. **This problem can be solved** by increasing $K$ to balance the diversity and annotation quality. 

The reason for DA's bad performance on Pubmed is when performing K-means clustering with $C-Density$, the number of clustering centers $K$ is fixed as the number of classes. This approach can achieve good labeling quality in most cases. On datasets like Pubmed where the number of classes is small (only 3), the points selected by $C-Density$ tend to cause class imbalance (the annotation quality is still very good). To solve this problem, we find that by setting a larger value for the number of clustering centers $K$, the new $ C-Density^{\prime}$ obtained can effectively solve this problem. We show the results in the following table. $k$ means the number of clustering centers to calculate $C-Density^{\prime}$, $C$ means the number of classes, and budgets means the number of selected annotated nodes. We use AGE as a demonstration to show the idea. From the results, we can see that setting a larger $K$ can improve the performance of DA on Pubmed.

|                  	| 	 Pubmed 	|
|----------------|------|
|        AGE       	|  74.55 	|
|    DA-AGE(k=C)   	|   55.36 	|
|   DA-AGE(k=2C)   	|   57.38 	|
| DA-AGE(k=budget) 	|  75.71 	|


In the following table, we demonstrate that DA can improve the performance of diversity-aware selection methods most of the time. 

|              	|     Cora     	|   CiteSeer   	|    Pubmed    	|    WikiCS    	|
|------------|------------|------------|------------|------------|
|      AGE	     	| 69.15 ± 0.38 	| 54.25 ± 0.31 	| 74.55 ± 0.54 	| 55.51 ± 0.12 	|
|    DA-AGE    	| 74.38 ± 0.24 	| 59.92 ± 0.42 	| 74.20 ± 0.51 	| 59.39 ± 0.21 	|
|      RIM     	| 69.86 ± 0.38 	| 63.44 ± 0.42 	| 76.22 ± 0.16 	|  66.72± 0.16 	|
|    DA-RIM    	| 73.99 ± 0.44 	| 60.33 ± 0.40 	| 79.17 ± 0.11 	|  67.82± 0.32 	|
|   GraphPart  	| 68.57 ± 2.18 	| 66.59 ± 1.34 	| 77.50 ± 1.23 	| 67.28 ± 0.87 	|
| DA-GraphPart 	| 69.35 ± 1.92 	| 69.37 ± 1.27 	| 79.49 ± 0.85 	| 68.72 ± 1.01 	|

We can see for 10/12 cases, DA significantly improves the performance. In other cases, it also achieves comparable performance.

Reviewer comment (Evaluation):
Though the accuracy from the LLM-GNN model is good (and of course, the model is efficient), it couldn't outperform LLM as a predictor (Table 3).

Author response:
**Response** Thanks for your great comments. Despite the superior performance of LLMs-as-Predictors in some datasets, it's not practical to be deployed in real-world scenarios because of **efficiency**. Our methods can **scale to large-scale graphs with promising performance**. 

 Although the original LLMs-as-predictors can achieve good performance, it's not practical to be applied to even medium-scale datasets like Arxiv. In the original paper [1], it's only applied to a small subset of the datasets to evaluate the performance. Taking Arxiv as an example, considering OpenAI's current limit of 60 accesses per minute, completing all predictions on Arxiv would take 44 hours, whereas LLMGNN only requires a few minutes. Our methods extend the original pipeline to real-world large-scale datasets. 

Then, we compare the performance and the economic costs of these two pipelines to further demonstrate the effectiveness of our methods in the following two tables. The cost is estimated in units of dollars. On datasets like Citeseer, and Products, the gap between LLMGNN and LLMs-as-predictors is very small. On Cora, LLMGNN can even outperform LLMs-as-predictors. Only on PubMed, there's a relatively large gap between LLM-GNN and PubMed. It's because of LLMs' superior performance on this dataset, which is related to the shortcut prediction in attribute [1]. When LLMs' performance is superior, we can also add more budgets to further improve the performance of LLMGNN. ***For example, if we raise the budget in PubMed from 0.05 to 0.25, we can raise the accuracy of LLMGNN from 74.98 to 82.*** 



| Performance        	| Cora  	| CiteSeer 	| PubMed 	| WikiCS 	| Arxiv 	| Products 	|
|--------------------|-------|----------|--------|--------|-------|----------|
| LLMGNN             	| 75.54 	| 69.06    	| 74.98  	| 66.09  	| 66.14 	| 74.91    	|
| LLMs-as-Predictors 	| 67.33 	| 66.33    	| 87.33  	| 71     	| 73.67 	| 75.33    	|

| Costs              	| Cora  	| CiteSeer 	| PubMed 	| WikiCS 	| Arxiv 	| Products 	|
|--------------------|------|----------|--------|--------|-------|----------|
| LLMGNN             	| 0.11  	| 0.1      	| 0.05   	| 0.16   	| 0.63  	| 0.74     	|
| LLMs-as-Predictors 	| 1.26  	| 1.5      	| 9.2    	| 5.46   	| 79    	| 1952     	|

In summary, the economic costs and efficiency of our methods are consistently much lower than LLMs-as-predictors. It's a valuable future direction to study how to further improve the performance of LLMGNN. 


[1] Chen Z, Mao H, Li H, et al. Exploring the potential of large language models (llms) in learning on graphs[J]. arXiv preprint arXiv:2307.03393, 2023.

Reviewer comment (Theory):
The methods are heuristics and do not have theoretical evidence.

Author response:
**Response** Thanks for your great comments. We agree that theoretical analysis is important to support the effectiveness of our proposed methods. We show why Difficulty-aware selection can improve the annotation quality of LLMs. We also add the following part into Appendix L.

Given the parameter distribution of LLMs $\mathcal{Q}$, the parameter distribution of encoder $\mathcal{P}$ (SBERT), we assume ground truth $Y_L\in \mathcal{R}^{N \times M}$, pseudo label $Y \in \mathcal{R}^{N \times M}$,  and node features encoded by the encoder $X\in \mathcal{R}^{N \times d}$. Here, $M$ denotes the number of classes and $d$ denotes the hidden dimension. Our objective is to maximize the accuracy of annotations, which is thus to minimize the discrepancies between $Y$ and $Y_{L}$. 

$$
\\min_{(n_{1},...,n_{k})} {\mathbb{E}_{\theta\sim Q}{f(Y)}} = \\ell(Y,Y\_{L})=\sum\_{i=1}^{k} \\ell({y}^{n\_{i}} , y\_{L}^{n\_{i}})
$$

where $(n_{1},...,n_{k})$ is the index of the $k$ selected nodes, and $f(Y)$ is defined as follows, where ${x}^{n_{i}}$ represents the feature of node ${n_{i}}$, while $x_{L}^{n_{i}}$ represents the unknown latent embedding of node ${n_{i}}$. 
$$
f(Y)=\\ell(Y,Y_{L})=\\sum_{i=1}^{k} \\ell({y}^{n_{i}} , y_{L}^{n_{i}})= \\ln(1+\\sum_{i=1}^{k}\\|{x}^{n_{i}} - x_{L}^{n_{i}}\\|^{2})= \\ln(1+\\|X-X_{L}\\|^{2})
$$


Since we only have access to the node features $X$ generated by encoder $\theta$, we need to make a connection between $\mathcal{Q}$ and $\mathcal{P}$. 

Lemma 1. For any annotation $y$ generated by LLMs, $\\mathbb{E}\_{\\theta\\sim \\mathcal{Q}}f(y)\\leq \\log \mathbb{E}_{\\theta\\sim \\mathcal{P}} \\exp(f(y))+KL( \\mathcal{Q} \\| \\mathcal{P})$

Proof:
$$
\\begin{aligned}
&\\\\
&{\\mathbb{E}\_{\\theta^{\\prime}\\sim \\mathcal{P}}{f(y)}}=\\int f(y) p(y) d y=\\int f(y) \\frac{p(y)}{q(y)} q(y) d y=\\mathbb{E}\_{\\theta\\sim \\mathcal{Q}} f(y) \\frac{p(y)}{q(y)}\\\\
&\\text{Since}\\quad KL(\\mathcal{Q} \\| \\mathcal{P})=\\mathbb{E}\_{\\theta\\sim Q} \\log \\frac{q(y)}{p(y)}\\\\
&\\log \\mathbb{E}\_{\\theta^{\\prime}\\sim \\mathcal{P}} f(y)= \\log\\mathbb{E}\_{\\theta\\sim \\mathcal{Q}} f(y) \\frac{p(y)}{q(y)} \\geq\\mathbb{E}\_{\\theta\\sim \\mathcal{Q}}\\log f(y) \\frac{p(y)}{q(y)}=\\mathbb{E}\_{\\theta\\sim \\mathcal{Q}}\\log f(y)-KL( \\mathcal{Q} \\| \\mathcal{P}) \\\\
\\end{aligned}
$$

Then, we transform the objective into 
$$
\\min\_{(n_{1},...,n_{k})} \\mathbb{E}\_{\\theta^{\\prime}\\sim \\mathcal{P}} \\exp(f(Y)) 
$$

Assuming the label distribution $\\mathcal{H}$, we further have
$$
\\min\_{(n_{1},...,n_{k})} \\mathbb{E}\_{Y\\sim \\mathcal{H}} \\mathbb{E}\_{\\theta^{\\prime}\\sim \\mathcal{P}} \\exp(f(Y)) = \\min\_{(n_{1},...,n_{k})} \\mathbb{E}\_{Y\\sim \\mathcal{H}} \\mathbb{E}\_{\\theta\\sim \\mathcal{P}}\\|X-X_{L}\\|^{2} \\\\
$$

Assuming $X$ and $X_{L}$ follows gaussian distribution, where $X\sim N(\mu_{i},\sigma_{i});X_L\sim N(\mu_{j},\sigma_{j})$, then

$$
\\begin{aligned}
\\mathrm{E}\\left(\\|X-X_{L}\\|^{2}\\right) & =\\mathrm{E}\\left(\\left\\|\\left(X-\\mu_{i}\\right)-\\left(X_{L}-\\mu_{j}\\right)+\\left(\\mu_{i}-\\mu_{j}\\right)\\right\\|^{2}\\right) \\\\
& =\\mathrm{E}\\left(\\left\\|X-\\mu_{i}\\right\\|^{2}\\right)+\\mathrm{E}\\left(\\left\\|X_{L}-\\mu_{j}\\right\\|^{2}\\right)+\\left\\|\\mu_{i}-\\mu_{j}\\right\\|^{2} \\\\
& =n \\sigma_{i}^{2}+n \\sigma_{j}^{2}+\\left\\|\\mu_{i}-\\mu_{j}\\right\\|^{2}
\\end{aligned}
$$

Since $(\mu_{j},\sigma_{j})$ is unknown, $\mu_{i}$ is fixed, thus we want to minimize $\sigma_{i}$.  Given an arbitrary node, $\sigma_{i}$ can be viewed as the distance to the clustering centers. The smaller $\sigma_{i}$ is, the smaller the corresponding $\mathrm{E}\left(\|X-X_{L}\|^{2}\right)$ also becomes, indicating that the minimum value of $ \\mathbb{E}\_{ \\theta \\sim \\mathcal{Q}} f(y)$ is attained when $\\sigma_{i}$ is at its smallest. This demonstrates why nodes closer to the clustering centers are "preferred" by LLMs and thus achieve better annotation quality.

Reviewer comment (Experiments):
Providing the number of nodes selected in each step in the experiment will also help in understanding the effects of the steps. For instance, could you please provide the number of selected nodes during active selection and DA-active selection? Also, what is the pruning ratio when applying PS? All this information can help in understanding the efficacy of the steps.

Author response:
**Response** Thanks for your great comments. For the number of nodes selected during active selection and DA-active selection, we adopt the same number of samples for each dataset as follows: the number of selected nodes is equal to the number of classes multiplied by $20$. For example, for Arxiv with $40$ classes, we select a total of $800$ nodes. For the pruning ratio, we drop 20\% of the selected nodes for every dataset except Pubmed. For Pubmed, we drop 10\% of the selected nodes because LLMs-as-predictors can achieve high accuracy on this dataset.

Reviewer comment (Writing):
Detail explanation on f_{act}(vi) is missing.

Author response:
**Response** Thanks for your great comments. $f_{act}(vi)$ can be any traditional graph active learning score functions, and we add a detailed explanation in the revision. Furthermore, we add more examples of the prompt and detailed descriptions in the appendix.

Reviewer comment (Writing):
Page 3: GNN modelson- models on (space missing)

Author response:
**Response** Thanks for your great comments. We've fixed these typos in the revision.

Reviewer comment (Experiments):
In Figure 2 or Figure 12, the trend of decreasing accuracy as the distance between nodes and cluster centers increases seems somewhat weak in average accuracy. Even in Table 2, when DA is added to traditional graph active selection and when DA is added to the use of PS, there are many cases where performance actually decreases. I acknowledge that tuning was not performed, but there are still too many cases where performance declines. While the authors said that grid search would improve the performance in a specific case, it seems necessary to perform more tuning across a broader range of cases to clearly demonstrate the effectiveness of DA.

Author response:
Q1-1: In Figure 2 or Figure 12, the trend of decreasing accuracy as the distance between nodes and cluster centers increases seems somewhat weak in average accuracy. 


R1-1: Thanks for your great question which inspires us to explain the details of our paper. There are line plots and bar plots in the figure. The changing trend is reflected by the line plot, which **shows a clear declining trend** while the bar plot only provides auxiliary information and may not affect the conclusion.

For Figure 2 and Figure 12, **the bar** shows the average values within each region, while the blue line represents the overall mean accuracy across different regions. Although the accuracy inside each bar doesn't show a clear trend, the line plot shows a clear decreasing trend across different regions. The nodes to be annotated are selected based on the accuracy across different groups (the blue line) rather than the accuracy inside each group (the bar). As a result, **the declining trend is clear** and difficulty-aware selection can effectively select those nodes with high-quality annotations.

Q1-2: Even in Table 2, when DA is added to traditional graph active selection and when DA is added to the use of PS, there are many cases where performance actually decreases. I acknowledge that tuning was not performed, but there are still too many cases where performance declines. While the authors said that grid search would improve the performance in a specific case, it seems necessary to perform more tuning across a broader range of cases to clearly demonstrate the effectiveness of DA.

R1-2: Thanks for your great question. We first summarize our responses and then provide more details for the responses.
(1) Applying the original DA may lead to clear performance degrade, which is due to the class imbalance problem by a small clustering center number $K$ on datasets like Pubmed. **This problem can be solved** by increasing $K$ to balance the diversity and annotation quality. (2) The original way to apply PS and DA together can not outperform DA. Although hyper-parameter tuning can definitely help, we agree that hyper-parameter tuning could not be flexible in practice. Therefore to further explore the potential of combining DA and PS, **we introduce a simple way to utilize the confidence as weight for loss functions and we find that the new variants can outperform DA in most cases without any hyper-parameter tuning.**


R1-2 (1) For the effectiveness of DA, we admit that sometimes DA will degrade the performance, especially on the Pubmed dataset. The reason for this phenomenon is that when performing K-means clustering with $C-Density$, the number of clustering centers $K$ is fixed as the number of classes for labels. This approach can achieve good labeling quality in most cases. On datasets like Pubmed where the number of classes is small (only 3), the points selected by $C-Density$ tend to cause class imbalance (the annotation quality is still very good). To solve this problem, we find that by setting a larger value for the number of clustering center $K$, the new $C-Density^{\prime}$ obtained can effectively solve this problem. We show the results in the following table. $k$ means the number of clustering centers to caluclate $C-Density^{\prime}$, $C$ means the number of classes, budgets means the number of selected annotated nodes. We use AGE as a demonstration to show the idea. From the results, we can see that setting a larger $K$ can improve the performance of DA on Pubmed.

|                  	| Pubmed |
|----------------|------|
|        AGE       	|  74.55 	|
|    DA-AGE(k=C)   	|  55.36 	|
|   DA-AGE(k=2C)   	|  57.38 	|
| DA-AGE(k=budget)|  75.71 	|

Reviewer comment (Experiments):
While there is an example in Figure 5, it would be beneficial to demonstrate on different datasets that LLM-GNN achieves competitive performance with GNNs trained on ground truth labeled data, while significantly reducing costs.

Author response:
**Response** Thanks for your great comments which help us to refine the details of our paper.  Compared to traditional GNN and LLMs-as-Predictors, **our methods can achieve promising performance with much lower costs.**

We demonstrate the economic comparison among LLM-GNN, LLMs-as-Predictors, and traditional pipelines to show the effectiveness. The first table demonstrates the performance of each pipeline. For LLMGNN and GNN, we deliberately let them produce a similar performance to conduct a fair cost comparison. For Pubmed*, we demonstrate the performance of LLMGNN with a larger budget since the performance of LLMs-as-Predictors is superior on this dataset, and increasing a little budget can improve the performance a lot. 
|     (Accuracy)     	|  Cora 	| CiteSeer 	| Pubmed 	| Pubmed* 	| WikiCS 	| Arxiv 	| Products 	|
|------------------|-----|--------|------|-------|------|-----|--------|
|       LLMGNN       	| 75.54 	|   69.06  	|  74.98 	|  81.95  	|  66.09 	| 66.14 	|   74.9   	|
| LLMs-as-Predictors 	| 68.33 	|   66.33  	|  87.33 	|  87.33  	|   71   	| 73.33 	|   75.33  	|
|         GNN        	| 75.54 	|   69.06  	|  74.98 	|  81.95  	|  66.09 	| 66.14 	|   74.9   	|

The second table compares the number of training samples needed to achieve the performance in the first table. Here, PL stands for pseudo labels generated by LLMs. GT stands for ground truth labels.
| (Number of   annotations) 	| Cora 	| CiteSeer 	| Pubmed 	| Pubmed* 	| WikiCS 	|  Arxiv 	| Products 	|
|-------------------------|----|--------|------|-------|------|------|--------|
|         LLMGNN (PL)        	|  140 	|    120   	|   60   	|   300   	|   200  	|   800  	|    940   	|
|   LLMs-as-Predictors (PL)  	| 2708 	|   3186   	|  19717 	|  19717  	|  11701 	| 169343 	|  2449029 	|
|          GNN (GT)         	|  50  	|    50    	|   42   	|   210   	|   40   	|   560  	|    400   	|


From these two tables, we can see that **LLMGNN significantly reduces the costs compared to LLMs-as-Predictors**. A recent study [1], shows that the annotations generated by ChatGPT are more than **20 times** cheaper than human annotators. If we use this ratio, we can see that although LLMGNN requires more pseudo labels, the overall costs are still much cheaper than traditional GNN-based pipelines. Using the OGBN-Products dataset as an example, if we estimate the cost, using LLMs-as-Predictors would incur an expense of 1572 dollars, while manual annotation would require at least 20 dollars (assuming each sample has 3 workers on a crowdsourcing platform), and it would also necessitate a considerable amount of waiting time. In contrast, using LLMGNN can complete annotation and prediction in a matter of minutes at a cost of less than one dollar.

[1] Gilardi F, Alizadeh M, Kubli M. Chatgpt outperforms crowd-workers for text-annotation tasks[J]. arXiv preprint arXiv:2303.15056, 2023.