Paper: Training Graph Transformers via Curriculum-Enhanced Attention Distillation (ICLR-2024)

Reviewer comment (Experiments):
My major concern is the motivation of the distillation of GTs. Why not apply distillation on other graph networks to obtain more significant performance, e.g. GCNs? Can the proposed method be applied on other networks?

Author response:
Thank you for the question. 

First of all, our proposed method can surely be applied on other graph networks, including GCNs. 


The motivation behind distilling GTs lies in our aim to enhance the training of Graph Transformer models through knowledge distillation, with the ultimate goal of unleashing the potential of Transformer models in node classification tasks, surpassing the performance of GNN models. Therefore, we choose Transformer models as both teacher and student models, aligning strictly with our motivation. In other words, we do not prefer GNNs to ensure a fair comparison. We aim to show the outstanding potential of Transformer-based or attention-based models enhanced by our proposed distillation framework.


While initially tailored for Transformers and not specifically intended for GNNs, our method is easily adaptable and extendable to other GNN models, including GCN [1] and SGC [2], with some necessary minor modifications. Specifically, we can generate similarity matrices for non-attention models via their node embeddings. These matrices can be rough approximations of the attention matrices in our knowledge distillation framework.


Below are the results of our preliminary experiments involving distillation from GCN to GGT, GCN to GCN, LGT to GCN, LGT to GGT.

| **Method** | **Cora** | **Citersser** |
|:--------:|:----------:|:----------:|
| Vanilla GCN     | 82.06 $\pm$ 0.69 | 71.6 $\pm$ 0.32 |
| Vanilla LGT      | 77.40 $\pm$ 0.56 | 68.05 $\pm$ 1.06 |
| Vanilla GGT      | 54.24 $\pm$ 3.16 | 50.60 $\pm$ 3.83 |
| GCN $\rightarrow$ GGT      | 82.90 $\pm$ 0.76 | 72.46 $\pm$ 0.95 |
| GCN $\rightarrow$ GCN      | 82.28 $\pm$ 0.37 | 72.43 $\pm$ 0.35  |
| LGT $\rightarrow$ GCN      | 82.85 $\pm$ 0.85 | 73.73 $\pm$ 0.55 |
| LGT $\rightarrow$ GGT      | **84.70 $\pm$ 0.56** | **76.40 $\pm$ 0.70** |

The above results indicate that all four distillation methods can lead to a certain degree of performance improvement in the student model. However, our method achieved the best results on both datasets, significantly outperforming other variants. We attribute this performance gap to the inherent advantages of using Transformer models as both teacher and student models.

As mentioned and discussed in Section 3.1 of the main text, the internal attention coefficients in Transformer models are automatically and inherently learned, eliminating the need to generate additional relational structures based on embeddings as required in GCNs. This ensures the reliability of attention mechanisms, as the teacher model strictly follows these learned coefficients for aggregation. In contrast, considering other models (such as GCN), the similarity matrix is derived from their node embeddings in a post-hoc manner, and thus, its reliability, interpretability, and quality heavily depend on and are restricted by the embeddings. That's why subsequent construction of relational structures (e.g., similarity matrix) cannot be guaranteed to be equally reliable, interpretable, and high-quality. From this perspective, inherent attention-based models might be more suitable and can benefit more from our distillation framework, just as consistent larger performance gains shown in the above table (average absolutely 2.35\% improvement for the listed datasets).

We sincerely thank you for your advice again, and we have included all the discussion in Appendix H of this revision.

Reviewer comment (Experiments):
In Ablation, which components are the most important? Why "w.o. $\gamma$" , "Global uniform distribution" bring such damage for the performance?

Author response:
In ablation, attention distillation emerges as the most crucial component in our method. The significance of attention distillation is thoroughly analyzed in Section 3.1. Also we list the reason as below: 

Attention distillation, characterized by the transfer of attention coefficients from the teacher to the student model, serves as a foundational component in our method. It enables the student to leverage the teacher’s acquired attention patterns. This critical process allows the student model to inherit and benefit from the teacher's learned attention patterns, enabling the student to utilize the acquired implicit knowledge in out-of-class exploration to discover more valuable interactions. Consequently, it plays a central role in enhancing the student's ability to capture important features in the data.


The ablation experiment "w.o. $\gamma$" involves setting the hyperparameter $\gamma$ to 0. In this case, the loss function comprises only the supervised loss and out-of-distribution entropy, essentially eliminating the distillation loss. This implies that the student receives no guidance from the teacher. Driven by out-of-distribution entropy, the student engages in out-of-class exploration without acquiring much beneficial knowledge. This impedes model training and results in a performance decline. This ablation experiment emphasizes the importance of teacher guidance and demonstrates that the utilization of out-of-distribution entropy is contingent on teacher guidance.

As for the "Global uniform distribution”, this ablation involves replacing all the teacher model's attention coefficients with a global uniform distribution, eliminating the effective guidance provided by the teacher model to the student model. By only providing a uniform distribution, the student struggles to learn implicit knowledge from valuable neighbor interactions, hindering its ability to discover distant yet valuable interactions during out-of-class exploration. Therefore, guidance that lacks valuable knowledge and is almost equivalent to having no guidance (removing the teacher model and letting the student train independently) does not lead to significant improvements and may even be detrimental. This ablation experiment helps us realize this point, confirming that the crucial factor for performance improvement lies in the guidance provided by the teacher.

Reviewer comment (Experiments):
Why not compare to some general KD methods for classification task, e.g. KD, AT, FitNets, DKD, CRD?

Author response:
We appreciate the reviewer's insightful question.

Indeed, we have conducted a comparison with several general knowledge distillation methods, including KD and FitNets. The experimental results on the Cora and Citeseer datasets are listed as below:

| Datasets / Models | Teacher  |  Student |  Distilled |
|-------------------|----------|----------|------------|
| **Cora**          |          |          |            |
| KD                |   77.40  |   55.90  |   77.34    |
| FitNets           |   77.40  |   54.80  |   77.19    |
| Ours              |   77.40  |   54.24  | **84.70** |
| **Citeseer**      |          |          |            |
| KD                |   68.60  |   55.80  |   70.27    |
| FitNets           |   68.60  |   57.50  |   69.51    |
| Ours              |   68.05  |   50.60   |  **76.40** |

From the experimental results, it can be observed that on the Cora and Citeseer datasets, KD achieves an average absolute performance improvement of 17.95\% compared to the student model without guidance. Similarly, FitNets also achieves an average absolute performance improvement of 17.2\%. In general, distilling knowledge from a teacher model by using its outputs or intermediate layer representations to train a student model can facilitate the student's training. 

However, after careful consideration, we decided not to include these results in the paper at that time. The primary reason for this decision stems from the distinct focus and motivation behind our method, which diverges from the objectives of traditional knowledge distillation methods. Unlike methods primarily designed for model compression, our method specifically addresses challenges related to Graph Transformers, particularly in semi-supervised scenarios.

we appreciate the reviewer's suggestion regarding a more comprehensive comparative analysis. In response, we have included the results of some general knowledge distillation methods on multiple datasets and provide a more detailed analysis in Appendix F.1 of the revision. Thank you once again for your suggestion, as it contributes to the improvement of the quality of our paper.

Reviewer comment (Reproducibility):
Please provide the details of Teacher and student, including architecture, parameters, etc. If the utilized "Graph Transformer" equal to "Graphormer" or "Nodeformer"? If not, why not conduct distillation on them?

Author response:
For the LGT model, we implement it using a GT [5] model with attention computations limited to the first-order neighborhood of each node, including itself. As for the GGT model, we opt for the vanilla Transformer [6] implementation, where GGT’s Transformer layers are identical to those of the standard Transformer. Both LGT and GGT models have their Transformer layers fixed at 2, and the specific ranges for other hyperparameters can be found in Appendix E.2. In the revision, we have included specific hyper-parameter configurations on each dataset in Appendix E.2. 


In our model selection, the chosen model is neither Graphormer nor Nodeformer. We opt for simple frameworks to showcase the potential of our method, hence choosing the naive GGT encoders. As long as it aligns with our motivation – using a teacher model with reliable performance to guide a student model with significant potential but susceptible to overfitting – we can replace the student model with other architectures, such as Graphormer [7] and Nodeformer [8]. We attempt to replace the GGT model we use with Graphormer and SAT [9] (another type of Global Graph Transformer), and the experimental results are presented below. Due to Nodeformer not explicitly constructing attention matrices, implementation requires more advanced techniques, which we plan to explore in the future. It can be observed that, compared to unguided student models, our method significantly improves the performance of student models, surpassing that of teachers. This also demonstrates the superiority of our method. We plan to explore the integration of methods and techniques from prior research with our framework in future work, aiming to achieve enhanced performance.


| Datasets / Models | Teacher  |  Student |  Distilled |
|-------------------|----------|----------|------------|
| **Cora**          |          |          |            |
| LGT $\rightarrow$ Graphormer   |   78.90  |   56.50  |   **81.18**    |
| LGT $\rightarrow$ SAT          |   78.10  |   60.80  |   79.80    |
| **Citeseer**      |          |          |            |
| LGT $\rightarrow$ Graphormer   |   67.80  |   57.90  |   **73.28**    |
| LGT $\rightarrow$ SAT          |   67.80  |   58.30  |   72.20    |

Thank you for your valuable suggestion, as it significantly contributes to enhancing the quality of our paper.

Reviewer comment (Experiments):
The loss function is complex with a few components. In Table 4 the authors evaluated their pipeline's performance sensitivity to each of the components. My first question is how sensitive the model is to variation of these hyperparameters (e.g. $\gamma$, $\beta$)? Both in case of stability and performance.

Author response:
**A1**:Thank you for your insightful question. We have conducted a thorough sensitivity analysis of hyperparameters in Appendix F.1 of our submission. In Appendix F.1, we investigate the impact of variations in hyperparameters $\gamma$ and $\beta$ on the model's stability and performance. Specifically, when keeping $\gamma$ fixed at an appropriate value, we observe that increasing $\beta$ leads to a decline in model performance when $\beta$ exceeds a certain threshold. Conversely, when keeping $\beta$ fixed at an appropriate value, we observe that increasing $\gamma$ gradually improves model performance, reaching a plateau after a certain point. Please kindly find a comprehensive analysis in Appendix F.1. We’ll ensure a clear link between Section 4 and Appenxix F.1 in the revision.

Reviewer comment (Experiments):
Can authors please comment on the scalability of this model? Does it scale to bigger datasets? If yes, have they tried running their model on those?

Author response:
**A2**:We appreciate the reviewer's insight into the scalability of our method. The scalability of our method depends on the scalability of the chosen student model. We choose the vanilla Transformer [1] as our student model for simplicity. Though the specific chosen student model, i.e. GGT, in our framework has relatively lower scalability, Our method allows for the replacement of the student model effortlessly. For instance, our method can scale to larger graphs by substituting the student model with a more scalable Transformer variant, such as SGFormer [2]. Given the orthogonality and complementarity of our work, we aim to conduct in-depth exploration, investigation, and discussion of these extensions and integrations in future research, accompanied by comprehensive experimental studies. We believe that by incorporating certain methods and techniques from previous research, we can effortlessly enhance scalability and expect further performance improvements.

Reviewer comment (Evaluation):
While outperforming baselines, the absolute performance gaps are sometimes small (~1-2%). The gains may not justify the added complexity in some applications.

Author response:
**A2**:Thank you for the insightful question. While, on certain datasets, the absolute performance improvement of our method over baselines may not be substantial, our method consistently improves performance across all seven datasets, with an average absolute improvement of 4.9\%. Notably, we achieve a significant gain of 7.74\% on the Texas dataset, as detailed in Section 4.2. Different from traditional knowledge distillation methods that mainly focus on model compression, our method presents a novel and interesting motivation, i.e., enabling the student model to surpass the performance of the teacher model (See detailed discussions in Section 4.3 in the main text for a specific distillation comparison), which is definitely ignored by prior works. Additionally, our framework's adaptability and scalability are noteworthy, relying on a proposed loss function that is easy to implement in practice.

Reviewer comment (Experiments):
Curriculum scheduling adds hyperparameters like epoch boundaries. The impact of these hyperparameters could be studied more systematically. In addition, I highly recommend that authors use auto-distillation (KD-Zero: Evolving Knowledge Distiller for Any Teacher-Student Pairs (NeurIPS-2023, Automated Knowledge Distillation via Monte Carlo Tree Search (ICCV2023)) to optimize different parameter options.

Author response:
**A3**:Thank you for the valuable feedback. We find your advice quite helpful and beneficial for our work. While we have discussed some important hyperparameters in Curriculum scheduling in Appendix F.1, we acknowledge that our analysis may not have been exhaustive. We appreciate the suggestion to employ auto-distillation techniques such as KD-Zero [1] and Automated Knowledge Distillation via Monte Carlo Tree Search [2], which could provide a more systematic exploration of hyperparameter options. We will thoroughly explore and incorporate these extensions into our future work. We sincerely appreciate your guidance, and we believe your advice will significantly contribute to refining our research in upcoming studies.

Reviewer comment (Reproducibility):
What is the exact distance metric $d$ do you use?

Author response:
**A2**: In our experiments, we use cross-entropy as the distance metric, which is equivalent to the Kullback-Leibler (KL) divergence. We opt for cross-entropy mainly for its simplicity and widespread use in the machine learning community as a common metric for probability distributions. We have emphasized this point in Appendix E.2 of the revision to enhance the paper's clarity. 

In addition, most distance metrics are applicable in our experiments, such as Euclidean distance. We notice that different distance metrics may impact our distillation method. An appropriate distance metric has the potential to facilitate better knowledge transfer and improve student training. However, we feel that a more in-depth exploration of this aspect goes beyond the scope of this paper and is orthogonal to our current focus. We plan to explore and extend our framework in future work to delve into this aspect more thoroughly.

Reviewer comment (Experiments):
For the top-k pruning, what is the exact $k$ do you use? Does it vary significantly between datasets? For example, Pubmed has 19,717 nodes while Cornell and Texas only have 183 nodes, does the k values different?

Author response:
**A3**:Thank you for raising a crucial question. For top-k pruning, we customize the selection of $k$ values based on dataset size. Smaller datasets like Cornell and Texas utilize $k$ values such as 5,10,15, and 20, while larger datasets have a broader but still constrained range, typically from 10 to 50. Considering that different datasets require different $k$ values, the specific $k$ values are determined through grid search, ensuring the most suitable $k$ for each dataset. For the Pubmed dataset, we set $k=20$, while for the Cornell and Texas datasets, $k=10$. In our submission, we have already included the hyperparameter search space in Appendix E.2. We acknowledge that we overlooked detailing the specific hyperparameter configurations for each dataset in the paper. In the revision, we have provided detailed information regarding the hyperparameter configurations for each dataset in Appendix E.2. We appreciate your guidance and understand the importance of this clarification for reproducibility. These additions will enhance the overall quality of our study. Once again, we express our gratitude for your professional advice.

Reviewer comment (Reproducibility):
What are the specific LGT and GGT models do you use in the experiments?

Author response:
**A4**:Thank you for addressing a vital question. For the LGT model, we implement it based on the GT model [1], which computes attention by considering only the first-order neighborhood (including the node itself) for each node. As for the GGT model, we adopt the vanilla Transformer architecture [2]. In particular, the implementation of the Transformer layer in our GGT model closely follows that of the vanilla Transformer model. In our model selection, we lean towards straightforward frameworks to demonstrate the potential of our method. The chosen LGT and GGT models are based on the fundamental Transformer encoder, emphasizing the effectiveness achievable with the native attention layer. This choice aligns with our motivation: employing a teacher model with decent performance and stability to guide a student model with significant potential but prone to overfitting. This represents a paradigm distinct from traditional knowledge distillation. We also explore various teacher-student model combinations to validate the rationale and effectiveness of our choices in Section 4.4 of the main text.

**Reference**

[1] Vijay Prakash Dwivedi and Xavier Bresson. A generalization of transformer networks to graphs. arXiv preprint arXiv:2012.09699, 2020.

[2] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.