Original: Geometric Knowledge Distillation: Topology Compression for Graph Neural Networks
Revised: Geometric Knowledge Distillation: Topology Compression for Graph Neural Networks

Original: Abstract
Revised: Abstract

Original: We study a new paradigm of knowledge transfer that aims at encoding graph topological information into graph neural networks (GNNs) by distilling knowledge from a teacher GNN model trained on a complete graph to a student GNN model operating on a smaller or sparser graph.
Revised: We study a new paradigm of knowledge transfer that aims at encoding graph topological information into graph neural networks (GNNs) by distilling knowledge from a teacher GNN model trained on a complete graph to a student GNN model operating on a smaller or sparser graph.

Original: To this end, we revisit the connection between thermodynamics and the behavior of GNN, based on which we propose
Revised: To this end, we revisit the connection between thermodynamics and the behavior of GNN, based on which we propose

Original: Neural Heat Kernel (NHK) to encapsulate the geometric property of the underlying manifold concerning the architecture of GNNs.
Revised: Neural Heat Kernel (NHK) to encapsulate the geometric property of the underlying manifold concerning the architecture of GNNs.

Original: A fundamental and principled solution is derived by aligning NHKs on teacher and student models, dubbed as
Revised: A fundamental and principled solution is derived by aligning NHKs on teacher and student models, dubbed as

Original: Geometric Knowledge Distillation.
Revised: Geometric Knowledge Distillation.

Original: We develop non-parametric and parametric instantiations and demonstrate their efﬁcacy in various experimental settings for knowledge distillation regarding different types of privileged topological information and teacher-student schemes.
Revised: We develop non-parametric and parametric instantiations and demonstrate their efﬁcacy in various experimental settings for knowledge distillation regarding different types of privileged topological information and teacher-student schemes.

Original:  Introduction
Revised:  Introduction

Original: Modern graph neural networks (GNNs) [28; 48; 52] have shown remarkable performance in learning representations for structured instances.
Revised: Modern graph neural networks (GNNs) [28; 49; 53] have shown remarkable performance in learning representations for structured instances.

Original: From the perspective of geometric deep learning [5; 4; 37], part of the achievement of GNNs can be attributed to their implementation of the permutation invariance property as geometric priors 2 into the architecture design.
Revised: From the perspective of geometric deep learning [5; 4; 38], part of the achievement of GNNs can be attributed to their implementation of the permutation invariance property as geometric priors 2 into the architecture design.

Original: Nevertheless, in practice, GNNs highly rely on graph topology, as essential input information, to explore the relational knowledge implicit in interactions of instance pairs throughout the entire message passing process, termed as geometric knowledge in this paper.
Revised: Nevertheless, in practice, GNNs highly rely on graph topology, as essential input information, to explore the relational knowledge implicit in interactions of instance pairs throughout the entire message passing process, termed as geometric knowledge in this paper.

Original: As advances in generalized distillation [32; 46] reveal the possibility of encoding input features into model construction, natural questions arise as to:
Revised: As advances in generalized distillation [33; 47] reveal the possibility of encoding input features into model construction, natural questions arise as to:

Original: Is it possible and if so, how can we encode graph topology as a special type of ‘geometric prior’ into a GNN model, such that the model could precisely capture the underlying geometric knowledge even without full graph topology as input?
Revised: Is it possible and if so, how can we encode graph topology as a special type of ‘geometric prior’ into a GNN model, such that the model could precisely capture the underlying geometric knowledge even without full graph topology as input?

Original: In speciﬁc we are interested in the following geometric knowledge transfer problem: a GNN model (with node-speciﬁc outputs for node-level prediction [23]) is exposed with a partial graph, which is a subset of the complete graph.
Revised: In speciﬁc we are interested in the following geometric knowledge transfer problem: a GNN model (with node-speciﬁc outputs for node-level prediction [23]) is exposed with a partial graph, which is a subset of the complete graph.

Original: Formally speaking, we have notations:
Revised: Formally speaking, we have notations:

Original: [Equation]
Revised: [Equation]

Original: Our goal is to transfer or encode geometric knowledge extracted from ˜ G to the target GNN model that is only aware of G .
Revised: Our goal is to transfer or encode geometric knowledge extracted from ˜ G to the target GNN model that is only aware of G .

Original: Studying this problem is also of much practical value.
Revised: Studying this problem is also of much practical value.

Original: As a non-exhaustive
Revised: As a non-exhaustive

Original: A B
Revised: A B

Original: H P O 
Revised: H P O (cid:48)

Original: M R V
Revised: R V

Original: W
Revised: W

Original: N C
Revised: N (cid:38)

Original: D E
Revised: D E

Original: L
Revised: L

Original: G
Revised: G

Original: U S F
Revised: U (cid:54) F

Original: Y X T Q
Revised: Y (cid:59) T Q

Original: I J
Revised: I J

Original: K
Revised: K

Original: A
Revised: A

Original: O
Revised: O

Original: M
Revised: M

Original: R V
Revised: R V

Original: W
Revised: W

Original: N
Revised: N

Original: D E
Revised: D E

Original: L
Revised: L

Original: G
Revised: G

Original: U S F
Revised: U S F

Original: A B
Revised: A B

Original: H P O 
Revised: H P O (cid:48)

Original: M R V
Revised: R V

Original: W
Revised: W

Original: N C
Revised: N (cid:38)

Original: D E
Revised: D E

Original: L
Revised: L

Original: G
Revised: G

Original: U S F
Revised: U (cid:54) F

Original: T Q
Revised: T Q

Original: I J
Revised: I J

Original: K
Revised: K

Original: [Figure]
Revised: [Figure]

Original: list of applications: improving efﬁciency without compromising on effectiveness for coarsened graphs [14; 24; 66], privacy constrained scenarios in social recommenders or federated learning where the complete graph is unavailable [33; 49; 65], promoting concentration on targeted community to bring up economic beneﬁts [55].
Revised: list of applications: improving efﬁciency without compromising on effectiveness for coarsened graphs [14; 24; 69], privacy constrained scenarios in social recommenders or federated learning where the complete graph is unavailable [34; 50; 68], promoting concentration on targeted community to bring up economic beneﬁts [57].

Original: Achieving this target is non-trivial for that we need to ﬁrst ﬁnd a principled and fundamental way to encapsulate the geometric knowledge extracted by GNN model, which requires in-depth investigation on the role of graph topology throughout the progressive process of message passing.
Revised: Achieving this target is non-trivial for that we need to ﬁrst ﬁnd a principled and fundamental way to encapsulate the geometric knowledge extracted by GNN model, which requires in-depth investigation on the role of graph topology throughout the progressive process of message passing.

Original: Therefore, we take a thermodynamic view borrowed from physics and propose a new methodology built upon recent advances revealing the connection between heat diffusion and architectures of GNNs [8; 50; 7].
Revised: Therefore, we take a thermodynamic view borrowed from physics and propose a new methodology built upon recent advances revealing the connection between heat diffusion and architectures of GNNs [8; 51; 7].

Original: Speciﬁcally, we interpret feature propagation as heat ﬂows on the underlying Riemmanian manifold, whose characteristics (that are dependent on graph topology and the GNN model) pave the way for a principled representation of the latent geometric knowledge.
Revised: Speciﬁcally, we interpret feature propagation as heat ﬂows on the underlying Riemmanian manifold, whose characteristics (that are dependent on graph topology and the GNN model) pave the way for a principled representation of the latent geometric knowledge.

Original: 1.1 Our Contributions
Revised: 1.1 Our Contributions

Original: New theoretical perspective for analyzing latent graph geometry.
Revised: New theoretical perspective for analyzing latent graph geometry.

Original: On top of the connection between heat equation and GNNs, we step further to inspect the implication of heat kernel for
Revised: On top of the connection between heat equation and GNNs, we step further to inspect the implication of heat kernel for

Original: GNNs, and propose a novel notion of Neural Heat Kernel (NHK) with rigorous proof of its existence.
Revised: GNNs, and propose a novel notion of Neural Heat Kernel (NHK) with rigorous proof of its existence.

Original: Heat kernel intrinsically deﬁnes the unique solution to the heat equation and can be a fundamental characterization for the geometric property of the underlying manifold [18; 19].
Revised: Heat kernel intrinsically deﬁnes the unique solution to the heat equation and can be a fundamental characterization for the geometric property of the underlying manifold [18; 19].

Original: Likewise, NHK uncovers geometric property of the latent graph manifold for GNNs, and governs how information ﬂows between pairs of instances, which lends us a mathematical tool to encapsulate geometric knowledge extracted from GNN model and enables geometric knowledge transfer.
Revised: Likewise, NHK uncovers geometric property of the latent graph manifold for GNNs, and governs how information ﬂows between pairs of instances, which lends us a mathematical tool to encapsulate geometric knowledge extracted from GNN model and enables geometric knowledge transfer.

Original: This result alone might also be useful in broader contexts for understanding GNNs.
Revised: This result alone might also be useful in broader contexts for understanding GNNs.

Original: Flexible distillation framework with versatile instantiations.
Revised: Flexible distillation framework with versatile instantiations.

Original: Based on the above insights, we treat NHK matrices as representation of the latent geometric knowledge, upon which we build a ﬂexible and principled distillation framework dubbed as Geometric Knowledge Distillation (GKD), which aims at encoding and transferring geometric knowledge by aligning latent manifolds behind
Revised: Based on the above insights, we treat NHK matrices as representation of the latent geometric knowledge, upon which we build a ﬂexible and principled distillation framework dubbed as Geometric Knowledge Distillation (GKD), which aims at encoding and transferring geometric knowledge by aligning latent manifolds behind

Original: GNN models as illustrated in Fig.
Revised: GNN models as illustrated in Fig.

Original: We also develop non-parametirc and parametric versions of
Revised: We also develop non-parametirc and parametric versions of

Original: GKD, in terms of different ways to approximate NHK computation.
Revised: GKD, in terms of different ways to approximate NHK computation.

Original: Speciﬁcally, the former derives explicit NHKs via assumptions on latent space, and the later learns NHK in a data-driven manner.
Revised: Speciﬁcally, the former derives explicit NHKs via assumptions on latent space, and the later learns NHK in a data-driven manner.

Original: Applications for geometric knowledge transfer and conventional KD purposes.
Revised: Applications for geometric knowledge transfer and conventional KD purposes.

Original: We verify the efﬁcacy of GKD in terms of different geometric knowledge types (i.e., edge-aware and node-aware ones), and further show its effectiveness for conventional KD purposes (e.g., model compression, self distillation, online distillation) for broader applicability.
Revised: We verify the efﬁcacy of GKD in terms of different geometric knowledge types (i.e., edge-aware and node-aware ones), and further show its effectiveness for conventional KD purposes (e.g., model compression, self distillation, online distillation) for broader applicability.

Original: We highlight that our methods consistently exceed teacher model and rival with the oracle model that gives the performance upper bound in principle.
Revised: We highlight that our methods consistently exceed teacher model and rival with the oracle model that gives the performance upper bound in principle.

Original: 1.2 Links to Related Works
Revised: 1.2 Links to Related Works

Original: Geometric Deep Learning.
Revised: Geometric Deep Learning.

Original: The study of geometric deep learning [5; 37] provides fundamental principles and methodology to generalize deep learning methods to non-Euclidean domains (e.g., graphs and manifolds).
Revised: The study of geometric deep learning [5; 38] provides fundamental principles and methodology to generalize deep learning methods to non-Euclidean domains (e.g., graphs and manifolds).

Original: From this perspective, architectures for off-the-shelf GNNs [28; 48; 52; 20]  have naturally incorporated the geometric prior knowledge for graphs such as permutation invariance .
Revised: From this perspective, architectures for off-the-shelf GNNs [28; 49; 53; 20] and graph Transformers [67; 56] have naturally incorporated the geometric prior knowledge for graphs such as permutation invariance .

Original: Despite their remarkable success, they highly rely on the graph topology.
Revised: Despite their remarkable success, they highly rely on the graph topology.

Original: This work extends the idea of geometric deep learning by treating the global graph topology as a special type of prior knowledge, and attempts to encode it into GNNs themselves, such that the trained model would leverage information from the global graph topology even without explicitly taking it as input.
Revised: This work extends the idea of geometric deep learning by treating the global graph topology as a special type of prior knowledge, and attempts to encode it into GNNs themselves, such that the trained model would leverage information from the global graph topology even without explicitly taking it as input.

Original: Graph-Based Knowledge Distillation.
Revised: Graph-Based Knowledge Distillation.

Original: Knowledge distillation (KD)
Revised: Knowledge distillation (KD)

Original: [22; 1] uses the outputs of a teacher model as alternative supervised signals to teach a student model, with various new paradigms including feature-based [41; 64] and relation-based [62; 63] ones.
Revised: [22; 1] uses the outputs of a teacher model as alternative supervised signals to teach a student model, with various new paradigms including feature-based [42; 66] and relation-based [64; 65] ones.

Original: While some prior arts [9;59; 49; 66; 61; 58] attempted to combine KD and GNNs, i.e., graph-based KD, they are nearly straight-forward adaptations of KD without in-depth investigation on the role of graph topology, also restricted by a speciﬁc choice of GNN architecture or application scenarios.
Revised: While some prior arts [9; 61; 50; 69; 63; 60] attempted to combine KD and GNNs, i.e., graph-based KD, they are nearly straight-forward adaptations of KD without in-depth investigation on the role of graph topology, also restricted by a speciﬁc choice of GNN architecture or application scenarios.

Original: In contrast, we ﬁrst formalize the problem of geometric knowledge transfer, theoretically answer the question “how to represent graph geometric knowledge and encode it into GNN models", and propose geometric distillation approach based on the theoretical results.
Revised: In contrast, we ﬁrst formalize the problem of geometric knowledge transfer, theoretically answer the question “how to represent graph geometric knowledge and encode it into GNN models", and propose geometric distillation approach based on the theoretical results.

Original: More discussions on related works are deferred to Appendix G.
Revised: More discussions on related works are deferred to Appendix G.

Original:  Preliminaries
Revised:  Preliminaries

Original: We commence with a brief detour to heat equation on Riemannian manifolds, and its connection with modern GNN architectures.
Revised: We commence with a brief detour to heat equation on Riemannian manifolds, and its connection with modern GNN architectures.

Original: Moreover, we bring forth the notion of heat kernel to motivate this work.
Revised: Moreover, we bring forth the notion of heat kernel to motivate this work.

Original: Heat Equation on Manifolds.
Revised: Heat Equation on Manifolds.

Original: We are interested in heat equation deﬁned on a smooth kdimensional Riemannian manifold M .
Revised: We are interested in heat equation deﬁned on a smooth kdimensional Riemannian manifold M .

Original: Suppose the manifold is associated with a scalar- or vectorvalued function x ( u, t ) :
Revised: Suppose the manifold is associated with a scalar- or vectorvalued function x ( u, t ) :

Original: M ⇥ [0 , 1 ) !
Revised: M ⇥ [0 , 1 ) !

Original: R d , quantifying a speciﬁc type of signals such as heat at a point u 2 M and time t .
Revised: R d , quantifying a speciﬁc type of signals such as heat at a point u 2 M and time t .

Original: Fourier’s law of heat conductivity describes the ﬂow of heat with respect to time and space, via a partial differential equation (PDE) called heat equation [6], i.e., 
Revised: Fourier’s law of heat conductivity describes the ﬂow of heat with respect to time and space, via a partial differential equation (PDE) called heat equation [6], i.e., [Equation]

Original: [Equation]
Revised: [Equation]

Original:  where c > 0 is the thermal conductivity coefﬁcient, and  is the natural Laplace–Beltrami operator associated with M .
Revised: ) where c > 0 is the thermal conductivity coefﬁcient, and  is the natural Laplace–Beltrami operator associated with M .

Original: Rewriting  as the functional composition of the divergence operator r ⇤ and gradient operator r , i.e.,  = r
Revised: Rewriting  as the functional composition of the divergence operator r ⇤ and gradient operator r , i.e.,  = r

Original: ⇤  r , we can interpret the heat equation as: the variation of temperature within an inﬁnitesimal time interval at a point is equivalent to the divergence between its own temperature and the average temperature on an inﬁnitesimal sphere around it.
Revised: ⇤  r , we can interpret the heat equation as: the variation of temperature within an inﬁnitesimal time interval at a point is equivalent to the divergence between its own temperature and the average temperature on an inﬁnitesimal sphere around it.

Original: Implications on Graphs.
Revised: Implications on Graphs.

Original: A spatial discretisation of a continuous manifold yields a graph G = {V , E} , whose nodes can be thought of as embedded on the base manifold.
Revised: A spatial discretisation of a continuous manifold yields a graph G = {V , E} , whose nodes can be thought of as embedded on the base manifold.

Original: In fact, the heat equation along with variants thereof (e.g., Schrödinger equation) have found widespread use in modeling graph dynamics [11; 25; 35].
Revised: In fact, the heat equation along with variants thereof (e.g., Schrödinger equation) have found widespread use in modeling graph dynamics [11; 25; 36].

Original: More importantly, it has been recently revealed to be intimately related with the architectures of modern GNNs [50; 8; 7]: suppose X (0) =
Revised: More importantly, it has been recently revealed to be intimately related with the architectures of modern GNNs [51; 8; 7]: suppose X (0) =

Original: { x ( u, 0) } u 2 V 2 R n ⇥ d denotes the initial condition for Eqn.
Revised: { x ( u, 0) } u 2 V 2 R n ⇥ d denotes the initial condition for Eqn.

Original: (2) determined by input node features, then solving the heat equation under certain deﬁnitions of r ⇤ and r (i.e., deﬁnition of  ) amounts to different architectures of GNNs.
Revised: (2) determined by input node features, then solving the heat equation under certain deﬁnitions of r ⇤ and r (i.e., deﬁnition of  ) amounts to different architectures of GNNs.

Original: For instance:
Revised: For instance:

Original: Example 1.
Revised: Example 1.

Original: [50] Deﬁne the discretised counterpart of  as the graph Laplacian matrix L = eD  12 ( e
Revised: [51] Deﬁne the discretised counterpart of  as the graph Laplacian matrix L = e

Original: D 
Revised: D

Original: [Equation]
Revised: [Equation]

Original: e
Revised: e

Original: D  12 .
Revised: D  12 .

Original: Numerically solving Eqn.
Revised: Numerically solving Eqn.

Original: (2) using the forward Euler method with step size ⌧ = 1 yields the formulation of Simple Graph Convolution (SGC) [52], where ⇥ denote learnable transformation matrix
Revised: (2) using the forward Euler method with step size ⌧ = 1 yields the formulation of Simple Graph Convolution (SGC) [53], where ⇥ denote learnable transformation matrix

Original: [Equation]
Revised: [Equation]

Original: Example 2.
Revised: Example 2.

Original: [8] Deﬁne the gradient operator r ij as the difference of source and target node features, the divergence operator
Revised: [8] Deﬁne the gradient operator r ij as the difference of source and target node features, the divergence operator

Original: r
Revised: r

Original: ⇤ i as the sum of features of all edges for the node.
Revised: ⇤ i as the sum of features of all edges for the node.

Original: Numerically solving
Revised: Numerically solving

Original: Eqn.
Revised: Eqn.

Original: (2) using the explicit Euler scheme with step size ⌧ yields the following recursive formulation
Revised: (2) using the explicit Euler scheme with step size ⌧ yields the following recursive formulation

Original: [Equation]
Revised: [Equation]

Original: (4) where G is a diffusivity coefﬁcient matrix in place of c .
Revised: where G is a diffusivity coefﬁcient matrix in place of c .

Original: Moreover, stacking a non-linear transformation layer after each step yields the formulation of Graph
Revised: Moreover, stacking a non-linear transformation layer after each step yields the formulation of Graph

Original: Convolution Networks (GCN) [28] for Eqn.
Revised: Convolution Networks (GCN) [28] for Eqn.

Original: (3), Graph Attention Networks (GAT) [48] with residual connection for Eqn.
Revised: (3), Graph Attention Networks (GAT) [49] with residual connection for Eqn.

Original: (4), and even more GNN architectures by virtue of the ﬂexibility of interpretion for heat equation on graphs.
Revised: (4), and even more GNN architectures by virtue of the ﬂexibility of interpretion for heat equation on graphs.

Original: Heat Kernels.
Revised: Heat Kernels.

Original: Intriguingly, it turns out that the initial value problem of heat equation on any manifold M has a smallest positive fundamental solution depending on the Laplace operator  , known as the heat kernel [2].
Revised: Intriguingly, it turns out that the initial value problem of heat equation on any manifold M has a smallest positive fundamental solution depending on the Laplace operator  , known as the heat kernel [2].

Original: It is denoted as a kernel function  ( x, y, t ) , such that
Revised: It is denoted as a kernel function  ( x, y, t ) , such that

Original: [Equation]
Revised: [Equation]

Original: where µ is a non-negative measure associated with M .
Revised: where µ is a non-negative measure associated with M .

Original: In physics, the heat kernel  ( x, y, t ) can be interpreted as a transition density that describes the asymptotic behavior of a natural Brownian motion on the manifold.
Revised: In physics, the heat kernel  ( x, y, t ) can be interpreted as a transition density that describes the asymptotic behavior of a natural Brownian motion on the manifold.

Original: Its formulation thus can be treated as a unique reﬂection or representation of the geometry of the underlying manifold .
Revised: Its formulation thus can be treated as a unique reﬂection or representation of the geometry of the underlying manifold .

Original: For example, if the manifold is a k -dimensional Euclidean
Revised: For example, if the manifold is a k -dimensional Euclidean

Original: Space R k or a Hyperbolic Space H k , the explicit formula of heat kernel is respectively given by,  (
Revised: Space R k or a Hyperbolic Space H k , the explicit formula of heat kernel is respectively given by,  (

Original: [Equation]
Revised: [Equation]

Original: i ,
Revised: i ,

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: u i ,
Revised: u i ,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: sinh ⇢ @
Revised: sinh ⇢ @

Original: [Equation]
Revised: [Equation]

Original: where ⇢ =
Revised: where ⇢ =

Original: d ( u i , u j ) denote geodesic distance.
Revised: d ( u i , u j ) denote geodesic distance.

Original: Heat kernel has also been adopted for graph-related applications such as community detection [31], graph clustering [56].
Revised: Heat kernel has also been adopted for graph-related applications such as community detection [31], graph clustering [58].

Original:  Extending Heat Kernel to GNNs
Revised:  Extending Heat Kernel to GNNs

Original: The starting point of this work is the development of neural heat kernel , built upon the previouslymentioned connection of GNNs and heat equation.
Revised: The starting point of this work is the development of neural heat kernel , built upon the previouslymentioned connection of GNNs and heat equation.

Original: As will be discussed later, this novel notion lends us a thermodynamic perspective to the intrinsic geometric property of the latent graph manifold embodied in GNNs, and hence paves the way for distilling geometric knowledge.
Revised: As will be discussed later, this novel notion lends us a thermodynamic perspective to the intrinsic geometric property of the latent graph manifold embodied in GNNs, and hence paves the way for distilling geometric knowledge.

Original: 3.1 Neural Heat Kernel
Revised: 3.1 Neural Heat Kernel

Original: Consider the graph signal X ( t ) at time t and node features H ( l ) at layer l as interchangeable notions.
Revised: Consider the graph signal X ( t ) at time t and node features H ( l ) at layer l as interchangeable notions.

Original: Consequently, feature propagation using one layer of GNN amounts to heat diffusion on the base manifold M within a certain time interval ⌧ , leading to the equivalences of X ( t + ⌧ ) and H ( l +1) :
Revised: Consequently, feature propagation using one layer of GNN amounts to heat diffusion on the base manifold M within a certain time interval ⌧ , leading to the equivalences of X ( t + ⌧ ) and H ( l +1) :

Original: [Equation]
Revised: [Equation]

Original: where f ✓ denotes an arbitrary GNN model with parameter ✓ , and  ( f ✓ , G ) denotes a generalization of Laplace-Beltrami operator deﬁned over the base manifold M associated with graph G and the arbitrary backbone GNN model
Revised: where f ✓ denotes an arbitrary GNN model with parameter ✓ , and  ( f ✓ , G ) denotes a generalization of Laplace-Beltrami operator deﬁned over the base manifold M associated with graph G and the arbitrary backbone GNN model

Original: f ✓ .
Revised: f ✓ .

Original: Remark.
Revised: Remark.

Original: The equivalence of two equations in Eqn.
Revised: The equivalence of two equations in Eqn.

Original: 7 is based on the recently established connection between heat equation and GNNs [8; 7; 50; 13; 12], which reveal that the formulation of a GNN layer could be thought of as discretisations (that correspond to the left equation in Eqn. 7) of the continuous diffusion process (that correspond to the right equation in Eqn. 7) described by the heat equation.
Revised: 7 is based on the recently established connection between heat equation and GNNs [8; 7; 51; 13; 12], which reveal that the formulation of a GNN layer could be thought of as discretisations (that correspond to the left equation in Eqn. 7) of the continuous diffusion process (that correspond to the right equation in Eqn. 7) described by the heat equation.

Original: Furthermore, different deﬁnitions of Laplace-Beltrami operator  and schemes for solving Eqn.could yield different GNNs (e.g., SGC [52], GAT [48], GRAND [8]).
Revised: Furthermore, different deﬁnitions of Laplace-Beltrami operator  and schemes for solving Eqn.could yield different GNNs (e.g., SGC [53], GAT [49], GRAND [8]).

Original: While it is unclear whether there exists such a deﬁnition of  for every GNN architecture, we write the operator as  ( f ✓ , G ) to associate it with model f ✓ , and then use the analogy between GNN and heat equation as an analytical tool, in a similar manner with [3;  45;50; 44; 8; 7], for studying the geometry property of GNNs.
Revised: While it is unclear whether there exists such a deﬁnition of  for every GNN architecture, we write the operator as  ( f ✓ , G ) to associate it with model f ✓ , and then use the analogy between GNN and heat equation as an analytical tool, in a similar manner with [3; 46; 51; 45; 8; 7], for studying the geometry property of GNNs.

Original: See more detailed justiﬁcations in Appendix E.
Revised: See more detailed justiﬁcations in Appendix E.

Original: In light of this connection, we consider a natural generalization of heat kernel for GNNs, termed as neural heat kernel (NHK) to highlight its difference with heat kernel in the thermodynamic context.
Revised: In light of this connection, we consider a natural generalization of heat kernel for GNNs, termed as neural heat kernel (NHK) to highlight its difference with heat kernel in the thermodynamic context.

Original: In particular, a single-layer NHK is deﬁned as a positive deﬁnite symmetric kernel function denoted as  ( l ) ✓ ( v i , v j ) , where the sub-script ✓ implies that it is associated with the architecture and parameters of the backbone GNN, and the super-script ( l ) implies that it is speciﬁc to each layer, analogous to the role of continuous time t in Eqn.
Revised: In particular, a single-layer NHK is deﬁned as a positive deﬁnite symmetric kernel function denoted as  ( l ) ✓ ( v i , v j ) , where the sub-script ✓ implies that it is associated with the architecture and parameters of the backbone GNN, and the super-script ( l ) implies that it is speciﬁc to each layer, analogous to the role of continuous time t in Eqn.

Original: Theorem 1.
Revised: Theorem 1.

Original: (Existence of Single-Layer NHK)
Revised: (Existence of Single-Layer NHK)

Original: Suppose two expressions in Eqn.
Revised: Suppose two expressions in Eqn.

Original: (7) are equivalent (see Appendix E for more discussions), then for any graph G and GNN model f ✓ , there exist a unique single-layer NHK function  ( l ) ✓ ( · ) such that for any node v i 2 V and l > 0 ,
Revised: (7) are equivalent (see Appendix E for more discussions), then for any graph G and GNN model f ✓ , there exist a unique single-layer NHK function  ( l ) ✓ ( · ) such that for any node v i 2 V and l > 0 ,

Original: [Equation]
Revised: [Equation]

Original: where h ( l ) i 2 R d denotes the feature of node v i at l -th layer, and µ is a measure over vertices that could be speciﬁed as the inverse of node degree
Revised: where h ( l ) i 2 R d denotes the feature of node v i at l -th layer, and µ is a measure over vertices that could be speciﬁed as the inverse of node degree

Original: 1 /d i .
Revised: 1 /d i .

Original: To push further, we can generalize NHK across multiple layers of GNN, termed as a cross-layer NHK  ✓ ( v i , v j , l 7! l + k ) (e.g., from l -th layer to ( l + k ) -th layer of GNN).
Revised: To push further, we can generalize NHK across multiple layers of GNN, termed as a cross-layer NHK  ✓ ( v i , v j , l 7! l + k ) (e.g., from l -th layer to ( l + k ) -th layer of GNN).

Original: Its existence could be induced recursively by the semi-group identity property of NHK concerning consecutive GNN layers.
Revised: Its existence could be induced recursively by the semi-group identity property of NHK concerning consecutive GNN layers.

Original: Theorem 2.
Revised: Theorem 2.

Original: (Semigroup Identity Property of NHK)
Revised: (Semigroup Identity Property of NHK)

Original: The NHK satisﬁes the semigroup identity property: 8 v i , v j 2 V and l > 0 , there exists a cross-layer NHK across two consecutive layers
Revised: The NHK satisﬁes the semigroup identity property: 8 v i , v j 2 V and l > 0 , there exists a cross-layer NHK across two consecutive layers

Original: [Equation]
Revised: [Equation]

Original: This theorem indicates that stacks of multiple GNN layers also constitute a valid kernel, i.e.,
Revised: This theorem indicates that stacks of multiple GNN layers also constitute a valid kernel, i.e.,

Original: [Equation]
Revised: [Equation]

Original: Analogous to heat kernel as an unique characterization of the underlying space, NHK characterizes the geometric property of the latent graph manifold for GNNs.
Revised: Analogous to heat kernel as an unique characterization of the underlying space, NHK characterizes the geometric property of the latent graph manifold for GNNs.

Original: Additionally, NHK is dependent on GNN models through the deﬁnition of the associated Laplace-Beltrami operator  ( f ✓ , G ) , inheriting the expressiveness of neural networks and varying through the course of training.
Revised: Additionally, NHK is dependent on GNN models through the deﬁnition of the associated Laplace-Beltrami operator  ( f ✓ , G ) , inheriting the expressiveness of neural networks and varying through the course of training.

Original: Intuitively, NHK can be thought of as a model-driven encoding for topological information, encapsulating the geometric knowledge learned by GNNs into a tractable functional form.
Revised: Intuitively, NHK can be thought of as a model-driven encoding for topological information, encapsulating the geometric knowledge learned by GNNs into a tractable functional form.

Original: 3.2 Application in Geometric Distillation
Revised: 3.2 Application in Geometric Distillation

Original: Consider the problem of distilling geometric knowledge, which involves an intelligent teacher model f ✓ ⇤ , which is exposed to and pre-trained over the (relatively) complete graph ˜ G =
Revised: Consider the problem of distilling geometric knowledge, which involves an intelligent teacher model f ✓ ⇤ , which is exposed to and pre-trained over the (relatively) complete graph ˜ G =

Original: (˜ V , ˜ E ) , and a student model f ✓ that is exposed to the partial graph G =
Revised: (˜ V , ˜ E ) , and a student model f ✓ that is exposed to the partial graph G =

Original: ( V , E ) , where V ✓ ˜ V and E ✓ {V ⇥ V} \ ˜ E .
Revised: ( V , E ) , where V ✓ ˜ V and E ✓ {V ⇥ V} \ ˜ E .

Original: Our target is to train a student model (with the help of teacher model) that operates on
Revised: Our target is to train a student model (with the help of teacher model) that operates on

Original: G to be as competitive as models operating on ˜ G during inference.
Revised: G to be as competitive as models operating on ˜ G during inference.

Original: Since G is a sub-graph of ˜ G , they should lie in the same space (i.e., latent manifold) governed by the underlying mechanism of data generation, and hence we expect student and teacher models to capture the same geometric property of this shared space.
Revised: Since G is a sub-graph of ˜ G , they should lie in the same space (i.e., latent manifold) governed by the underlying mechanism of data generation, and hence we expect student and teacher models to capture the same geometric property of this shared space.

Original: This leads to the principle of Geometric Knowledge Distillation (GKD): transfer the geometric knowledge of the intelligent teacher to the student such that the student can propagate features as if it is aware of the complete graph topology (see the example in Fig. 1).
Revised: This leads to the principle of Geometric Knowledge Distillation (GKD): transfer the geometric knowledge of the intelligent teacher to the student such that the student can propagate features as if it is aware of the complete graph topology (see the example in Fig. 1).

Original: To this end, we resort to NHK matrices on the teacher (resp.
Revised: To this end, we resort to NHK matrices on the teacher (resp.

Original: student) model over the complete (resp. partial) graph as instantiations of their geometric knowledge, denoted as
Revised: student) model over the complete (resp. partial) graph as instantiations of their geometric knowledge, denoted as

Original: [Equation]
Revised: [Equation]

Original: written compactly as K ( l +1) ( G ) when k = 1 .
Revised: written compactly as K ( l +1) ( G ) when k = 1 .

Original: The NHK matrix is a positive semi-deﬁnite symmetric matrix, and alike  , is dependent on the GNN model f ✓ and graph G .
Revised: The NHK matrix is a positive semi-deﬁnite symmetric matrix, and alike  , is dependent on the GNN model f ✓ and graph G .

Original: Denote K ( l ) ✓ ⇤ , V ( ˜ G ) 2 R |V| ⇥ |V| as the sub-matrix of K ( l ) ✓ ⇤ ( ˜ G ) with row and column indices in V .
Revised: Denote K ( l ) ✓ ⇤ , V ( ˜ G ) 2 R |V| ⇥ |V| as the sub-matrix of K ( l ) ✓ ⇤ ( ˜ G ) with row and column indices in V .

Original: The distillation loss for GKD is
Revised: The distillation loss for GKD is

Original: [Equation]
Revised: [Equation]

Original: where d( · , · ) is a similarity measure, for which we choose Frobenius distance as implementation, i.e.,
Revised: where d( · , · ) is a similarity measure, for which we choose Frobenius distance as implementation, i.e.,

Original: [Equation]
Revised: [Equation]

Original: where W 2 R |V| ⇥ |V| is a weighting matrix to trade-off distillation loss with respect to different node pairs depending on their connectivity.
Revised: where W 2 R |V| ⇥ |V| is a weighting matrix to trade-off distillation loss with respect to different node pairs depending on their connectivity.

Original: For k = 1 , the loss can be re-written as L ( l +1) dis
Revised: For k = 1 , the loss can be re-written as L ( l +1) dis

Original: ( K ✓ ⇤ , V , K ✓ ) .
Revised: ( K ✓ ⇤ , V , K ✓ ) .

Original: Note that one can also specify different k for teacher and student models in Eqn.
Revised: Note that one can also specify different k for teacher and student models in Eqn.

Original: (11) in case when the teacher model is deeper.
Revised: (11) in case when the teacher model is deeper.

Original:  Instantiations for Geometric Knowledge Distillation
Revised:  Instantiations for Geometric Knowledge Distillation

Original: Unfortunately, deriving explicit formulas for NHKs is prohibitively challenging due to introduction of non-linearity.
Revised: Unfortunately, deriving explicit formulas for NHKs is prohibitively challenging due to introduction of non-linearity.

Original: To circumvent it, we propose two types of instantiations for GKD, i.e., non-parametric and parametric.
Revised: To circumvent it, we propose two types of instantiations for GKD, i.e., non-parametric and parametric.

Original: The former considers explicit NHKs by making assumptions on the underlying space, and the latter learns NHK in a data-driven manner.
Revised: The former considers explicit NHKs by making assumptions on the underlying space, and the latter learns NHK in a data-driven manner.

Original: 4.1 Non-Parametric Geometric Distillation
Revised: 4.1 Non-Parametric Geometric Distillation

Original: Deterministic Kernel.
Revised: Deterministic Kernel.

Original: One instantiation of NHK is a Gauss-Weierstrass kernel in the form of
Revised: One instantiation of NHK is a Gauss-Weierstrass kernel in the form of

Original: Eqn.
Revised: Eqn.

Original: (6), by assuming the underlying space is a Euclidean space.
Revised: (6), by assuming the underlying space is a Euclidean space.

Original: Since the distillation loss in
Revised: Since the distillation loss in

Original: Eqn.
Revised: Eqn.

Original: (11) is a homogeneous function, we can remove its scaling factor and deﬁne NHK as
Revised: (11) is a homogeneous function, we can remove its scaling factor and deﬁne NHK as

Original: [Equation]
Revised: [Equation]

Original: where T denotes the estimation of the accumulated time interval.
Revised: where T denotes the estimation of the accumulated time interval.

Original: Alternatively, we can use Sigmoid kernel and deﬁne non-parametric NHK as:
Revised: Alternatively, we can use Sigmoid kernel and deﬁne non-parametric NHK as:

Original: [Equation]
Revised: [Equation]

Original: where a, b are positive constants depending on l and k .
Revised: where a, b are positive constants depending on l and k .

Original: It is a natural and intuitive choice as similarity measurement and empirically found as-well effective, albeit does not correspond to any named manifold to our knowledge.
Revised: It is a natural and intuitive choice as similarity measurement and empirically found as-well effective, albeit does not correspond to any named manifold to our knowledge.

Original: Randomized Kernel.
Revised: Randomized Kernel.

Original: We can also deﬁne Randomized kernel based on the following theorem.
Revised: We can also deﬁne Randomized kernel based on the following theorem.

Original: Theorem 3.
Revised: Theorem 3.

Original: (Expansion of NHK) Let { ' k 0 } 1 k 0 =
Revised: (Expansion of NHK) Let { ' k 0 } 1 k 0 =

Original: be orthonormal basis of eigenfunctions of   ( f ✓ , G ) with eigenvalues 0 <  0   1   2  . .
Revised: be orthonormal basis of eigenfunctions of   ( f ✓ , G ) with eigenvalues 0 <  0   1   2  . .

Original: . , NHK allows the expansion:
Revised: . , NHK allows the expansion:

Original: [Equation]
Revised: [Equation]

Original: Based on this result, we resort to the approximation of NHK by deﬁning a randomized kernel in a similar form as Eqn.
Revised: Based on this result, we resort to the approximation of NHK by deﬁning a randomized kernel in a similar form as Eqn.

Original: (15), leading to the following formulation of randomized NHK:
Revised: (15), leading to the following formulation of randomized NHK:

Original: [Equation]
Revised: [Equation]

Original: where  ( W k 0 h i ) is used to proximate ' k 0 ( v i ) , W k 

Revised: where  ( W k 0 h i ) is used to proximate ' k 0 ( v i ) , W k

Original: =
Revised: =

Original: [  1 ,k 0 ,  2 ,k 0 , · · · ,  s,k 0 ] > is a transformation matrix,  ⇠ N ( 0 , I d ) is a d -dimensional random variable from Gaussian distribution.
Revised: [  1 ,k 0 ,  2 ,k 0 , · · · ,  s,k 0 ] > is a transformation matrix,  ⇠ N ( 0 , I d ) is a d -dimensional random variable from Gaussian distribution.

Original: In fact, under certain choice of activation function  , Eqn.
Revised: In fact, under certain choice of activation function  , Eqn.

Original: (16) could approximate a diversity of kernels [40; 10].
Revised: (16) could approximate a diversity of kernels [41; 10].

Original: This design essentially enforces the alignment of teacher and student for arbitrary underlying manifold.
Revised: This design essentially enforces the alignment of teacher and student for arbitrary underlying manifold.

Original: Training Scheme.
Revised: Training Scheme.

Original: We follow the standard training paradigm in KD literature [22; 17]: the teacher is pre-trained by a supervised prediction loss involving all labeled nodes in ˜ V .
Revised: We follow the standard training paradigm in KD literature [22; 17]: the teacher is pre-trained by a supervised prediction loss involving all labeled nodes in ˜ V .

Original: After teacher is well-trained, we ﬁx ✓ ⇤ and train the student model according to
Revised: After teacher is well-trained, we ﬁx ✓ ⇤ and train the student model according to

Original: [Equation]
Revised: [Equation]

Original: where Y denotes ground-truth labels of labeled nodes in V , and ˆ Y ✓ denotes the predictions of student model f ✓ on G , L dis is the distillation loss deﬁned by Eqn.
Revised: where Y denotes ground-truth labels of labeled nodes in V , and ˆ Y ✓ denotes the predictions of student model f ✓ on G , L dis is the distillation loss deﬁned by Eqn.

Original: (11), L denotes the total number of layers.
Revised: (11), L denotes the total number of layers.

Original: 4.2 Parametric Geometric Distillation
Revised: 4.2 Parametric Geometric Distillation

Original: Inheriting the similar spirit of auto-encoding Bayes [27], we introduce a variational inverse-NHK that is independently parameterized, denoted as  †  , whose existence is guaranteed by the invertibility of
Revised: Inheriting the similar spirit of auto-encoding Bayes [27], we introduce a variational inverse-NHK that is independently parameterized, denoted as  †  , whose existence is guaranteed by the invertibility of

Original: NHK matrices.
Revised: NHK matrices.

Original: Together with  ✓ , they deﬁne a symmetric form characterizing feature propagation:
Revised: Together with  ✓ , they deﬁne a symmetric form characterizing feature propagation:

Original: [Equation]
Revised: [Equation]

Original: In practice, we follow existing kernel learning approaches [51] and parameterize the inverse-NHK as
Revised: In practice, we follow existing kernel learning approaches [52] and parameterize the inverse-NHK as

Original: [Equation]
Revised: [Equation]

Original: where g  : R d !
Revised: where g  : R d !

Original: R s is the associated learnable non-linear mapping.
Revised: R s is the associated learnable non-linear mapping.

Original: Given a pre-trained teacher model, distilling geometric knowledge boils down to 1) establishing equivalence of Eqn.
Revised: Given a pre-trained teacher model, distilling geometric knowledge boils down to 1) establishing equivalence of Eqn.

Original: (18) and
Revised: (18) and

Original: Eqn. (19), and 2) matching pseudo-inverse NHK matrices for teacher and student models (respectively denoted as K † ✓ ⇤ , V and K † ✓ with clear meanings), leading to the training scheme as follows.
Revised: Eqn. (19), and 2) matching pseudo-inverse NHK matrices for teacher and student models (respectively denoted as K † ✓ ⇤ , V and K † ✓ with clear meanings), leading to the training scheme as follows.

Original: Training Scheme.
Revised: Training Scheme.

Original: Based on Eqn.
Revised: Based on Eqn.

Original: (19), we can deﬁne a reconstruction loss with respect to the teacher model (similar applies to the student model) as
Revised: (19), we can deﬁne a reconstruction loss with respect to the teacher model (similar applies to the student model) as

Original: [Equation]
Revised: [Equation]

Original: Then, minimizing the reconstruction loss with ﬁxed GNN model parameter ✓ amounts to optimizing the variational parameter  , and minimizing prediction and distillation losses given ﬁxed  amounts to optimizing the student model parameter ✓ :
Revised: Then, minimizing the reconstruction loss with ﬁxed GNN model parameter ✓ amounts to optimizing the variational parameter  , and minimizing prediction and distillation losses given ﬁxed  amounts to optimizing the student model parameter ✓ :

Original: [Equation]
Revised: [Equation]

Original: Applying two steps iteratively adds up to an EM-like algorithm for training the student model.
Revised: Applying two steps iteratively adds up to an EM-like algorithm for training the student model.

Original: In practice, we set l + k as the last layer, and l as the ﬁrst layer to use as much information as possible.
Revised: In practice, we set l + k as the last layer, and l as the ﬁrst layer to use as much information as possible.

Original: We justify the parametric GKD approach in Appendix.
Revised: We justify the parametric GKD approach in Appendix.

Original: D by showing it essentially explores the true
Revised: D by showing it essentially explores the true

Original: NHK behind GNN.
Revised: NHK behind GNN.

Original:  Experiments
Revised:  Experiments

Original: We conduct experiments to validate the efﬁcacy of our method on graph-structured data in terms of various types of privileged geometric knowledge, combinations of teacher-student GNN architectures and potential application scenarios.
Revised: We conduct experiments to validate the efﬁcacy of our method on graph-structured data in terms of various types of privileged geometric knowledge, combinations of teacher-student GNN architectures and potential application scenarios.

Original: We use three benchmark datasets Cora [34], Citeseer [43],
Revised: We use three benchmark datasets Cora [35], Citeseer [44],

Original: Pubmed [38], and a larger dataset OGBN-Arxiv [23] for node classiﬁcation tasks.
Revised: Pubmed [39], and a larger dataset OGBN-Arxiv [23] for node classiﬁcation tasks.

Original: More implementation details and experimental results are deferred to Appendix.
Revised: More implementation details and experimental results are deferred to Appendix.

Original: The codes are available at https://github.com/chr26195/GKD .
Revised: The codes are available at https://github.com/chr26195/GKD .

Original: Implementation and Competitors.
Revised: Implementation and Competitors.

Original: We consider the following variants of the proposed GKD.
Revised: We consider the following variants of the proposed GKD.

Original: 1) GKD-G : non-parametric Gaussian NHK; 2) GKD-S : non-parametric Sigmoid NHK; 3) GKDR : randomized NHK; 3) PGKD : parametric NHK.
Revised: 1) GKD-G : non-parametric Gaussian NHK; 2) GKD-S : non-parametric Sigmoid NHK; 3) GKDR : randomized NHK; 3) PGKD : parametric NHK.

Original: We choose KD methods that is representative in its own category for comparison, including KD [22], FSP [62], LSP [61].
Revised: We choose KD methods that is representative in its own category for comparison, including KD [22], FSP [64], LSP [63].

Original: We also report the performances of teacher and student model trained with the standard classiﬁcation loss, short as
Revised: We also report the performances of teacher and student model trained with the standard classiﬁcation loss, short as

Original: Teacher and Student .
Revised: Teacher and Student .

Original: The teacher model is trained using the complete graph ˜ G , and, to calibrate with all other methods, tested using the partial graph G .
Revised: The teacher model is trained using the complete graph ˜ G , and, to calibrate with all other methods, tested using the partial graph G .

Original: Besides, we consider an Oracle model which is both trained and tested on ˜ G , which naturally takes an advantaged place given more information during inference.
Revised: Besides, we consider an Oracle model which is both trained and tested on ˜ G , which naturally takes an advantaged place given more information during inference.

Original: Since our method is compatible with the vanilla KD paradigm [22], we report the performance delivered by their combinations (i.e., GKD+KD and PGKD+KD).
Revised: Since our method is compatible with the vanilla KD paradigm [22], we report the performance delivered by their combinations (i.e., GKD+KD and PGKD+KD).

Original: Experiment Settings.
Revised: Experiment Settings.

Original: We investigate on various experimental settings according to different types of privileged geometric knowledge.
Revised: We investigate on various experimental settings according to different types of privileged geometric knowledge.

Original: In the case of edge-aware geometric knowledge , the teacher model has access to additional edge information, i.e., E ⇢ ˜ E and V = ˜ V .
Revised: In the case of edge-aware geometric knowledge , the teacher model has access to additional edge information, i.e., E ⇢ ˜ E and V = ˜ V .

Original: In the case of node-aware geometric knowledge , the teacher model has access to additional node information, i.e., V ⇢ ˜ V and
Revised: In the case of node-aware geometric knowledge , the teacher model has access to additional node information, i.e., V ⇢ ˜ V and

Original: E =
Revised: E =

Original: ˜
Revised: ˜

Original: E \ {V ⇥ V} .
Revised: E \ {V ⇥ V} .

Original: We also consider other conventional KD settings including model compression, self-distillation and online distillation, which will be illustrated in detail.
Revised: We also consider other conventional KD settings including model compression, self-distillation and online distillation, which will be illustrated in detail.

Original: The backbone f ✓ is set as 3 -layer GCN [28] for both student and teacher models, unless otherwise stated.
Revised: The backbone f ✓ is set as 3 -layer GCN [28] for both student and teacher models, unless otherwise stated.

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 5.1 Main Results
Revised: 5.1 Main Results

Original: Edge-Aware Geometric Knowledge.
Revised: Edge-Aware Geometric Knowledge.

Original: We report results for the edge-aware geometric knowledge setting.
Revised: We report results for the edge-aware geometric knowledge setting.

Original: To quantify the privileged information, we set the quantity ( | ˜ E|  |E| ) / | ˜ E| , called privileged information ratio (PIR), as 0 .
Revised: To quantify the privileged information, we set the quantity ( | ˜ E|  |E| ) / | ˜ E| , called privileged information ratio (PIR), as 0 .

Original: As shown in Tab. 1, all variants of GKD outperform other KD baselines on both datasets, and signiﬁcantly exceeds both Student and Teacher models.
Revised: As shown in Tab. 1, all variants of GKD outperform other KD baselines on both datasets, and signiﬁcantly exceeds both Student and Teacher models.

Original: Further, GKD and its variants rival, if not surpass, the Oracle model.
Revised: Further, GKD and its variants rival, if not surpass, the Oracle model.

Original: In other words, the student model trained using GKD could use far less graph topological information to achieve very close performance to competitors that are aware of the full graph topology during inference.
Revised: In other words, the student model trained using GKD could use far less graph topological information to achieve very close performance to competitors that are aware of the full graph topology during inference.

Original: Furthermore, the parametric
Revised: Furthermore, the parametric

Original: PGKD performs better than its non-parametric counterparts in most cases, and GKD-R is the most effective non-parametric method in general.
Revised: PGKD performs better than its non-parametric counterparts in most cases, and GKD-R is the most effective non-parametric method in general.

Original: Despite that, GKD-G and GKD-S are also effective while being simpler.
Revised: Despite that, GKD-G and GKD-S are also effective while being simpler.

Original: We presume that the performance variation of different GKD realizations stem from the different geometric property governed by the feature of datasets.
Revised: We presume that the performance variation of different GKD realizations stem from the different geometric property governed by the feature of datasets.

Original: Node-Aware Geometric Knowledge.
Revised: Node-Aware Geometric Knowledge.

Original: We further investigate on the node-aware geometric knowledge setting where the teacher model has access to more labeled nodes and their relations with the rest nodes.
Revised: We further investigate on the node-aware geometric knowledge setting where the teacher model has access to more labeled nodes and their relations with the rest nodes.

Original: We set the PIR w.r.t.
Revised: We set the PIR w.r.t.

Original: labeled nodes, deﬁned as ( | ˜ V train |  |V train | ) / |
Revised: labeled nodes, deﬁned as ( | ˜ V train |  |V train | ) / |

Original: ˜ V train | , to 0 .
Revised: ˜ V train | , to 0 .

Original: A unique challenge of this setting compared to the edge-ware counterpart is that, apart from graph topological information, the student model has less labeled training samples.
Revised: A unique challenge of this setting compared to the edge-ware counterpart is that, apart from graph topological information, the student model has less labeled training samples.

Original: As we can see from
Revised: As we can see from

Original: Tab. 2.
Revised: Tab. 2.

Original: The proposed GKD and its variants again consistently outperform KD baselines throughout all the cases, surpasses both Student and Teacher models, and are even as competitive as Oracle.
Revised: The proposed GKD and its variants again consistently outperform KD baselines throughout all the cases, surpasses both Student and Teacher models, and are even as competitive as Oracle.

Original: Larger Dataset.
Revised: Larger Dataset.

Original: Table 3 presents results on a large graph, i.e., OGBN-Arxiv.
Revised: Table 3 presents results on a large graph, i.e., OGBN-Arxiv.

Original: We use the same
Revised: We use the same

Original: PIR setting as citation networks, and choose the best variant of GKD to report in the table.
Revised: PIR setting as citation networks, and choose the best variant of GKD to report in the table.

Original: Since the space complexity for GKD is O ( n 2 ) , we randomly draw a mini-batch of nodes for computing the distillation loss in practice.
Revised: Since the space complexity for GKD is O ( n 2 ) , we randomly draw a mini-batch of nodes for computing the distillation loss in practice.

Original: Note that the loss function is unbiased as long as all nodes are evenly covered.
Revised: Note that the loss function is unbiased as long as all nodes are evenly covered.

Original: It could also be seamlessly integrated with the original mini-batch method (by sampling ego-graphs) used for training large graphs without further modiﬁcation.
Revised: It could also be seamlessly integrated with the original mini-batch method (by sampling ego-graphs) used for training large graphs without further modiﬁcation.

Original: Again, we found our methods consistently outperform Teacher and Student models, and are close to the performance of the Oracle model, which suggests the effectiveness of GKD in large graphs.
Revised: Again, we found our methods consistently outperform Teacher and Student models, and are close to the performance of the Oracle model, which suggests the effectiveness of GKD in large graphs.

Original: Performance Variation with Privileged Ratio.
Revised: Performance Variation with Privileged Ratio.

Original: The results with respect to varying privileged (edge-aware and node-ware) information ratio are given respectively in Fig.
Revised: The results with respect to varying privileged (edge-aware and node-ware) information ratio are given respectively in Fig.

Original: 2 and Fig.
Revised: 2 and Fig.

Original: The performance of Oracle model is invariant as it is trained and tested on the same (complete) graph.
Revised: The performance of Oracle model is invariant as it is trained and tested on the same (complete) graph.

Original: In general, for Teacher model, Student model, and vanilla KD, their performance drops with increasing
Revised: In general, for Teacher model, Student model, and vanilla KD, their performance drops with increasing

Original: PIR quantifying the information loss.
Revised: PIR quantifying the information loss.

Original: In contrast, our method is signiﬁcantly more robust, only showing slight performance deterioration, exceeding the Teacher model, and approaching the Oracle model.
Revised: In contrast, our method is signiﬁcantly more robust, only showing slight performance deterioration, exceeding the Teacher model, and approaching the Oracle model.

Original: Besides, we ﬁnd an interesting phenomenon that in the edge-aware setting on Pubmed dataset, the performance of Teacher model is the worst, presumably because that the Teacher is trained using
Revised: Besides, we ﬁnd an interesting phenomenon that in the edge-aware setting on Pubmed dataset, the performance of Teacher model is the worst, presumably because that the Teacher is trained using

Original: [Table]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table] [Figure] [Figure] [Table]
Revised: fully observed graph

Original: [Figure]
Revised: [Figure]

Original: fully observed graph, and may perform poorly once the privileged part of graph topology becomes unavailable at test time.
Revised: and may perform poorly once the privileged part of graph topology becomes unavailable at test time.

Original: 5.2 Scalability Test
Revised: 5.2 Scalability Test

Original: Training Time.
Revised: Training Time.

Original: We report the the training time per epoch of different methods in the left three panels of Fig.
Revised: We report the the training time per epoch of different methods in the left three panels of Fig.

Original: 4, where we use the whole graph for training on Cora and Citeseer, and set the batch size as 5 , 000 for Pubmed.
Revised: 4, where we use the whole graph for training on Cora and Citeseer, and set the batch size as 5 , 000 for Pubmed.

Original: While the computation complexity for distillation loss of GKD (in both non-parametric and parametric cases) is O ( dn 2 ) , we ﬁnd that in practice their training time is not signiﬁcantly worse than other KD baselines.
Revised: While the computation complexity for distillation loss of GKD (in both non-parametric and parametric cases) is O ( dn 2 ) , we ﬁnd that in practice their training time is not signiﬁcantly worse than other KD baselines.

Original: In speciﬁc, all variants of GKD take less than two times of the training time of vanilla KD while yielding better performance, and the simplest GKD variant using Sigmoid NHK is even more efﬁcient than baselines in some cases.
Revised: In speciﬁc, all variants of GKD take less than two times of the training time of vanilla KD while yielding better performance, and the simplest GKD variant using Sigmoid NHK is even more efﬁcient than baselines in some cases.

Original: It is also worth mentioning that the distilled student models are equally efﬁcient by using sparser graph structure (in edge-aware setting) or smaller graph (in node-aware setting), which are suitable for being deployed in latency-constrained scenarios.
Revised: It is also worth mentioning that the distilled student models are equally efﬁcient by using sparser graph structure (in edge-aware setting) or smaller graph (in node-aware setting), which are suitable for being deployed in latency-constrained scenarios.

Original: Convergence Speed.
Revised: Convergence Speed.

Original: We also compare the convergence speed of different variants of GKD and other methods in the right panel of Fig.
Revised: We also compare the convergence speed of different variants of GKD and other methods in the right panel of Fig.

Original: 4, where we use the same model architecture and optimizer for different methods and ﬁnetune other hyper-parameters to ensure fair comparison.
Revised: 4, where we use the same model architecture and optimizer for different methods and ﬁnetune other hyper-parameters to ensure fair comparison.

Original: Since the loss function is different for different methods, we report the convergence of training accuracy.
Revised: Since the loss function is different for different methods, we report the convergence of training accuracy.

Original: As shown by the ﬁgure, GKD variants using Sigmoid and Gaussian NHKs can converge as fast as the vanilla
Revised: As shown by the ﬁgure, GKD variants using Sigmoid and Gaussian NHKs can converge as fast as the vanilla

Original: KD.
Revised: KD.

Original: While GKD-R and PGKD in general take more epochs to converge, they can converge within 200 epochs (which is the default setting recommended in [28]) and the best validation accuracy is achieved within the ﬁrst 100 epochs.
Revised: While GKD-R and PGKD in general take more epochs to converge, they can converge within 200 epochs (which is the default setting recommended in [28]) and the best validation accuracy is achieved within the ﬁrst 100 epochs.

Original: We also ﬁnd that the EM-style algorithm used for training
Revised: We also ﬁnd that the EM-style algorithm used for training

Original: PGKD does not cost too much extra epochs for convergence than the non-parametric GKD which is reasonable since the learnable mapping g  used in Eqn.
Revised: PGKD does not cost too much extra epochs for convergence than the non-parametric GKD which is reasonable since the learnable mapping g  used in Eqn.

Original: (22) is an independent module that is not involved in GNN’s feed-forward computation.
Revised: (22) is an independent module that is not involved in GNN’s feed-forward computation.

Original: [Table]
Revised: [Table]

Original: 5.3 Other Settings
Revised: 5.3 Other Settings

Original: [Table]
Revised: [Table]

Original: Model Compression.
Revised: Model Compression.

Original: We report the performance of GKD in the conventional model compression setting [22] where the teacher and student are different-sized
Revised: We report the performance of GKD in the conventional model compression setting [22] where the teacher and student are different-sized

Original: GNN models.
Revised: GNN models.

Original: In this setting, we train and test both teacher and student models on the complete graph.
Revised: In this setting, we train and test both teacher and student models on the complete graph.

Original: We use GCN as the backbone for teacher with a relatively large hidden size 64 , and use SGC
Revised: We use GCN as the backbone for teacher with a relatively large hidden size 64 , and use SGC

Original: or GCN (with hidden size 8 and 16 ) as the student model
Revised: or GCN (with hidden size 8 and 16 ) as the student model

Original: We report
Revised: We report

Original: the best
Revised: the best

Original: result among
Revised: result among

Original: all variants of GKD.
Revised: all variants of GKD.

Original: As shown in the ﬁrst section of Tab. 5, our
Revised: As shown in the ﬁrst section of Tab. 5, our

Original: method
Revised: method

Original: achieves notable
Revised: achieves notable

Original: im
Revised: im

Original: - provements over the student model, rendering it as a useful KD approach for model compression
Revised: - provements over the student model, rendering it as a useful KD approach for model compression

Original: Self-Distillation.
Revised: Self-Distillation.

Original: We further report results for the setting of self-distillation [15; 36], which is a special case of KD when the teacher’s and student’s architectures are identical, often used for reﬁning their own performance.
Revised: We further report results for the setting of self-distillation [15; 37], which is a special case of KD when the teacher’s and student’s architectures are identical, often used for reﬁning their own performance.

Original: The results as shown in Tab.
Revised: The results as shown in Tab.

Original: 5 validate that GKD could also be used to effectively boost GNN’s own performance.
Revised: 5 validate that GKD could also be used to effectively boost GNN’s own performance.

Original: Online Distillation.
Revised: Online Distillation.

Original: Table.
Revised: Table.

Original: 4 shows the performance of different variants of GKD for online distillation where both the teacher model and the student model are trained in an end-to-end manner, in contrast to ofﬂine distillation where the teacher model is pre-trained.
Revised: 4 shows the performance of different variants of GKD for online distillation where both the teacher model and the student model are trained in an end-to-end manner, in contrast to ofﬂine distillation where the teacher model is pre-trained.

Original: The results demonstrate the potential usage of GKD in this setting.
Revised: The results demonstrate the potential usage of GKD in this setting.

Original:  Conclusion, Current Limitations and Future Works
Revised:  Conclusion, Current Limitations and Future Works

Original: This paper formalizes the problem of graph topological knowledge transfer for GNNs.
Revised: This paper formalizes the problem of graph topological knowledge transfer for GNNs.

Original: We investigate on the implication of heat kernel in GNNs and propose the novel notion of neural heat kernel.
Revised: We investigate on the implication of heat kernel in GNNs and propose the novel notion of neural heat kernel.

Original: We leverage it to characterize the geometric property of the underlying manifold for graphs, and propose the framework of geometric knowledge distillation to transfer geometric knowledge from a teacher
Revised: We leverage it to characterize the geometric property of the underlying manifold for graphs, and propose the framework of geometric knowledge distillation to transfer geometric knowledge from a teacher

Original: GNN model to a student GNN.
Revised: GNN model to a student GNN.

Original: Experimental results validate the effectiveness of our approach in various practical settings.
Revised: Experimental results validate the effectiveness of our approach in various practical settings.

Original: Despite that the proposed GKD is effective in various tasks and possesses decent training efﬁciency in practice, its theoretical space and time complexities are O ( n 2 ) and O ( dn 2 ) and the parametric instantiation may take some extra time to converge compared to the pure non-KD counterpart.
Revised: Despite that the proposed GKD is effective in various tasks and possesses decent training efﬁciency in practice, its theoretical space and time complexities are O ( n 2 ) and O ( dn 2 ) and the parametric instantiation may take some extra time to converge compared to the pure non-KD counterpart.

Original: The algorithmic complexity can be reduced by using mini-batch training, and there also exist ways to reduce the overhead such as pre-computing teacher’s NHK matrix or using low-rank approximation.
Revised: The algorithmic complexity can be reduced by using mini-batch training, and there also exist ways to reduce the overhead such as pre-computing teacher’s NHK matrix or using low-rank approximation.

Original: Finally, we do not foresee any direct negative societal impacts of this work.
Revised: Finally, we do not foresee any direct negative societal impacts of this work.

Original: Acknowledgement
Revised: Acknowledgement

Original: This work was partly supported by National Key Research and Development Program of China (2020AAA0107600), National Natural Science Foundation of China (61972250, 72061127003), and
Revised: This work was partly supported by National Key Research and Development Program of China (2020AAA0107600), National Natural Science Foundation of China (61972250, 72061127003), and

Original: Shanghai Municipal Science and Technology (Major) Project (22511105100, 2021SHZDZX0102).
Revised: Shanghai Municipal Science and Technology (Major) Project (22511105100, 2021SHZDZX0102).