Original: Geometric Knowledge Distillation: Topology Compression for Graph Neural Networks
Revised: Geometric Knowledge Distillation: Topology Compression for Graph Neural Networks

Original: Abstract
Revised: Abstract

Original: We study a new paradigm of knowledge transfer that aims at encoding graph topological information into graph neural networks (GNNs) by distilling knowledge from a teacher GNN model trained on a complete graph to a student GNN model operating on a smaller or sparser graph.
Revised: We study a new paradigm of knowledge transfer that aims at encoding graph topological information into graph neural networks (GNNs) by distilling knowledge from a teacher GNN model trained on a complete graph to a student GNN model operating on a smaller or sparser graph.

Original: To this end, we revisit the connection between thermodynamics and the behavior of GNN, based on which we propose
Revised: To this end, we revisit the connection between thermodynamics and the behavior of GNN, based on which we propose

Original: Neural Heat Kernel (NHK) to encapsulate the geometric property of the underlying manifold concerning the architecture of GNNs.
Revised: Neural Heat Kernel (NHK) to encapsulate the geometric property of the underlying manifold concerning the architecture of GNNs.

Original: A fundamental and principled solution is derived by aligning NHKs on teacher and student models, dubbed as
Revised: A fundamental and principled solution is derived by aligning NHKs on teacher and student models, dubbed as

Original: Geometric Knowledge Distillation.
Revised: Geometric Knowledge Distillation.

Original: We develop non- and parametric instantiations and demonstrate their efficacy in various experimental settings for knowledge distillation regarding different types of privileged topological information and teacher-student schemes.
Revised: We develop non- and parametric instantiations and demonstrate their efficacy in various experimental settings for knowledge distillation regarding different types of privileged topological information and teacher-student schemes.

Original:  Introduction
Revised:  Introduction

Original: Modern graph neural networks (GNNs) [27, 46, 50] have shown remarkable performance in learning representations for structured instances.
Revised: Modern graph neural networks (GNNs) [28, 48, 52] have shown remarkable performance in learning representations for structured instances.

Original: From the perspective of geometric deep learning [5, 4, 36], part of the achievement of GNNs can be attributed to their implementation of the permutation invariance property as geometric priors 2 into the architecture design.
Revised: From the perspective of geometric deep learning [5, 4, 37], part of the achievement of GNNs can be attributed to their implementation of the permutation invariance property as geometric priors 2 into the architecture design.

Original: Nevertheless, in practice, GNNs highly rely on graph topology, as essential input information, to explore the relational knowledge implicit in interactions of instance pairs throughout the entire message passing process, termed as geometric knowledge in this paper.
Revised: Nevertheless, in practice, GNNs highly rely on graph topology, as essential input information, to explore the relational knowledge implicit in interactions of instance pairs throughout the entire message passing process, termed as geometric knowledge in this paper.

Original: As advances in generalized distillation [31, 44] reveal the possibility of encoding input features into model construction, natural questions arise as to:
Revised: As advances in generalized distillation [32, 46] reveal the possibility of encoding input features into model construction, natural questions arise as to:

Original: Is it possible and if so, how can we encode graph topology as a special type of ‘geometric prior’ into a GNN model, such that the model could precisely capture the underlying geometric knowledge even without full graph topology as input?
Revised: Is it possible and if so, how can we encode graph topology as a special type of ‘geometric prior’ into a GNN model, such that the model could precisely capture the underlying geometric knowledge even without full graph topology as input?

Original: In specific, we are interested in the following geometric knowledge transfer problem: a GNN model (with node-specific outputs for node-level prediction [23]) is exposed with a partial graph, which is a subset of the complete graph.
Revised: In specific we are interested in the following geometric knowledge transfer problem: a GNN model (with node-specific outputs for node-level prediction [23]) is exposed with a partial graph, which is a subset of the complete graph.

Original: Formally speaking, we have notations:
Revised: Formally speaking, we have notations:

Original: [Equation]
Revised: [Equation]

Original: Our goal is to transfer or encode geometric knowledge extracted from ˜ G to the target GNN model that is only aware of G .
Revised: Our goal is to transfer or encode geometric knowledge extracted from ˜ G to the target GNN model that is only aware of G .

Original: Studying this problem is also of much practical value.
Revised: Studying this problem is also of much practical value.

Original: As a non-exhaustive
Revised: As a non-exhaustive

Original: A B
Revised: A B

Original: H P O
Revised: H P O

Original: M R V
Revised: M R V

Original: W
Revised: W

Original: N
Revised: N

Original: C
Revised: C

Original: D E
Revised: D E

Original: L
Revised: L

Original: G
Revised: G

Original: U S F
Revised: U S F

Original: Y X T Q
Revised: Y X T Q

Original: I J
Revised: I J

Original: K
Revised: K

Original: A
Revised: A

Original: O
Revised: O

Original: M
Revised: M

Original: R V
Revised: R V

Original: W
Revised: W

Original: N
Revised: N

Original: D E
Revised: D E

Original: L
Revised: L

Original: G
Revised: G

Original: U S F
Revised: U S F

Original: A B
Revised: A B

Original: H P O
Revised: H P O

Original: M R V
Revised: M R V

Original: W
Revised: W

Original: N
Revised: N

Original: C
Revised: C

Original: D E
Revised: D E

Original: L
Revised: L

Original: G
Revised: G

Original: U S F
Revised: U S F

Original: T Q
Revised: T Q

Original: I J
Revised: I J

Original: K
Revised: K

Original: [Figure]
Revised: [Figure]

Original: list of applications: improving efficiency without compromising on effectiveness for coarsened graphs [14, 24, 64], privacy constrained scenarios in social recommenders or federated learning where the complete graph is unavailable [32, 47, 63], promoting concentration on targeted community to bring up economic benefits [54].
Revised: list of applications: improving efficiency without compromising on effectiveness for coarsened graphs [14, 24, 66], privacy constrained scenarios in social recommenders or federated learning where the complete graph is unavailable [33, 49, 65], promoting concentration on targeted community to bring up economic benefits [55].

Original: Achieving this target is non-trivial for that we need to first find a principled and fundamental way to encapsulate the geometric knowledge extracted by GNN model, which requires in-depth investigation on the role of graph topology throughout the progressive process of message passing.
Revised: Achieving this target is non-trivial for that we need to first find a principled and fundamental way to encapsulate the geometric knowledge extracted by GNN model, which requires in-depth investigation on the role of graph topology throughout the progressive process of message passing.

Original: Therefore, we take a thermodynamic view borrowed from physics and propose a new methodology built upon recent advances revealing the connection between heat diffusion and architectures of GNNs [8, 48, 7].
Revised: Therefore, we take a thermodynamic view borrowed from physics and propose a new methodology built upon recent advances revealing the connection between heat diffusion and architectures of GNNs [8, 50, 7].

Original: Specifically, we interpret feature propagation as heat flows on the underlying Riemmanian manifold, whose characteristics (that are dependent on graph topology and the GNN model) pave the way for a principled representation of the latent geometric knowledge.
Revised: Specifically, we interpret feature propagation as heat flows on the underlying Riemmanian manifold, whose characteristics (that are dependent on graph topology and the GNN model) pave the way for a principled representation of the latent geometric knowledge.

Original: 1.1 Our Contributions
Revised: 1.1 Our Contributions

Original: New theoretical perspective for analyzing latent graph geometry.
Revised: New theoretical perspective for analyzing latent graph geometry.

Original: On top of the connection between heat equation and GNNs, we step further to inspect the implication of heat kernel for
Revised: On top of the connection between heat equation and GNNs, we step further to inspect the implication of heat kernel for

Original: GNNs, and propose a novel notion of Neural Heat Kernel (NHK) with rigorous proof of its existence.
Revised: GNNs, and propose a novel notion of Neural Heat Kernel (NHK) with rigorous proof of its existence.

Original: Heat kernel intrinsically defines the unique solution to the heat equation and can be a fundamental characterization for the geometric property of the underlying manifold [18, 19].
Revised: Heat kernel intrinsically defines the unique solution to the heat equation and can be a fundamental characterization for the geometric property of the underlying manifold [18, 19].

Original: Likewise, NHK uncovers geometric property of the latent graph manifold for GNNs, and governs how information flows between pairs of instances, which lends us a mathematical tool to encapsulate geometric knowledge extracted from GNN model and enables geometric knowledge transfer.
Revised: Likewise, NHK uncovers geometric property of the latent graph manifold for GNNs, and governs how information flows between pairs of instances, which lends us a mathematical tool to encapsulate geometric knowledge extracted from GNN model and enables geometric knowledge transfer.

Original: This result alone might also be useful in broader contexts for understanding GNNs.
Revised: This result alone might also be useful in broader contexts for understanding GNNs.

Original: Flexible distillation framework with versatile instantiations.
Revised: Flexible distillation framework with versatile instantiations.

Original: Based on the above insights, we treat NHK matrices as representation of the latent geometric knowledge, upon which we build a flexible and principled distillation framework dubbed as Geometric Knowledge Distillation (GKD), which aims at encoding and transferring geometric knowledge by aligning latent manifolds behind
Revised: Based on the above insights, we treat NHK matrices as representation of the latent geometric knowledge, upon which we build a flexible and principled distillation framework dubbed as Geometric Knowledge Distillation (GKD), which aims at encoding and transferring geometric knowledge by aligning latent manifolds behind

Original: GNN models as illustrated in Fig.
Revised: GNN models as illustrated in Fig.

Original: We also develop non-parametirc and parametric versions of
Revised: We also develop non-parametirc and parametric versions of

Original: GKD, in terms of different ways to approximate NHK computation.
Revised: GKD, in terms of different ways to approximate NHK computation.

Original: Specifically, the former derives explicit NHKs via assumptions on latent space, and the later learns NHK in a data-driven manner.
Revised: Specifically, the former derives explicit NHKs via assumptions on latent space, and the later learns NHK in a data-driven manner.

Original: Applications for geometric knowledge transfer and conventional KD purposes.
Revised: Applications for geometric knowledge transfer and conventional KD purposes.

Original: We verify the efficacy of GKD in terms of different geometric knowledge types (i.e., edge-aware and node-aware ones), and further show its effectiveness for conventional KD purposes (e.g., model compression, self distillation, online distillation) for broader applicability.
Revised: We verify the efficacy of GKD in terms of different geometric knowledge types (i.e., edge-aware and node-aware ones), and further show its effectiveness for conventional KD purposes (e.g., model compression, self distillation, online distillation) for broader applicability.

Original: We highlight that our methods consistently exceed teacher model and rival with the oracle model that gives the performance upper bound in principle.
Revised: We highlight that our methods consistently exceed teacher model and rival with the oracle model that gives the performance upper bound in principle.

Original: 1.2 Links to Related Works
Revised: 1.2 Links to Related Works

Original: Geometric Deep Learning.
Revised: Geometric Deep Learning.

Original: The study of geometric deep learning [5, 36] provides fundamental principles and methodology to generalize deep learning methods to non-Euclidean domains (e.g., graphs and manifolds).
Revised: The study of geometric deep learning [5, 37] provides fundamental principles and methodology to generalize deep learning methods to non-Euclidean domains (e.g., graphs and manifolds).

Original: From this perspective, architectures for off-the-shelf GNNs [27, 46, 50, 20] have naturally incorporated the geometric prior knowledge for graphs such as permutation invariance .
Revised: From this perspective, architectures for off-the-shelf GNNs [28, 48, 52, 20] have naturally incorporated the geometric prior knowledge for graphs such as permutation invariance .

Original: Despite their remarkable success, they highly rely on the graph topology.
Revised: Despite their remarkable success, they highly rely on the graph topology.

Original: This work extends the idea of geometric deep learning by treating the global graph topology as a special type of prior knowledge, and attempts to encode it into GNNs themselves, such that the trained model would leverage information from the global graph topology even without explicitly taking it as input.
Revised: This work extends the idea of geometric deep learning by treating the global graph topology as a special type of prior knowledge, and attempts to encode it into GNNs themselves, such that the trained model would leverage information from the global graph topology even without explicitly taking it as input.

Original: Graph-Based Knowledge Distillation.
Revised: Graph-Based Knowledge Distillation.

Original: Knowledge distillation (KD)
Revised: Knowledge distillation (KD)

Original: [22, 1] uses the outputs of a teacher model as alternative supervised signals to teach a student model, with various new paradigms including feature-based [39, 62] and relation-based [60] ones.
Revised: [22, 1] uses the outputs of a teacher model as alternative supervised signals to teach a student model, with various new paradigms including feature-based [41, 64] and relation-based [62, 63] ones.

Original: While some prior arts [9,58, 47, 59, 57] attempted to combine KD and GNNs, i.e., graph-based KD, they are nearly straight-forward adaptations of KD without in-depth investigation on the role of graph topology, also restricted by a specific choice of GNN architecture or application scenarios.
Revised: While some prior arts [9, 59, 49, 61, 58] attempted to combine KD and GNNs, i.e., graph-based KD, they are nearly straight-forward adaptations of KD without in-depth investigation on the role of graph topology, also restricted by a specific choice of GNN architecture or application scenarios.

Original: In contrast, we first formalize the problem of geometric knowledge transfer, theoretically answer the question “how to represent graph geometric knowledge and encode it into GNN models", and propose geometric distillation approach based on the theoretical results.
Revised: In contrast, we first formalize the problem of geometric knowledge transfer, theoretically answer the question “how to represent graph geometric knowledge and encode it into GNN models", and propose geometric distillation approach based on the theoretical results.

Original: We also compare with other distilling methods in Appendix F.6. Generalization of GNNs. Recent advances shed lights on the generalization ability of GNNs from various perspectives, e.g., the in-distribution generalization error [16], extrapolation capability [56], out-of-distribution generalization under distribution shifts [53]. Our work can be seen a specific embodiment for GNN generalization w.r.t.
Revised: More discussions on related works are deferred to Appendix G.

Original: topological domain transfer.

Original: Furthermore, there are some recent studies exploring learning under the varied data space between training and inference [52, 51], using GNNs as an encoding and reasoning tool.

Original: GKD focuses on transferring across varied structural information and aims at compressing topological information for GNNs.

Original:  Preliminaries
Revised:  Preliminaries

Original: We commence with a brief detour to heat equation on Riemannian manifolds, and its connection with modern GNN architectures.
Revised: We commence with a brief detour to heat equation on Riemannian manifolds, and its connection with modern GNN architectures.

Original: Moreover, we bring forth the notion of heat kernel to motivate this work.
Revised: Moreover, we bring forth the notion of heat kernel to motivate this work.

Original: Heat Equation on Manifolds.
Revised: Heat Equation on Manifolds.

Original: We are interested in heat equation defined on a smooth kdimensional Riemannian manifold M .
Revised: We are interested in heat equation defined on a smooth kdimensional Riemannian manifold M .

Original: Suppose the manifold is associated with a scalar- or vectorvalued function x ( u, t ) : M × [0 , ∞ )
Revised: Suppose the manifold is associated with a scalar- or vectorvalued function x ( u, t ) : M × [0 , ∞ )

Original: → R d , quantifying a specific type of signals such as heat at a point u ∈ M and time t .
Revised: → R d , quantifying a specific type of signals such as heat at a point u ∈ M and time t .

Original: Fourier’s law of heat conductivity describes the flow of heat with respect to time and space, via a partial differential equation (PDE) called heat equation [6], i.e.,
Revised: Fourier’s law of heat conductivity describes the flow of heat with respect to time and space, via a partial differential equation (PDE) called heat equation [6], i.e.,

Original: [Equation]
Revised: [Equation]

Original: where c > 0 is the thermal conductivity coefficient, and ∆ is the natural Laplace–Beltrami operator associated with M .
Revised: where c > 0 is the thermal conductivity coefficient, and ∆ is the natural Laplace–Beltrami operator associated with M .

Original: Rewriting ∆ as the functional composition of the divergence operator ∇ ∗ and gradient operator ∇ , i.e., ∆ = ∇ ∗ ◦ ∇ , we can interpret the heat equation as: the variation of temperature within an infinitesimal time interval at a point is equivalent to the divergence between its own temperature and the average temperature on an infinitesimal sphere around it.
Revised: Rewriting ∆ as the functional composition of the divergence operator ∇ ∗ and gradient operator ∇ , i.e., ∆ = ∇ ∗ ◦ ∇ , we can interpret the heat equation as: the variation of temperature within an infinitesimal time interval at a point is equivalent to the divergence between its own temperature and the average temperature on an infinitesimal sphere around it.

Original: Implications on Graphs.
Revised: Implications on Graphs.

Original: A spatial discretisation of a continuous manifold yields a graph G = {V , E} , whose nodes can be thought of as embedded on the base manifold.
Revised: A spatial discretisation of a continuous manifold yields a graph G = {V , E} , whose nodes can be thought of as embedded on the base manifold.

Original: In fact, the heat equation along with variants thereof (e.g., Schrödinger equation) have found widespread use in modeling graph dynamics [11, 25, 34].
Revised: In fact, the heat equation along with variants thereof (e.g., Schrödinger equation) have found widespread use in modeling graph dynamics [11, 25, 35].

Original: More importantly, it has been recently revealed to be intimately related with the architectures of modern GNNs [48, 8, 7]: suppose X (0) =
Revised: More importantly, it has been recently revealed to be intimately related with the architectures of modern GNNs [50, 8, 7]: suppose X (0) =

Original: { x ( u, 0) } u ∈V ∈ R n × d denotes the initial condition for Eqn.
Revised: { x ( u, 0) } u ∈V ∈ R n × d denotes the initial condition for Eqn.

Original: (2) determined by input node features, then solving the heat equation under certain definitions of ∇ ∗ and ∇ (i.e., definition of ∆ ) amounts to different architectures of GNNs.
Revised: (2) determined by input node features, then solving the heat equation under certain definitions of ∇ ∗ and ∇ (i.e., definition of ∆ ) amounts to different architectures of GNNs.

Original: For instance:
Revised: For instance:

Original: Example 1.
Revised: Example 1.

Original: [48] Define the discretised counterpart of ∆ as the graph Laplacian matrix L = (cid:101)
Revised: [50] Define the discretised counterpart of ∆ as the graph Laplacian matrix L = (cid:101)

Original: D − 12 ( (cid:101) D − (cid:101) A ) (cid:101) D − 12 .
Revised: D − 12 ( [Equation] D − (cid:101) A ) (cid:101) D − 12 .

Original: Numerically solving Eqn.
Revised: Numerically solving Eqn.

Original: (2) using the forward Euler method with step size τ = 1 yields the formulation of Simple Graph Convolution (SGC) [50], where Θ denote learnable transformation matrix
Revised: (2) using the forward Euler method with step size τ = 1 yields the formulation of Simple Graph Convolution (SGC) [52], where Θ denote learnable transformation matrix

Original: [Equation]
Revised: [Equation]

Original: Example 2.
Revised: Example 2.

Original: [8] Define the gradient operator ∇ ij as the difference of source and target node features, the divergence operator ∇ ∗ i as the sum of features of all edges for the node.
Revised: [8] Define the gradient operator ∇ ij as the difference of source and target node features, the divergence operator ∇ ∗ i as the sum of features of all edges for the node.

Original: Numerically solving
Revised: Numerically solving

Original: Eqn.
Revised: Eqn.

Original: (2) using the explicit Euler scheme with step size τ yields the following recursive formulation
Revised: (2) using the explicit Euler scheme with step size τ yields the following recursive formulation

Original: [Equation]
Revised: [Equation]

Original: where G is a diffusivity coefficient matrix in place of c .
Revised: where G is a diffusivity coefficient matrix in place of c .

Original: Moreover, stacking a non-linear transformation layer after each step yields the formulation of Graph
Revised: Moreover, stacking a non-linear transformation layer after each step yields the formulation of Graph

Original: Convolution Networks (GCN) [27] for Eqn.
Revised: Convolution Networks (GCN) [28] for Eqn.

Original: (3), Graph Attention Networks (GAT) [46] with residual connection for Eqn.
Revised: (3), Graph Attention Networks (GAT) [48] with residual connection for Eqn.

Original: (4), and even more GNN architectures by virtue of the flexibility of interpretion for heat equation on graphs.
Revised: (4), and even more GNN architectures by virtue of the flexibility of interpretion for heat equation on graphs.

Original: Heat Kernels.
Revised: Heat Kernels.

Original: Intriguingly, it turns out that the initial value problem of heat equation on any manifold M has a smallest positive fundamental solution depending on the Laplace operator ∆ , known as the heat kernel [2].
Revised: Intriguingly, it turns out that the initial value problem of heat equation on any manifold M has a smallest positive fundamental solution depending on the Laplace operator ∆ , known as the heat kernel [2].

Original: It is denoted as a kernel function κ ( x, y, t ) , such that
Revised: It is denoted as a kernel function κ ( x, y, t ) , such that

Original: [Equation]
Revised: [Equation]

Original: where µ is a non-negative measure associated with M .
Revised: where µ is a non-negative measure associated with M .

Original: In physics, the heat kernel κ ( x, y, t ) can be interpreted as a transition density that describes the asymptotic behavior of a natural Brownian motion on the manifold.
Revised: In physics, the heat kernel κ ( x, y, t ) can be interpreted as a transition density that describes the asymptotic behavior of a natural Brownian motion on the manifold.

Original: Its formulation thus can be treated as a unique reflection or representation of the geometry of the underlying manifold .
Revised: Its formulation thus can be treated as a unique reflection or representation of the geometry of the underlying manifold .

Original: For example, if the manifold is a k -dimensional Euclidean
Revised: For example, if the manifold is a k -dimensional Euclidean

Original: Space R k or a Hyperbolic Space H k , the explicit formula of heat kernel is respectively given by, κ ( u i , u j , t ) =
Revised: Space R k or a Hyperbolic Space H k , the explicit formula of heat kernel is respectively given by, κ ( u i , u j , t ) =

Revised: (4 πt ) k/ 2 exp (cid:18) − ρ

Original: [Equation]
Revised: [Equation]

Revised: t (cid:19) and κ ( u i , u j , t ) = ( − 1) m 2 m π m

Original: [Equation] 
Revised: [Equation] [Equation]

Revised: sinh ρ ∂

Original: [Equation]
Revised: [Equation]

Original:  where ρ = d ( u i , u j ) denote geodesic distance.
Revised: ) where ρ = d ( u i , u j ) denote geodesic distance.

Original: Heat kernel has also been adopted for graph-related applications such as community detection [30], graph clustering [55].
Revised: Heat kernel has also been adopted for graph-related applications such as community detection [31], graph clustering [56].

Original:  Extending Heat Kernel to GNNs
Revised:  Extending Heat Kernel to GNNs

Original: The starting point of this work is the development of neural heat kernel , built upon the previouslymentioned connection of GNNs and heat equation.
Revised: The starting point of this work is the development of neural heat kernel , built upon the previouslymentioned connection of GNNs and heat equation.

Original: As will be discussed later, this novel notion lends us a thermodynamic perspective to the intrinsic geometric property of the latent graph manifold embodied in GNNs, and hence paves the way for distilling geometric knowledge.
Revised: As will be discussed later, this novel notion lends us a thermodynamic perspective to the intrinsic geometric property of the latent graph manifold embodied in GNNs, and hence paves the way for distilling geometric knowledge.

Original: 3.1 Neural Heat Kernel
Revised: 3.1 Neural Heat Kernel

Original: Consider the graph signal X ( t ) at time t and node features H ( l ) at layer l as interchangeable notions.
Revised: Consider the graph signal X ( t ) at time t and node features H ( l ) at layer l as interchangeable notions.

Original: Consequently, feature propagation using one layer of GNN amounts to heat diffusion on the base manifold M within a certain time interval τ , leading to the equivalences of X ( t + τ ) and H ( l +1) :
Revised: Consequently, feature propagation using one layer of GNN amounts to heat diffusion on the base manifold M within a certain time interval τ , leading to the equivalences of X ( t + τ ) and H [Equation]

Original: [Equation]
Revised: [Equation]

Original: where f θ denotes an arbitrary GNN model with parameter θ , and ∆( f θ , G ) denotes a generalization of Laplace-Beltrami operator defined over the base manifold M associated with graph G and the arbitrary backbone GNN model f θ .
Revised: where f θ denotes an arbitrary GNN model with parameter θ , and ∆( f θ , G ) denotes a generalization of Laplace-Beltrami operator defined over the base manifold M associated with graph G and the arbitrary backbone GNN model f θ .

Original: Remark.
Revised: Remark.

Original: The equivalence of two equations in Eqn.
Revised: The equivalence of two equations in Eqn.

Original: 7 is based on the recently established connection between heat equation and GNNs [8, 7, 48, 13, 12], which reveal that the formulation of a GNN layer could be thought of as discretisations (that correspond to the left equation in Eqn. 7) of the continuous diffusion process (that correspond to the right equation in Eqn. 7) described by the heat equation.
Revised: 7 is based on the recently established connection between heat equation and GNNs [8, 7, 50, 13, 12], which reveal that the formulation of a GNN layer could be thought of as discretisations (that correspond to the left equation in Eqn. 7) of the continuous diffusion process (that correspond to the right equation in Eqn. 7) described by the heat equation.

Original: Furthermore, different definitions of Laplace-Beltrami operator ∆ and schemes for solving Eqn.could yield different GNNs (e.g., SGC [50], GAT [46], GRAND [8]).
Revised: Furthermore, different definitions of Laplace-Beltrami operator ∆ and schemes for solving Eqn.could yield different GNNs (e.g., SGC [52], GAT [48], GRAND [8]).

Original: While it is unclear whether there exists such a definition of ∆ for every GNN architecture, we write the operator as ∆( f θ , G ) to associate it with model f θ , and then use the analogy between GNN and heat equation as an analytical tool, in a similar manner with [3, 43, 48, 42, 8, 7], for studying the geometry property of GNNs.
Revised: While it is unclear whether there exists such a definition of ∆ for every GNN architecture, we write the operator as ∆( f θ , G ) to associate it with model f θ , and then use the analogy between GNN and heat equation as an analytical tool, in a similar manner with [3, 45, 50, 44, 8, 7], for studying the geometry property of GNNs.

Original: See more detailed justifications in Appendix E.
Revised: See more detailed justifications in Appendix E.

Original: In light of this connection, we consider a natural generalization of heat kernel for GNNs, termed as neural heat kernel (NHK) to highlight its difference with heat kernel in the thermodynamic context.
Revised: In light of this connection, we consider a natural generalization of heat kernel for GNNs, termed as neural heat kernel (NHK) to highlight its difference with heat kernel in the thermodynamic context.

Original: In particular, a single-layer NHK is defined as a positive definite symmetric kernel function denoted as κ ( l ) θ ( v i , v j ) , where the sub-script θ implies that it is associated with the architecture and parameters of the backbone GNN, and the super-script ( l ) implies that it is specific to each layer, analogous to the role of continuous time t in Eqn.
Revised: In particular, a single-layer NHK is defined as a positive definite symmetric kernel function denoted as κ ( l ) θ ( v i , v j ) , where the sub-script θ implies that it is associated with the architecture and parameters of the backbone GNN, and the super-script ( l ) implies that it is specific to each layer, analogous to the role of continuous time t in Eqn.

Original: Theorem 1.
Revised: Theorem 1.

Original: (Existence of Single-Layer NHK)
Revised: (Existence of Single-Layer NHK)

Original: Suppose two expressions in Eqn.
Revised: Suppose two expressions in Eqn.

Original: (7) are equivalent (see Appendix E for more discussions), then for any graph G and GNN model f θ , there exist a unique single-layer NHK function κ ( l ) θ ( · ) such that for any node v i ∈ V and l > 0 ,
Revised: (7) are equivalent (see Appendix E for more discussions), then for any graph G and GNN model f θ , there exist a unique single-layer NHK function κ ( l ) θ ( · ) such that for any node v i ∈ V and l > 0 ,

Original: [Equation]
Revised: [Equation]

Original: where h ( l ) i ∈
Revised: where h ( l ) i ∈

Original: R
Revised: R

Original: d denotes the feature of node v i at l -th layer, and µ is a measure over vertices that could be specified as the inverse of node degree 1 /d i .
Revised: d denotes the feature of node v i at l -th layer, and µ is a measure over vertices that could be specified as the inverse of node degree 1 /d i .

Original: To push further, we can generalize NHK across multiple layers of GNN, termed as a cross-layer NHK κ θ ( v i , v j , l (cid:55)→ l + k ) (e.g., from l -th layer to ( l + k ) -th layer of GNN).
Revised: To push further, we can generalize NHK across multiple layers of GNN, termed as a cross-layer NHK κ θ ( v i , v j , l (cid:55)→ l + k ) (e.g., from l -th layer to ( l + k ) -th layer of GNN).

Original: Its existence could be induced recursively by the semi-group identity property of NHK concerning consecutive GNN layers.
Revised: Its existence could be induced recursively by the semi-group identity property of NHK concerning consecutive GNN layers.

Original: Theorem 2.
Revised: Theorem 2.

Original: (Semigroup Identity Property of NHK)
Revised: (Semigroup Identity Property of NHK)

Original: The NHK satisfies the semigroup identity property: ∀ v i , v j ∈ V and l > 0 , there exists a cross-layer NHK across two consecutive layers
Revised: The NHK satisfies the semigroup identity property: ∀ v i , v j ∈ V and l > 0 , there exists a cross-layer NHK across two consecutive layers

Original: [Equation]
Revised: [Equation]

Original: This theorem indicates that stacks of multiple GNN layers also constitute a valid kernel, i.e.,
Revised: This theorem indicates that stacks of multiple GNN layers also constitute a valid kernel, i.e.,

Original: [Equation]
Revised: [Equation]

Original: Analogous to heat kernel as an unique characterization of the underlying space, NHK characterizes the geometric property of the latent graph manifold for GNNs.
Revised: Analogous to heat kernel as an unique characterization of the underlying space, NHK characterizes the geometric property of the latent graph manifold for GNNs.

Original: Additionally, NHK is dependent on GNN models through the definition of the associated Laplace-Beltrami operator ∆( f θ , G ) , inheriting the expressiveness of neural networks and varying through the course of training.
Revised: Additionally, NHK is dependent on GNN models through the definition of the associated Laplace-Beltrami operator ∆( f θ , G ) , inheriting the expressiveness of neural networks and varying through the course of training.

Original: Intuitively, NHK can be thought of as a model-driven encoding for topological information, encapsulating the geometric knowledge learned by GNNs into a tractable functional form.
Revised: Intuitively, NHK can be thought of as a model-driven encoding for topological information, encapsulating the geometric knowledge learned by GNNs into a tractable functional form.

Original: 3.2 Application in Geometric Distillation
Revised: 3.2 Application in Geometric Distillation

Original: Consider the problem of distilling geometric knowledge, which involves an intelligent teacher model f θ ∗ , which is exposed to and pre-trained over the (relatively) complete graph ˜ G =
Revised: Consider the problem of distilling geometric knowledge, which involves an intelligent teacher model f θ ∗ , which is exposed to and pre-trained over the (relatively) complete graph ˜ G =

Original: (˜ V , ˜ E ) , and a student model f θ that is exposed to the partial graph G =
Revised: (˜ V , ˜ E ) , and a student model f θ that is exposed to the partial graph G =

Original: ( V , E ) , where V ⊆ ˜ V and E ⊆
Revised: ( V , E ) , where V ⊆ ˜ V and E ⊆

Original: {V ×V}∩ ˜ E .
Revised: {V ×V}∩ ˜ E .

Original: Our target is to train a student model (with the help of teacher model) that operates on G to be as competitive as models operating on ˜ G during inference.
Revised: Our target is to train a student model (with the help of teacher model) that operates on G to be as competitive as models operating on ˜ G during inference.

Original: Since G is a sub-graph of ˜ G , they should lie in the same space (i.e., latent manifold) governed by the underlying mechanism of data generation, and hence we expect student and teacher models to capture the same geometric property of this shared space.
Revised: Since G is a sub-graph of ˜ G , they should lie in the same space (i.e., latent manifold) governed by the underlying mechanism of data generation, and hence we expect student and teacher models to capture the same geometric property of this shared space.

Original: This leads to the principle of Geometric Knowledge Distillation (GKD): transfer the geometric knowledge of the intelligent teacher to the student such that the student can propagate features as if it is aware of the complete graph topology (see the example in Fig. 1).
Revised: This leads to the principle of Geometric Knowledge Distillation (GKD): transfer the geometric knowledge of the intelligent teacher to the student such that the student can propagate features as if it is aware of the complete graph topology (see the example in Fig. 1).

Original: To this end, we resort to NHK matrices on the teacher (resp.
Revised: To this end, we resort to NHK matrices on the teacher (resp.

Original: student) model over the complete (resp. partial) graph as instantiations of their geometric knowledge, denoted as[Equation] Student
Revised: student) model over the complete (resp. partial) graph as instantiations of their geometric knowledge, denoted as

Original: [Equation]
Revised: [Equation]

Original: written compactly as K ( l +1) ( G ) when k = 1 .
Revised: written compactly as K ( l +1) ( G ) when k = 1 .

Original: The NHK matrix is a positive semi-definite symmetric matrix, and alike κ , is dependent on the GNN model f θ and graph G .
Revised: The NHK matrix is a positive semi-definite symmetric matrix, and alike κ , is dependent on the GNN model f θ and graph G .

Original: Denote K ( l ) θ ∗ , V ( ˜ G ) ∈ R |V|×|V| as the sub-matrix of K ( l ) θ ∗ ( ˜ G ) with row and column indices in V .
Revised: Denote K ( l ) θ ∗ , V ( ˜ G ) ∈ R |V|×|V| as the sub-matrix of K ( l ) θ ∗ ( ˜ G ) with row and column indices in V .

Original: The distillation loss for GKD is
Revised: The distillation loss for GKD is

Original: [Equation]
Revised: [Equation]

Original: where d( · , · ) is a similarity measure, for which we choose Frobenius distance as implementation, i.e.,
Revised: where d( · , · ) is a similarity measure, for which we choose Frobenius distance as implementation, i.e.,

Original: [Equation]
Revised: [Equation]

Original: (12) where W ∈ R |V|×|V| is a weighting matrix to trade-off distillation loss with respect to different node pairs depending on their connectivity.
Revised: K [Equation] where W ∈ R |V|×|V| is a weighting matrix to trade-off distillation loss with respect to different node pairs depending on their connectivity.

Original: For k = 1 , the loss can be re-written as L ( l +1) dis
Revised: For k = 1 , the loss can be re-written as L ( l +1) dis

Original: ( K θ ∗ , V , K θ ) .
Revised: ( K θ ∗ , V , K θ ) .

Original: Note that one can also specify different k for teacher and student models in Eqn.
Revised: Note that one can also specify different k for teacher and student models in Eqn.

Original: (11) in case when the teacher model is deeper.
Revised: (11) in case when the teacher model is deeper.

Original: We also study the impact of different similarity measures in Appendix ?? .
Revised: We also study the impact of different similarity measures in Appendix ?? .

Original:  Instantiations for Geometric Knowledge Distillation
Revised:  Instantiations for Geometric Knowledge Distillation

Original: Unfortunately, deriving explicit formulas for NHKs is prohibitively challenging due to introduction of non-linearity.
Revised: Unfortunately, deriving explicit formulas for NHKs is prohibitively challenging due to introduction of non-linearity.

Original: To circumvent it, we propose two types of instantiations for GKD, i.e., non-parametric and parametric.
Revised: To circumvent it, we propose two types of instantiations for GKD, i.e., non-parametric and parametric.

Original: The former considers explicit NHKs by making assumptions on the underlying space, and the latter learns NHK in a data-driven manner.
Revised: The former considers explicit NHKs by making assumptions on the underlying space, and the latter learns NHK in a data-driven manner.

Original: 4.1 Non-Parametric Geometric Distillation
Revised: 4.1 Non-Parametric Geometric Distillation

Original: Deterministic Kernel.
Revised: Deterministic Kernel.

Original: One instantiation of NHK is a Gauss-Weierstrass kernel in the form of
Revised: One instantiation of NHK is a Gauss-Weierstrass kernel in the form of

Original: Eqn.
Revised: Eqn.

Original: (6), by assuming the underlying space is a Euclidean space.
Revised: (6), by assuming the underlying space is a Euclidean space.

Original: Since the distillation loss in
Revised: Since the distillation loss in

Original: Eqn.
Revised: Eqn.

Original: (11) is a homogeneous function, we can remove its scaling factor and define NHK as
Revised: (11) is a homogeneous function, we can remove its scaling factor and define NHK as

Original: [Equation]
Revised: [Equation]

Original: where T denotes the estimation of the accumulated time interval.
Revised: where T denotes the estimation of the accumulated time interval.

Original: Alternatively, we can use Sigmoid kernel and define non-parametric NHK as:(Sigmoid
Revised: Alternatively, we can use Sigmoid kernel and define non-parametric NHK as:

Original: [Equation]
Revised: [Equation]

Original: where a, b are positive constants depending on l and k .
Revised: where a, b are positive constants depending on l and k .

Original: It is a natural and intuitive choice as similarity measurement and empirically found as-well effective, albeit does not correspond to any named manifold to our knowledge.
Revised: It is a natural and intuitive choice as similarity measurement and empirically found as-well effective, albeit does not correspond to any named manifold to our knowledge.

Original: Randomized Kernel.
Revised: Randomized Kernel.

Original: We can also define Randomized kernel based on the following theorem.
Revised: We can also define Randomized kernel based on the following theorem.

Original: Theorem 3.
Revised: Theorem 3.

Original: (Expansion of NHK) Let { φ k ′ } ∞
Revised: (Expansion of NHK) Let { φ k ′ } ∞

Original: k ′ =0 be orthonormal basis of eigenfunctions of − ∆( f θ , G ) with eigenvalues 0 < λ 0 ≤ λ 1 ≤ λ 2 ≤ . . .
Revised: k ′ =0 be orthonormal basis of eigenfunctions of − ∆( f θ , G ) with eigenvalues 0 < λ 0 ≤ λ 1 ≤ λ 2 ≤ . . .

Original: , NHK allows the expansion:
Revised: , NHK allows the expansion:

Original: [Equation]
Revised: [Equation]

Original: Based on this result, we resort to the approximation of NHK by defining a randomized kernel in a similar form as Eqn.
Revised: Based on this result, we resort to the approximation of NHK by defining a randomized kernel in a similar form as Eqn.

Original: (15), leading to the following formulation of randomized NHK:
Revised: (15), leading to the following formulation of randomized NHK:

Original: [Equation]
Revised: [Equation]

Original: where σ ( W k ′ h i ) is used to proximate φ k ′ ( v i ) ,
Revised: where σ ( W k ′ h i ) is used to proximate φ k ′ ( v i ) ,

Original: W k ′ =
Revised: W k ′ =

Original: [ ϕ 1 ,k ′ , ϕ 2 ,k ′ , · · · , ϕ s,k ′ ]
Revised: [ ϕ 1 ,k ′ , ϕ 2 ,k ′ , · · · , ϕ s,k ′ ]

Original: ⊤ is a transformation matrix, ϕ ∼ N ( 0 , I d ) is a d -dimensional random variable from Gaussian distribution.
Revised: ⊤ is a transformation matrix, ϕ ∼ N ( 0 , I d ) is a d -dimensional random variable from Gaussian distribution.

Original: In fact, under certain choice of activation function σ , Eqn.
Revised: In fact, under certain choice of activation function σ , Eqn.

Original: (16) could approximate a diversity of kernels [38, 10].
Revised: (16) could approximate a diversity of kernels [40, 10].

Original: This design essentially enforces the alignment of teacher and student for arbitrary underlying manifold.
Revised: This design essentially enforces the alignment of teacher and student for arbitrary underlying manifold.

Original: Training Scheme.
Revised: Training Scheme.

Original: We follow the standard training paradigm in KD literature [22, 17]: the teacher is pre-trained by a supervised prediction loss involving all labeled nodes in ˜ V .
Revised: We follow the standard training paradigm in KD literature [22, 17]: the teacher is pre-trained by a supervised prediction loss involving all labeled nodes in ˜ V .

Original: After teacher is well-trained, we fix θ ∗ and train the student model according to
Revised: After teacher is well-trained, we fix θ ∗ and train the student model according to

Original: [Equation]
Revised: [Equation]

Original: where Y denotes ground-truth labels of labeled nodes in V , and ˆ Y θ denotes the predictions of student model f θ on G , L dis is the distillation loss defined by Eqn.
Revised: where Y denotes ground-truth labels of labeled nodes in V , and ˆ Y θ denotes the predictions of student model f θ on G , L dis is the distillation loss defined by Eqn.

Original: (11), L denotes the total number of layers.
Revised: (11), L denotes the total number of layers.

Original: 4.2 Parametric Geometric Distillation
Revised: 4.2 Parametric Geometric Distillation

Original: Inheriting the similar spirit of auto-encoding Bayes [26], we introduce a variational inverse-NHK that is independently parameterized, denoted as κ † ϕ , whose existence is guaranteed by the invertibility of
Revised: Inheriting the similar spirit of auto-encoding Bayes [27], we introduce a variational inverse-NHK that is independently parameterized, denoted as κ † ϕ , whose existence is guaranteed by the invertibility of

Original: NHK matrices.
Revised: NHK matrices.

Original: Together with κ θ , they define a symmetric form characterizing feature propagation:
Revised: Together with κ θ , they define a symmetric form characterizing feature propagation:

Original: [Equation]
Revised: [Equation]

Original: In practice, we follow existing kernel learning approaches [49] and parameterize the inverse-NHK as
Revised: In practice, we follow existing kernel learning approaches [51] and parameterize the inverse-NHK as

Original: [Equation]
Revised: [Equation]

Original: where g ϕ : R d → R s is the associated learnable non-linear mapping.
Revised: where g ϕ : R d → R s is the associated learnable non-linear mapping.

Original: Given a pre-trained teacher model, distilling geometric knowledge boils down to 1) establishing equivalence of Eqn.
Revised: Given a pre-trained teacher model, distilling geometric knowledge boils down to 1) establishing equivalence of Eqn.

Original: (18) and
Revised: (18) and

Original: Eqn. (19), and 2) matching pseudo-inverse NHK matrices for teacher and student models (respectively denoted as K † θ ∗ , V and K † θ with clear meanings), leading to the training scheme as follows.
Revised: Eqn. (19), and 2) matching pseudo-inverse NHK matrices for teacher and student models (respectively denoted as K † θ ∗ , V and K † θ with clear meanings), leading to the training scheme as follows.

Original: Training Scheme.
Revised: Training Scheme.

Original: Based on Eqn.
Revised: Based on Eqn.

Original: (19), we can define a reconstruction loss with respect to the teacher model (similar applies to the student model) as
Revised: (19), we can define a reconstruction loss with respect to the teacher model (similar applies to the student model) as

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Then, minimizing the reconstruction loss with fixed GNN model parameter θ amounts to optimizing the variational parameter ϕ , and minimizing prediction and distillation losses given fixed ϕ amounts to optimizing the student model parameter 
Revised: Then, minimizing the reconstruction loss with fixed GNN model parameter θ amounts to optimizing the variational parameter ϕ , and minimizing prediction and distillation losses given fixed ϕ amounts to optimizing the student model parameter θ :

Original: [Equation]
Revised: [Equation]

Original: Applying two steps iteratively adds up to an EM-like algorithm for training the student model.
Revised: Applying two steps iteratively adds up to an EM-like algorithm for training the student model.

Original: In practice, we set l + k as the last layer, and l as the first layer to use as much information as possible.
Revised: In practice, we set l + k as the last layer, and l as the first layer to use as much information as possible.

Original: We justify the parametric GKD approach in Appendix.
Revised: We justify the parametric GKD approach in Appendix.

Original: D by showing it essentially explores the true
Revised: D by showing it essentially explores the true

Original: NHK behind GNN.
Revised: NHK behind GNN.

Original:  Experiments
Revised:  Experiments

Original: We conduct experiments to validate the efficacy of our method on graph-structured data in terms of various types of privileged geometric knowledge, combinations of teacher-student GNN architectures and potential application scenarios.
Revised: We conduct experiments to validate the efficacy of our method on graph-structured data in terms of various types of privileged geometric knowledge, combinations of teacher-student GNN architectures and potential application scenarios.

Original: We use three benchmark datasets Cora [33], Citeseer [41],
Revised: We use three benchmark datasets Cora [34], Citeseer [43],

Original: Pubmed [37], and a larger dataset OGBN-Arxiv [23], for node classification tasks.
Revised: Pubmed [38], and a larger dataset OGBN-Arxiv [23], for node classification tasks.

Original: More implementation details and experimental results are deferred to Appendix.
Revised: More implementation details and experimental results are deferred to Appendix.

Original: The codes will be available at https://github.com/chr26195/GKD .
Revised: The codes will be available at https://github.com/chr26195/GKD .

Original: Implementation and Competitors We consider the following variants of the proposed GKD.1)
Revised: Implementation and Competitors. We consider the following variants of the proposed GKD.

Original:  GKD-G : non-parametric Gaussian NHK; 2) GKD-S : non-parametric Sigmoid NHK; 3) GKD-R : randomized NHK; 3) PGKD : parametric NHK.
Revised: 1) GKD-G : non-parametric Gaussian NHK; 2) GKD-S : non-parametric Sigmoid NHK; 3) GKDR : randomized NHK; 3) PGKD : parametric NHK.

Original: We choose KD methods that is representative in its own category for comparison, including KD [22], FSP [60], LSP [59].
Revised: We choose KD methods that is representative in its own category for comparison, including KD [22], FSP [62], LSP [61].

Original: We also report the performances of teacher and student model trained with the standard classification loss, short as
Revised: We also report the performances of teacher and student model trained with the standard classification loss, short as

Original: Teacher and Student .
Revised: Teacher and Student .

Original: The teacher model is trained using the complete graph ˜ G , and, to calibrate with all other methods, tested using the partial graph G .
Revised: The teacher model is trained using the complete graph ˜ G , and, to calibrate with all other methods, tested using the partial graph G .

Original: Besides, we consider an Oracle model which is both trained and tested on ˜ G , which naturally takes an advantaged place given more information during inference.
Revised: Besides, we consider an Oracle model which is both trained and tested on ˜ G , which naturally takes an advantaged place given more information during inference.

Original: Since our method is compatible with the vanilla KD paradigm [22], we report the performance delivered by their combinations (i.e., GKD+KD and PGKD+KD).
Revised: Since our method is compatible with the vanilla KD paradigm [22], we report the performance delivered by their combinations (i.e., GKD+KD and PGKD+KD).

Original: Experiment Settings We investigate on various experimental settings according to different types of privileged geometric knowledge.
Revised: Experiment Settings. We investigate on various experimental settings according to different types of privileged geometric knowledge.

Original: In the case of edge-aware geometric knowledge , the teacher

Original: [Table]

Original: [Table] [Table] model has access to additional edge information, i.e., E ⊂ ˜ E and V = ˜ V .
Revised: In the case of edge-aware geometric knowledge , the teacher model has access to additional edge information, i.e., E ⊂ ˜ E and V = ˜ V .

Original: In the case of node-aware geometric knowledge , the teacher model has access to additional node information, i.e., V ⊂ ˜ V and
Revised: In the case of node-aware geometric knowledge , the teacher model has access to additional node information, i.e., V ⊂ ˜ V and

Original: E = ˜
Revised: E = ˜

Original: E ∩ {V × V} .
Revised: E ∩ {V × V} .

Original: We also consider other conventional KD settings including model compression, self-distillation and online distillation, which will be illustrated in detail.
Revised: We also consider other conventional KD settings including model compression, self-distillation and online distillation, which will be illustrated in detail.

Original: The backbone f θ is set as 3 -layer GCN [27] for both student and teacher models, unless otherwise stated. 
Revised: The backbone f θ is set as 3 -layer GCN [28] for both student and teacher models, unless otherwise stated. [Table] [Table] [Table]

Original: 5.1 Main Results
Revised: 5.1 Main Results

Original: Edge-Aware Geometric Knowledge.
Revised: Edge-Aware Geometric Knowledge.

Original: We report results for the edge-aware geometric knowledge setting.
Revised: We report results for the edge-aware geometric knowledge setting.

Original: To quantify the privileged information, we set the quantity ( | ˜ E| − |E| ) / | ˜ E| , called privileged information ratio (PIR), as 0 .
Revised: To quantify the privileged information, we set the quantity ( | ˜ E| − |E| ) / | ˜ E| , called privileged information ratio (PIR), as 0 .

Original: As shown in Tab. 1, all variants of GKD outperform other KD baselines on both datasets, and significantly exceeds both Student and Teacher models.
Revised: As shown in Tab. 1, all variants of GKD outperform other KD baselines on both datasets, and significantly exceeds both Student and Teacher models.

Original: Further, GKD and its variants rival, if not surpass, the Oracle model.
Revised: Further, GKD and its variants rival, if not surpass, the Oracle model.

Original: In other words, the student model trained using GKD could use far less graph topological information to achieve very close performance to competitors that are aware of the full graph topology during inference.
Revised: In other words, the student model trained using GKD could use far less graph topological information to achieve very close performance to competitors that are aware of the full graph topology during inference.

Original: Furthermore, the parametric
Revised: Furthermore, the parametric

Original: PGKD performs better than its non-parametric counterparts in most cases, and GKD-R is the most effective non-parametric method in general.
Revised: PGKD performs better than its non-parametric counterparts in most cases, and GKD-R is the most effective non-parametric method in general.

Original: Despite that, GKD-G and GKD-S are also effective while being simpler.
Revised: Despite that, GKD-G and GKD-S are also effective while being simpler.

Original: We presume that the performance variation of different GKD realizations stem from the different geometric property governed by the feature of datasets.
Revised: We presume that the performance variation of different GKD realizations stem from the different geometric property governed by the feature of datasets.

Original: Node-Aware Geometric Knowledge.
Revised: Node-Aware Geometric Knowledge.

Original: We further investigate on the node-aware geometric knowledge setting where the teacher model has access to more labeled nodes and their relations with the rest nodes.
Revised: We further investigate on the node-aware geometric knowledge setting where the teacher model has access to more labeled nodes and their relations with the rest nodes.

Original: We set the PIR w.r.t.
Revised: We set the PIR w.r.t.

Original: labeled nodes, defined as ( | ˜ V train | − |V train | ) / | ˜
Revised: labeled nodes, defined as ( | ˜ V train | − |V train | ) / | ˜

Original: V train | , to 0 .
Revised: V train | , to 0 .

Original: A unique challenge of this setting compared to the edge-ware counterpart is that, apart from graph topological information, the student model has less labeled training samples.
Revised: A unique challenge of this setting compared to the edge-ware counterpart is that, apart from graph topological information, the student model has less labeled training samples.

Original: As we can see from
Revised: As we can see from

Original: Tab. 2.
Revised: Tab. 2.

Original: The proposed GKD and its variants again consistently outperform KD baselines throughout all the cases, surpasses both Student and Teacher models, and are even as competitive as Oracle.
Revised: The proposed GKD and its variants again consistently outperform KD baselines throughout all the cases, surpasses both Student and Teacher models, and are even as competitive as Oracle.

Original: Larger Dataset.
Revised: Larger Dataset.

Original: Table 3 presents results on a large graph, i.e., OGBN-Arxiv.
Revised: Table 3 presents results on a large graph, i.e., OGBN-Arxiv.

Original: We use the same PIR setting as citation networks, and choose the best variant of GKD to report in the table.
Revised: We use the same PIR setting as citation networks, and choose the best variant of GKD to report in the table.

Original: Since the space complexity for GKD is O ( n 2 ) , we randomly draw a mini-batch of nodes for computing the distillation loss in practice.
Revised: Since the space complexity for GKD is O ( n 2 ) , we randomly draw a mini-batch of nodes for computing the distillation loss in practice.

Original: Note that the loss function is unbiased as long as all nodes are evenly covered.
Revised: Note that the loss function is unbiased as long as all nodes are evenly covered.

Original: It could also be seamlessly integrated with the original mini-batch method (by sampling ego-graphs) used for training large graphs without further modification.
Revised: It could also be seamlessly integrated with the original mini-batch method (by sampling ego-graphs) used for training large graphs without further modification.

Original: Again, we found our methods consistently outperform Teacher and Student models, and are close to the performance of the Oracle model, which suggests the effectiveness of GKD in large graphs.
Revised: Again, we found our methods consistently outperform Teacher and Student models, and are close to the performance of the Oracle model, which suggests the effectiveness of GKD in large graphs.

Original: Performance Variation with Privileged Ratio.
Revised: Performance Variation with Privileged Ratio.

Original: The results with respect to varying privileged (edgeaware and node-ware) information ratio are given respectively in Fig.
Revised: The results with respect to varying privileged (edge-aware and node-ware) information ratio are given respectively in Fig.

Original: 2 and Fig.
Revised: 2 and Fig.

Original: The performance of Oracle model is invariant as it is trained and tested on the same (complete) graph.
Revised: The performance of Oracle model is invariant as it is trained and tested on the same (complete) graph.

Original: In general,

Original: [Figure] [Figure] [Figure] for Teacher model, Student model
Revised: In general, for Teacher model, Student model, and vanilla KD, their performance drops with increasing

Original: [Figure] and vanilla KD, their performance drops with increasing PIR quantifying the information loss.
Revised: PIR quantifying the information loss.

Original: In contrast, our method is significantly more robust, only showing slight performance deterioration, exceeding the Teacher model, and approaching the Oracle model.
Revised: In contrast, our method is significantly more robust, only showing slight performance deterioration, exceeding the Teacher model, and approaching the Oracle model.

Original: Besides
Revised: Besides, we find an interesting phenomenon that in the edge-aware setting on Pubmed dataset, the performance of Teacher model is the worst, presumably because that the Teacher is trained using [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: we find an interesting phenomenon that in the edge-aware setting on Pubmed dataset, the performance of Teacher model is the worst, presumably because that the Teacher is trained using fully observed graph
Revised: fully observed graph

Original: [Figure]
Revised: [Figure]

Original: and may perform poorly once the privileged part of graph topology becomes unavailable at test time.
Revised: and may perform poorly once the privileged part of graph topology becomes unavailable at test time.

Original: 5.2 Scalability Test
Revised: 5.2 Scalability Test

Original: Training Time.
Revised: Training Time.

Original: We report the the training time per epoch of different methods in the left three panels of Fig.
Revised: We report the the training time per epoch of different methods in the left three panels of Fig.

Original: 4, where we use the whole graph for training on Cora and Citeseer [Figure] and set the batch size as 5 , 000 for Pubmed.
Revised: 4, where we use the whole graph for training on Cora and Citeseer, and set the batch size as 5 , 000 for Pubmed.

Original: While the computation complexity for distillation loss of GKD (in both non-parametric and parametric cases) is O ( dn 2 ) , we find that in practice their training time is not significantly worse than other KD baselines.
Revised: While the computation complexity for distillation loss of GKD (in both non-parametric and parametric cases) is O ( dn 2 ) , we find that in practice their training time is not significantly worse than other KD baselines.

Original: In specific, all variants of GKD take less than two times of the training time of vanilla KD while yielding better performance, and the simplest GKD variant using Sigmoid NHK is even more efficient than baselines in some cases.
Revised: In specific, all variants of GKD take less than two times of the training time of vanilla KD while yielding better performance, and the simplest GKD variant using Sigmoid NHK is even more efficient than baselines in some cases.

Original: It is also worth mentioning that the distilled student models are equally efficient by using sparser graph structure (in edge-aware setting) or smaller graph (in node-aware setting), which are suitable for being deployed in latency-constrained scenarios.
Revised: It is also worth mentioning that the distilled student models are equally efficient by using sparser graph structure (in edge-aware setting) or smaller graph (in node-aware setting), which are suitable for being deployed in latency-constrained scenarios.

Original: Convergence Speed.
Revised: Convergence Speed.

Original: We also compare the convergence speed of different variants of GKD and other methods in the right panel of Fig.
Revised: We also compare the convergence speed of different variants of GKD and other methods in the right panel of Fig.

Original: 4, where we use the same model architecture and optimizer for different methods and finetune other hyper-parameters to ensure fair comparison.
Revised: 4, where we use the same model architecture and optimizer for different methods and finetune other hyper-parameters to ensure fair comparison.

Original: Since the loss function is different for different methods, we report the convergence of training accuracy.
Revised: Since the loss function is different for different methods, we report the convergence of training accuracy.

Original: As shown by the figure, GKD variants using Sigmoid and Gaussian NHKs can converge as fast as the vanilla
Revised: As shown by the figure, GKD variants using Sigmoid and Gaussian NHKs can converge as fast as the vanilla

Original: KD.
Revised: KD.

Original: While GKD-R and PGKD in general take more epochs to converge, they can converge within 200 epochs (which is the default setting recommended in [27]) and the best validation accuracy is[Table] achieved within the first 100 epochs.
Revised: While GKD-R and PGKD in general take more epochs to converge, they can converge within 200 epochs (which is the default setting recommended in [28]) and the best validation accuracy is achieved within the first 100 epochs.

Original: We also find that the EM-style algorithm used for training
Revised: We also find that the EM-style algorithm used for training

Original: PGKD does not cost too much extra epochs for convergence than the non-parametric GKD which is reasonable since the learnable mapping g ϕ used in Eqn.
Revised: PGKD does not cost too much extra epochs for convergence than the non-parametric GKD which is reasonable since the learnable mapping g ϕ used in Eqn.

Original: (22) is an independent module that is not involved in GNN’s feed-forward computation
Revised: (22) is an independent module that is not involved in GNN’s feed-forward computation.

Original: [Table]
Revised: [Table]

Original: 5.3 Other Settings

Original: [Table]
Revised: [Table]

Original: Model Compression.
Revised: Model Compression.

Original: We report the performance of GKD in the conventional model compression setting [22] where the teacher and student are different-sized GNN models.
Revised: We report the performance of GKD in the conventional model compression setting [22] where the teacher and student are different-sized GNN models.

Original: In this setting, we train and test both teacher and student models on the complete graph.
Revised: In this setting, we train and test both teacher and student models on the complete graph.

Original: We use GCN as the backbone for teacher with a relatively large hidden size 64 , and use SGC or GCN (with hidden size 8 and 16 ) as the student model.
Revised: We use GCN as the backbone for teacher with a relatively large hidden size 64 , and use SGC or GCN (with hidden size 8 and 16 ) as the student model.

Original: We report the best result among all variants of GKD.
Revised: We report the best result among all variants of GKD.

Original: As shown in the first section of Tab. 5, our method achieves notable improvements over the student model, rendering it as a useful KD approach for model compression.
Revised: As shown in the first section of Tab. 5, our method achieves notable improvements over the student model, rendering it as a useful KD approach for model compression.

Original: Self-Distillation.
Revised: Self-Distillation.

Original: We further report results for the setting of self-distillation [15, 35], which is a special case of KD when the teacher’s and student’s architectures are identical, often used for refining their own performance.
Revised: We further report results for the setting of self-distillation [15, 36], which is a special case of KD when the teacher’s and student’s architectures are identical, often used for refining their own performance.

Original: The results as shown in Tab.
Revised: The results as shown in Tab.

Original: 5 validate that GKD could also be used to effectively boost GNN’s own performance.
Revised: 5 validate that GKD could also be used to effectively boost GNN’s own performance.

Original: Online Distillation.
Revised: Online Distillation.

Original: Table.
Revised: Table.

Original: 4 shows the performance of different variants of GKD for online distillation where both the teacher model and the student model are trained in an end-to-end manner, in contrast to offline distillation where the teacher model is pre-trained.
Revised: 4 shows the performance of different variants of GKD for online distillation where both the teacher model and the student model are trained in an end-to-end manner, in contrast to offline distillation where the teacher model is pre-trained.

Original: The results demonstrate the potential usage of GKD in this setting.
Revised: The results demonstrate the potential usage of GKD in this setting.

Original:  Conclusion, Current Limitations and Future Works
Revised:  Conclusion, Current Limitations and Future Works

Original: This paper formalizes the problem of graph topological knowledge transfer for GNNs.
Revised: This paper formalizes the problem of graph topological knowledge transfer for GNNs.

Original: We investigate on the implication of heat kernel in GNNs and propose the novel notion of neural heat kernel.
Revised: We investigate on the implication of heat kernel in GNNs and propose the novel notion of neural heat kernel.

Original: We leverage it to characterize the geometric property of the underlying manifold for graphs, and propose the framework of geometric knowledge distillation to transfer geometric knowledge from a teacher
Revised: We leverage it to characterize the geometric property of the underlying manifold for graphs, and propose the framework of geometric knowledge distillation to transfer geometric knowledge from a teacher

Original: GNN model to a student GNN.
Revised: GNN model to a student GNN.

Original: Experimental results validate the effectiveness of our approach in various practical settings.
Revised: Experimental results validate the effectiveness of our approach in various practical settings.

Original: Despite that the proposed GKD is effective in various tasks and possesses decent training efficiency in practice, its theoretical space and time complexities are O ( n 2 ) and O ( dn 2 ) and the parametric instantiation may take some extra time to converge compared to the pure non-KD counterpart.
Revised: Despite that the proposed GKD is effective in various tasks and possesses decent training efficiency in practice, its theoretical space and time complexities are O ( n 2 ) and O ( dn 2 ) and the parametric instantiation may take some extra time to converge compared to the pure non-KD counterpart.

Original: The algorithmic complexity can be reduced by using mini-batch training, and there also exist ways to reduce the overhead such as pre-computing teacher’s NHK matrix or using low-rank approximation.
Revised: The algorithmic complexity can be reduced by using mini-batch training, and there also exist ways to reduce the overhead such as pre-computing teacher’s NHK matrix or using low-rank approximation.

Original: Finally, we do not foresee any direct negative societal impacts of this work.
Revised: Finally, we do not foresee any direct negative societal impacts of this work.

Original: Acknowledgement
Revised: Acknowledgement

Original: This work was partly supported by National Key Research and Development Program of China (2020AAA0107600), National Natural Science Foundation of China (61972250, 72061127003), and
Revised: This work was partly supported by National Key Research and Development Program of China (2020AAA0107600), National Natural Science Foundation of China (61972250, 72061127003), and

Original: Shanghai Municipal Science and Technology (Major) Project (22511105100, 2021SHZDZX0102).
Revised: Shanghai Municipal Science and Technology (Major) Project (22511105100, 2021SHZDZX0102).

Original: A Proof for Theorem
Revised: A Proof for Theorem

Original: Suppose ( M , µ ) is the base manifold of dimension n with respect to graph G and GNN model f θ , and ∆ µ ( G , f θ ) is the associated weighted Laplace operator with respect to G and f θ (in the following shortened as ∆ µ ).
Revised: Suppose ( M , µ ) is the base manifold of dimension n with respect to graph G and GNN model f θ , and ∆ µ ( G , f θ ) is the associated weighted Laplace operator with respect to G and f θ (in the following shortened as ∆ µ ).

Original: We leverage a Sobolev inequality on manifolds as Lemma 1.
Revised: We leverage a Sobolev inequality on manifolds as Lemma 1.

Original: Lemma 1.
Revised: Lemma 1.

Original: [40] Let x be a function from the local Sobolev space W 2 σ loc ( M ) for a positive integer σ > n/ 4 .
Revised: [42] Let x be a function from the local Sobolev space W 2 σ loc ( M ) for a positive integer σ > n/ 4 .

Original: Then, for any relatively compact open set Ω ⊂ M and any set K ⋐ Ω , there is a constant
Revised: Then, for any relatively compact open set Ω ⊂ M and any set K ⋐ Ω , there is a constant

Original: C such that
Revised: C such that

Original: [Equation]
Revised: [Equation]

Original: where the norm ∥ · ∥ W 2 σ is defined as
Revised: where the norm ∥ · ∥ W 2 σ is defined as

Original: [Equation]
Revised: [Equation]

Original: Suppose σ is the smallest integer larger than n/ 4 , and P t = e − t L is the heat kernel semigroup, where
Revised: Suppose σ is the smallest integer larger than n/ 4 , and P t = e − t L is the heat kernel semigroup, where

Original: L = − ∆ µ | W 2 0 is the Dirichlet Laplace operator for the base manifold regarding graph G and GNN model f θ , we have the following lemma.
Revised: L = − ∆ µ | W 2 0 is the Dirichlet Laplace operator for the base manifold regarding graph G and GNN model f θ , we have the following lemma.

Original: Lemma 2.
Revised: Lemma 2.

Original: For any function x ∈ L 2 ( M ) , t > 0 , and set K ⋐ M , it holds that
Revised: For any function x ∈ L 2 ( M ) , t > 0 , and set K ⋐ M , it holds that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where C is a constant depending on K, g , µ, n .
Revised: where C is a constant depending on K, g , µ, n .

Original: Proof.
Revised: Proof.

Original: Suppose { E λ } is the spectral resolution of the Dirichlet operator L for the base manifold.
Revised: Suppose { E λ } is the spectral resolution of the Dirichlet operator L for the base manifold.

Original: Consider the function Φ( λ ) =
Revised: Consider the function Φ( λ ) =

Original: λ k e − tλ , where t > 0 and k ∈ Z + .
Revised: λ k e − tλ , where t > 0 and k ∈ Z + .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: Since Φ( λ ) is bounded on [0 , + ∞ ) , L k e − t L is also bounded, and hence we have L k (cid:0) e − t L x (cid:1) ∈
Revised: Since Φ( λ ) is bounded on [0 , + ∞ ) , L k e − t L is also bounded, and hence we have L k (cid:0) e − t L x (cid:1) ∈

Original: L 2 ( M ) .
Revised: L 2 ( M ) .

Original: By noticing that the function λ (cid:55)→ λ k e − tλ takes its maximal value at λ = k/t , we have, for any x ∈ L 2 ,
Revised: By noticing that the function λ (cid:55)→ λ k e − tλ takes its maximal value at λ = k/t , we have, for any x ∈ L 2 ,

Original: [Equation]
Revised: [Equation]

Original: Using the definition of ∥ · ∥ W 2 σ in Eqn.
Revised: Using the definition of ∥ · ∥ W 2 σ in Eqn.

Original: (25) and the result of Eqn.
Revised: (25) and the result of Eqn.

Original: (28), we obtain
Revised: (28), we obtain

Original: [Equation]
Revised: [Equation]

Original: Substituting P t x into Lemma.
Revised: Substituting P t x into Lemma.

Original: 1 and using the result of Eqn.
Revised: 1 and using the result of Eqn.

Original: 29 yields Eqn. 26, and thus completes the proof.
Revised: 29 yields Eqn. 26, and thus completes the proof.

Original: Consider the following equivalent expressions describing heat diffusion and feature propagation respectively,
Revised: Consider the following equivalent expressions describing heat diffusion and feature propagation respectively,

Original: [Equation]
Revised: [Equation]

Original: where τ is a constant dependent on f θ and l .
Revised: where τ is a constant dependent on f θ and l .

Original: We can rewrite the single-layer NHK as a function p τ,v i such that for any v j ∈ V (or equivalently v j ∈ M )
Revised: We can rewrite the single-layer NHK as a function p τ,v i such that for any v j ∈ V (or equivalently v j ∈ M )

Original: [Equation]
Revised: [Equation]

Original: We define g : V → R d as a function that outputs the feature of a node in l -th layer, which is clearly
Revised: We define g : V → R d as a function that outputs the feature of a node in l -th layer, which is clearly

Original: Lebesgue integrable by thinking of nodes embedded on the manifold, i.e., g ∈ L 2 ( M ) .
Revised: Lebesgue integrable by thinking of nodes embedded on the manifold, i.e., g ∈ L 2 ( M ) .

Original: Proof for the existence of unique single-layer NHK boils down to proof that for any v i ∈ M , l >(i.e., τ > 0 ), there exists a unique function p τ,v i ∈
Revised: Proof for the existence of unique single-layer NHK boils down to proof that for any v i ∈ M , l >(i.e., τ > 0 ), there exists a unique function p τ,v i ∈

Original: L 2 ( M ) such that, for all g θ ,
Revised: L 2 ( M ) such that, for all g θ ,

Original: [Equation]
Revised: [Equation]

Original: Fix a relatively compact set K ⋐ M .
Revised: Fix a relatively compact set K ⋐ M .

Original: By Lemma 2, for all τ > 0 and g θ ∈
Revised: By Lemma 2, for all τ > 0 and g θ ∈

Original: L 2 ( M ) , the function
Revised: L 2 ( M ) , the function

Original: P τ g θ ( v i ) admits the estimate
Revised: P τ g θ ( v i ) admits the estimate

Original: [Equation]
Revised: [Equation]

Original: Therefore, for fixed l and GNN model f θ , the mapping g θ (cid:55)→ P
Revised: Therefore, for fixed l and GNN model f θ , the mapping g θ (cid:55)→ P

Original: τ
Revised: τ

Original: g
Revised: g

Original: θ is a bounded linear functional on
Revised: θ is a bounded linear functional on

Original: L 2 ( M ) .
Revised: L 2 ( M ) .

Original: By the Riesz representation theorem, there exists a function p τ,v
Revised: By the Riesz representation theorem, there exists a function p τ,v

Original: i ∈
Revised: i ∈

Original: L 2 ( M ) such that
Revised: L 2 ( M ) such that

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where ( · , · ) L 2 denotes inner product in L 2 , whence Eqn.
Revised: where ( · , · ) L 2 denotes inner product in L 2 , whence Eqn.

Original: (32) follows.
Revised: (32) follows.

Original: The uniqueness of p τ,v i is evident from Eqn.
Revised: The uniqueness of p τ,v i is evident from Eqn.

Original: (32) since for any point v
Revised: (32) since for any point v

Original: i ∈ M there is a compact set K containing v i , the function p τ,v is defined for all τ > 0 and v ∈ M .
Revised: i ∈ M there is a compact set K containing v i , the function p τ,v is defined for all τ > 0 and v ∈ M .

Original: B Proof for Theorem
Revised: B Proof for Theorem

Original: Lemma 3.
Revised: Lemma 3.

Original: For all v i , v j ∈ M and t > 0 , the inner product ( p s,x , p t − s,y ) does not depend on s ∈ (0 , t ] .
Revised: For all v i , v j ∈ M and t > 0 , the inner product ( p s,x , p t − s,y ) does not depend on s ∈ (0 , t ] .

Original: Proof.
Revised: Proof.

Original: Using P t + s = P s P t (by the definition of P t ), Eqn.
Revised: Using P t + s = P s P t (by the definition of P t ), Eqn.

Original: (34), and the symmetry of P t , we obtain that for all v ∈ M , t, s > 0 , and g θ ∈
Revised: (34), and the symmetry of P t , we obtain that for all v ∈ M , t, s > 0 , and g θ ∈

Original: L 2 ( M ) , it holds that
Revised: L 2 ( M ) , it holds that

Original: [Equation]
Revised: [Equation]

Original: Lemma 3 implies that, for all v i , v j ∈ M and 0 < s ≤ t p t ( v i , v k )
Revised: Lemma 3 implies that, for all v i , v j ∈ M and 0 < s ≤ t p t ( v i , v k )

Original: =
Revised: =

Original: ( p s,v i
Revised: ( p s,v i

Original: [Equation]
Revised: [Equation]

Original: p t − s
Revised: p t − s

Original: [Equation]
Revised: [Equation]

Original: v k )
Revised: v k )

Original: [Equation]
Revised: [Equation]

Original: Hence, it holds that
Revised: Hence, it holds that

Original: [Equation]
Revised: [Equation]

Original: By letting
Revised: By letting

Original: [Equation]
Revised: [Equation]

Original: (38) we can rewrite Eqn.
Revised: (38) we can rewrite Eqn.

Original: (37) as the expression of semigroup identity property of layer-wise NHK
Revised: (37) as the expression of semigroup identity property of layer-wise NHK

Original: [Equation]
Revised: [Equation]

Original: The proof also generalizes to the cross-layer case, inducing cross-layer NHK κ θ ( v i , v j , l (cid:55)→ l + k ) .
Revised: The proof also generalizes to the cross-layer case, inducing cross-layer NHK κ θ ( v i , v j , l (cid:55)→ l + k ) .

Original: C Proof for Theorem
Revised: C Proof for Theorem

Original: Let { φ k } ∞ k =1 be an orthonormal basis of eigenfunctions of L with an increasing sequence of nonnegative eigenvalues { λ k } ∞ k =1 , where λ k → + ∞ .
Revised: Let { φ k } ∞ k =1 be an orthonormal basis of eigenfunctions of L with an increasing sequence of nonnegative eigenvalues { λ k } ∞ k =1 , where λ k → + ∞ .

Original: In consideration of
Revised: In consideration of

Original: [Equation]
Revised: [Equation]

Original: we have the following expansion of p t,v i by referring to results in literature [45]
Revised: we have the following expansion of p t,v i by referring to results in literature [47]

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Let T be the accumulated time interval from l -th layer to ( l + k ) -th layer, in consideration of the equivalence shown in Eqn.
Revised: Let T be the accumulated time interval from l -th layer to ( l + k ) -th layer, in consideration of the equivalence shown in Eqn.

Original: (31), we could write Eqn.
Revised: (31), we could write Eqn.

Original: (41) as
Revised: (41) as

Original: [Equation]
Revised: [Equation]

Original: completing the proof.
Revised: completing the proof.

Original: D Justification for Parametric GKD
Revised: D Justification for Parametric GKD

Original: We justify parametric GKD from a variational inference perspective.
Revised: We justify parametric GKD from a variational inference perspective.

Original: From Eqn.
Revised: From Eqn.

Original: (18), the forward
Revised: (18), the forward

Original: GNN model f θ defines a model distribution
Revised: GNN model f θ defines a model distribution

Original: [Equation]
Revised: [Equation]

Original: where p θ ( K | H ( l ) ) is intractable, hindering the proceeding distillation.
Revised: where p θ ( K | H ( l ) ) is intractable, hindering the proceeding distillation.

Original: In this light, the variational inverse-NHK model κ † ϕ is proposed with a variational distribution
Revised: In this light, the variational inverse-NHK model κ † ϕ is proposed with a variational distribution

Original: [Equation]
Revised: [Equation]

Original: which has a tractable posterior q ϕ ( K | H ( l + k ) ) .
Revised: which has a tractable posterior q ϕ ( K | H ( l + k ) ) .

Original: Now, we justify our training scheme with iterative optimization for Eqn.
Revised: Now, we justify our training scheme with iterative optimization for Eqn.

Original: (22) and (23) by the following proposition.
Revised: (22) and (23) by the following proposition.

Original: Proposition 1.
Revised: Proposition 1.

Original: The optimization in Eqn.
Revised: The optimization in Eqn.

Original: (22) and (23) essentially minimizes the following Kullback–Leibler (KL) divergence,
Revised: (22) and (23) essentially minimizes the following Kullback–Leibler (KL) divergence,

Original: [Equation]
Revised: [Equation]

Original: and hence attempts to establish equivalence between two latent variable models p θ and q ϕ .
Revised: and hence attempts to establish equivalence between two latent variable models p θ and q ϕ .

Original: Proof.
Revised: Proof.

Original: In the following, we use X , Y , K to denote H ( l ) , H ( l + k ) , K ( G , l (cid:55)→ l + k ) for brevity.
Revised: In the following, we use X , Y , K to denote H ( l ) , H ( l + k ) , K ( G , l (cid:55)→ l + k ) for brevity.

Original: By definition in Section 4.2 we have a forward GNN model f θ with the joint distribution of latent variables
Revised: By definition in Section 4.2 we have a forward GNN model f θ with the joint distribution of latent variables

Original: [Equation]
Revised: [Equation]

Original: and a variational inverse-NHK model κ † ϕ with the joint distribution of latent variables
Revised: and a variational inverse-NHK model κ † ϕ with the joint distribution of latent variables

Original: [Equation]
Revised: [Equation]

Original: Based on these equations, we can write the KL-divergence between joint distributions of p θ and q ϕ as
Revised: Based on these equations, we can write the KL-divergence between joint distributions of p θ and q ϕ as

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: The first term C is a constant entropy with respect to q ϕ .
Revised: The first term C is a constant entropy with respect to q ϕ .

Original: The second term is the KL-divergence between the variational posterior q ϕ ( X | Y , K ) and the prior p θ ( X ) , which corresponds to the reconstruction loss in Eqn.
Revised: The second term is the KL-divergence between the variational posterior q ϕ ( X | Y , K ) and the prior p θ ( X ) , which corresponds to the reconstruction loss in Eqn.

Original: (22) that attempts to learn a NHK that faithfully reflects the latent heat diffusion process describing the GNN feature propagation.
Revised: (22) that attempts to learn a NHK that faithfully reflects the latent heat diffusion process describing the GNN feature propagation.

Original: The third term is negative log-likelihood, which corresponds to the prediction loss in Eqn.
Revised: The third term is negative log-likelihood, which corresponds to the prediction loss in Eqn.

Original: (23) that attempts to fit the dataset.
Revised: (23) that attempts to fit the dataset.

Original: Therefore, minimizing this KL-divergence amounts to the iterative optimization scheme of Eqn.
Revised: Therefore, minimizing this KL-divergence amounts to the iterative optimization scheme of Eqn.

Original: (22) and (23).
Revised: (22) and (23).

Original: E More Discussions on the Equivalence of Eqn. 
Revised: E More Discussions on the Equivalence of Eqn.

Original: The equivalence of two equations in Eqn.
Revised: The equivalence of two equations in Eqn.

Original: 7 is based on recent works [8, 7, 48, 13, 12] that built connection between heat equation and GNN.
Revised: 7 is based on recent works [8, 7, 50, 13, 12] that built connection between heat equation and GNN.

Original: The main result of these works is that by treating node features H as signal X (corresponding to x ( u, t ) in heat equation Eqn. 2) on the graph, solving the heat equation with Euler scheme yields the formulation of a GNN layer.
Revised: The main result of these works is that by treating node features H as signal X (corresponding to x ( u, t ) in heat equation Eqn. 2) on the graph, solving the heat equation with Euler scheme yields the formulation of a GNN layer.

Original: In other words, the GNN can be seen as the discretisations of the continuous diffusion process described by the heat equation.
Revised: In other words, the GNN can be seen as the discretisations of the continuous diffusion process described by the heat equation.

Original: Correspondingly in Eqn.
Revised: Correspondingly in Eqn.

Original: 7, the left equation is a general GNN layer (corresponding to discretized diffusion process), and the right equation is directly derived from Eqn.
Revised: 7, the left equation is a general GNN layer (corresponding to discretized diffusion process), and the right equation is directly derived from Eqn.

Original: 7 (corresponding to continuous diffusion process).
Revised: 7 (corresponding to continuous diffusion process).

Original: Moreover, different definitions of Laplace-Beltrami operator ∆ yield different GNNs.
Revised: Moreover, different definitions of Laplace-Beltrami operator ∆ yield different GNNs.

Original: To be more specific, the simplest definition [8, 48] of ∆ is by letting the gradient operator ∇ denote assigning each edge the difference of adjacent nodes’ features, and the divergence operator ∇ ∗ denote the summation of edge features obtained from last step.
Revised: To be more specific, the simplest definition [8, 50] of ∆ is by letting the gradient operator ∇ denote assigning each edge the difference of adjacent nodes’ features, and the divergence operator ∇ ∗ denote the summation of edge features obtained from last step.

Original: In this case, setting τ in Eqn.
Revised: In this case, setting τ in Eqn.

Original: 2 as 1 yields
Revised: 2 as 1 yields

Original: SGC [50], and since it is being pointed out to be a potential cause of over-smoothing (which is one example to show the thermodynamic and geometric perspective is helpful for understanding GNNs), the authors in [48] further set smaller τ which yields DGC.
Revised: SGC [52], and since it is being pointed out to be a potential cause of over-smoothing (which is one example to show the thermodynamic and geometric perspective is helpful for understanding GNNs), the authors in [50] further set smaller τ which yields DGC.

Original: Reinterpreting ∇ ∗ as weighted sum of edge features yields GAT [8, 46] and any other GNN following the message passing scheme:
Revised: Reinterpreting ∇ ∗ as weighted sum of edge features yields GAT [8, 48] and any other GNN following the message passing scheme:

Original: [Equation]
Revised: [Equation]

Original: where N u denotes neighbored nodes of u , and s denotes a parametric or non-parametric similarity score.
Revised: where N u denotes neighbored nodes of u , and s denotes a parametric or non-parametric similarity score.

Original: The neighborhood summation could also be modified by changing the definition of ∇ , which yields GNNs based on learned structures (e.g., GDC [29]) and those with residual links (e.g.,
Revised: The neighborhood summation could also be modified by changing the definition of ∇ , which yields GNNs based on learned structures (e.g., GDC [30]) and those with residual links (e.g.,

Original: APPNP [28]).
Revised: APPNP [29]).

Original: Further considering different discretization schemes (e.g., implicit scheme, multistep schemes) yields different variants of GRAND [8, 7, 42] that are found more powerful than its simplification (presumably because it better matches the continuous diffusion process).
Revised: Further considering different discretization schemes (e.g., implicit scheme, multistep schemes) yields different variants of GRAND [8, 7, 44] that are found more powerful than its simplification (presumably because it better matches the continuous diffusion process).

Original: Unfortunately, not all GNNs have a simple form of ∆ , and for some of them, whether there exists such ∆ is an open research question.
Revised: Unfortunately, not all GNNs have a simple form of ∆ , and for some of them, whether there exists such ∆ is an open research question.

Original: Therefore, we write the operator as ∆( f θ , G ) to associate it with model
Revised: Therefore, we write the operator as ∆( f θ , G ) to associate it with model

Original: f θ and use equivalence in Eqn. 2 as an analytical assumption.
Revised: f θ and use equivalence in Eqn. 2 as an analytical assumption.

Original: The thermodynamic and geometric perspective used in the paper is fundamental and useful for studying the geometry property of GNNs, which has also been adopted by other works [3, 43, 48, 42, 8, 7].
Revised: The thermodynamic and geometric perspective used in the paper is fundamental and useful for studying the geometry property of GNNs, which has also been adopted by other works [3, 45, 50, 44, 8, 7].

Original: For example, [3] attempts to explain heterophily and oversmoothing problems in GNNs by connecting GNNs to the heat equation defined by the sheaf Laplace-Beltrami operator ∆ F and propose new GNN architectures based on the theory.
Revised: For example, [3] attempts to explain heterophily and oversmoothing problems in GNNs by connecting GNNs to the heat equation defined by the sheaf Laplace-Beltrami operator ∆ F and propose new GNN architectures based on the theory.

Original: Similarly, [42, 8, 7] propose new GNN architectures / rewiring methods from the same thermodynamic perspective, [48] explains over-smoothing issue of SGC based on the same equivalence of GNN and solve of heat equation, and [43] explains the over-squashing issue from a geometric perspective by analyzing the curvature.
Revised: Similarly, [44, 8, 7] propose new GNN architectures / rewiring methods from the same thermodynamic perspective, [50] explains over-smoothing issue of SGC based on the same equivalence of GNN and solve of heat equation, and [45] explains the over-squashing issue from a geometric perspective by analyzing the curvature.

Original: A common trait of these works is to draw analogies between GNNs and differential geometry / diffusion process to obtain meaningful theoretical results (ours: NHK as a characterization of GNN’s underlying geometry) that are used to guide implementation (ours: GKD for geometric knowledge transfer).
Revised: A common trait of these works is to draw analogies between GNNs and differential geometry / diffusion process to obtain meaningful theoretical results (ours: NHK as a characterization of GNN’s underlying geometry) that are used to guide implementation (ours: GKD for geometric knowledge transfer).

Original: F Implementation Details
Revised: F Implementation Details

Original: We present implementation details for our experiments for reproducibility.
Revised: We present implementation details for our experiments for reproducibility.

Original: We implement our model as well as the baselines with Python 3.7, Pytorch 1.8 and Pytorch Geometric 1.7.
Revised: We implement our model as well as the baselines with Python 3.7, Pytorch 1.8 and Pytorch Geometric 1.7.

Original: All experiments are conducted on NVIDIA V100 with 16 GB memory.
Revised: All experiments are conducted on NVIDIA V100 with 16 GB memory.

Original: All parameters are initialized with Xavier initialization procedure.
Revised: All parameters are initialized with Xavier initialization procedure.

Original: We train the model by Adam optimizer.
Revised: We train the model by Adam optimizer.

Original: Both teacher and student models are trained from scratch.
Revised: Both teacher and student models are trained from scratch.

Original: For the main results reported in Tab. 1 and 2, we choose the backbone model as a 3 -layer GCN with hidden size 32 .
Revised: For the main results reported in Tab. 1 and 2, we choose the backbone model as a 3 -layer GCN with hidden size 32 .

Original: In case that the graph is too large, we use mini-batch training (draw a mini-batch of nodes from the vertex set V ) for computing the distillation loss.
Revised: In case that the graph is too large, we use mini-batch training (draw a mini-batch of nodes from the vertex set V ) for computing the distillation loss.

Original: All the experiments are repeated five times with random initialization.
Revised: All the experiments are repeated five times with random initialization.

Original: F.1 Dataset Description
Revised: F.1 Dataset Description

Original: We choose three benchmark citation network datasets, i
Revised: We choose three benchmark citation network datasets, i

Original: [Table]
Revised: [Table]

Original: e., Cora, Citeseer and Pubmed, and a largescale network dataset OGB-Arxivfor node classification.
Revised: e., Cora, Citeseer and Pubmed, and a largescale network dataset OGB-Arxivfor node classification.

Original: For citation networks, we randomly split them into train/valid/test data according to the ratio 2:1:1.
Revised: For citation networks, we randomly split them into train/valid/test data according to the ratio 2:1:1.

Original: For OGB dataset, we follow the original splitting [23] for evaluation.
Revised: For OGB dataset, we follow the original splitting [23] for evaluation.

Original: The statistics of these datasets are summarized in Tab.
Revised: The statistics of these datasets are summarized in Tab.

Original: [Table]
Revised: [Table]

Original: F.2 Hyper-Parameter Tuning
Revised: F.2 Hyper-Parameter Tuning

Original: For parameter tuning, we adopt grid search method to search for hyper-parameters on validation set.
Revised: For parameter tuning, we adopt grid search method to search for hyper-parameters on validation set.

Original: Since our model is only sensitive to α , one can use other more efficient searching strategies instead of grid search (e
Revised: Since our model is only sensitive to α , one can use other more efficient searching strategies instead of grid search (e

Original: [Table]
Revised: [Table]

Original: g., coordinate descent, Bayesian optimization) to achieve very similar results.
Revised: g., coordinate descent, Bayesian optimization) to achieve very similar results.

Original: The descriptions for several hyper-parameters and their search spaces are shown in Tab.
Revised: The descriptions for several hyper-parameters and their search spaces are shown in Tab.

Original: [Table]
Revised: [Table]

Original: and its variants.
Revised: and its variants.

Original: F.3 Implementation of non-parametric GKD
Revised: F.3 Implementation of non-parametric GKD

Original: For Gauss-Weierstrass NHK in Eqn.
Revised: For Gauss-Weierstrass NHK in Eqn.

Original: (13), we treat the accumulated time interval T as a hyperparameter that is consistent for all layers.
Revised: (13), we treat the accumulated time interval T as a hyperparameter that is consistent for all layers.

Original: For sigmoid NHK in Eqn.
Revised: For sigmoid NHK in Eqn.

Original: (13), we let a = 1 , b = 0 and the NHK is simple dot-product with activation.
Revised: (13), we let a = 1 , b = 0 and the NHK is simple dot-product with activation.

Original: For randomized NHK in Eqn.
Revised: For randomized NHK in Eqn.

Original: (16), we let s = 

Revised: (16), we let s =

Original: d for the random transformation matrix W , and we use tanh ( · ) as activation function for σ ( · ) .
Revised: d for the random transformation matrix W , and we use tanh ( · ) as activation function for σ ( · ) .

Original: Input: Complete graph ˜ G with labels ˜
Revised: Input: Complete graph ˜ G with labels ˜

Original: Y , partial graph G with labels Y , learning rate γ , initial parameters θ ∗ , θ .
Revised: Y , partial graph G with labels Y , learning rate γ , initial parameters θ ∗ , θ .

Original: Training Teacher GNN : while Not converged do
Revised: Training Teacher GNN : while Not converged do

Original: Teacher model conducts feature propagation on ˜ G
Revised: Teacher model conducts feature propagation on ˜ G

Original: Calculate L pre ( ˆ Y θ ∗ , ˜ Y ) as in Eqn.
Revised: Calculate L pre ( ˆ Y θ ∗ , ˜ Y ) as in Eqn.

Original: (17) θ ∗ ←
Revised: (17) θ ∗ ←

Original: [Equation]
Revised: [Equation]

Original: ˜ Y )
Revised: ˜ Y )

Original: Save teacher model as f θ ∗
Revised: Save teacher model as f θ ∗

Original: Training Student GNN :
Revised: Training Student GNN :

Original: Load teacher model f θ ∗ while Not converged do
Revised: Load teacher model f θ ∗ while Not converged do

Original: Teacher model conducts feature propagation on ˜ G
Revised: Teacher model conducts feature propagation on ˜ G

Original: Student model conducts feature propagation on G
Revised: Student model conducts feature propagation on G

Original: Calculate L pre and L dis as in Eqn.
Revised: Calculate L pre and L dis as in Eqn.

Original: [Equation]
Revised: [Equation]

Original: F.4 Implementation of parametric GKD
Revised: F.4 Implementation of parametric GKD

Original: For the case of parametric GKD, we realize the non-linear mapping g ϕ as a neural network one-layer feed-forward neural network with tanh ( · ) activation, and set s = 2 d , the same as the non-parametric setting.
Revised: For the case of parametric GKD, we realize the non-linear mapping g ϕ as a neural network one-layer feed-forward neural network with tanh ( · ) activation, and set s = 2 d , the same as the non-parametric setting.

Original: In case that the teacher model has larger hidden size than the student model, we follow standard approaches in feature-based knowledge distillation methods [21] and use independent mappings for teacher and student models that are customized for their own hidden sizes.
Revised: In case that the teacher model has larger hidden size than the student model, we follow standard approaches in feature-based knowledge distillation methods [21] and use independent mappings for teacher and student models that are customized for their own hidden sizes.

Original: We use an EM-style algorithm for training the student model as in Eqn.
Revised: We use an EM-style algorithm for training the student model as in Eqn.

Original: (22) and Eqn.
Revised: (22) and Eqn.

Original: For variants of
Revised: For variants of

Original: GKD+KD and PGKD+KD, we consider additional standard label-based distillation loss in [22] with respect to labeled nodes, inducing an additional hyper-parameter α kd that controls its importance.
Revised: GKD+KD and PGKD+KD, we consider additional standard label-based distillation loss in [22] with respect to labeled nodes, inducing an additional hyper-parameter α kd that controls its importance.

Original: F.5 Descriptions and Implementations of Baselines
Revised: F.5 Descriptions and Implementations of Baselines

Original: KD [22]: is the seminal work of knowledge distillation, which uses the predictions of the teacher model as soft labels to teach the student model.
Revised: KD [22]: is the seminal work of knowledge distillation, which uses the predictions of the teacher model as soft labels to teach the student model.

Original: FitNets [39]: is the representative work of feature-based knowledge distillation, using the intermediate layers of training instances to teach the student model.
Revised: FitNets [41]: is the representative work of feature-based knowledge distillation, using the intermediate layers of training instances to teach the student model.

Original: FSP [60]: is a relation-based knowledge distillation method, using the Gram matrix between two intermediate layers to explore the relationships between different feature maps.
Revised: FSP [62]: is a relation-based knowledge distillation method, using the Gram matrix between two intermediate layers to explore the relationships between different feature maps.

Original: LSP [59]: is a knowledge distillation method for graph convolutional networks, which aims to match the local distribution (similarity with adjacent nodes) of teacher and student models.
Revised: LSP [61]: is a knowledge distillation method for graph convolutional networks, which aims to match the local distribution (similarity with adjacent nodes) of teacher and student models.

Original: Note that some of these baselines (FitNets and FSP) are originally designed specifically for computer vision tasks, we adapt them to the setting of graph neural networks with slight modifications.
Revised: Note that some of these baselines (FitNets and FSP) are originally designed specifically for computer vision tasks, we adapt them to the setting of graph neural networks with slight modifications.

Original: While
Revised: While

Original: FSP is designed for graph neural networks, its original formulation of distillation loss does not account for the difference of graph topology for teacher and student models.
Revised: FSP is designed for graph neural networks, its original formulation of distillation loss does not account for the difference of graph topology for teacher and student models.

Original: To make it compatible with the settings considered in this paper, we fix the definition of local structures with respect to either student or teacher graph topology, and choose the best result for report.
Revised: To make it compatible with the settings considered in this paper, we fix the definition of local structures with respect to either student or teacher graph topology, and choose the best result for report.

Original: We refer to the hyper-parameter settings in their papers and also finetune them on different datasets.
Revised: We refer to the hyper-parameter settings in their papers and also finetune them on different datasets.

Original: Input:
Revised: Input

Original: Training Teacher GNN : while Not converged do
Revised: Training Teacher GNN : while Not converged do

Original: Teacher model conducts feature propagation on ˜ G
Revised: Teacher model conducts feature propagation on ˜ G

Original: Calculate L pre ( ˆ Y θ ∗ , ˜ Y ) as in Eqn.
Revised: Calculate L pre ( ˆ Y θ ∗ , ˜ Y ) as in Eqn.

Original: (17) θ ∗ ← θ ∗ − γ 1 ∇ θ ∗ L pre ( ˆ Y θ , ˜ Y )
Revised: (17) θ ∗ ← θ ∗ − γ 1 ∇ θ ∗ L pre ( ˆ Y θ , ˜ Y )

Original: Save teacher model as f θ ∗
Revised: Save teacher model as f θ ∗

Original: Training Student GNN :
Revised: Training Student GNN :

Original: Load teacher model f θ ∗ while Not converged do
Revised: Load teacher model f θ ∗ while Not converged do

Original: Teacher model conducts feature propagation on ˜ G
Revised: Teacher model conducts feature propagation on ˜ G

Original: Optimization for ϕ
Revised: Optimization for ϕ

Original: Student model conducts feature propagation on G
Revised: Student model conducts feature propagation on G

Original: Calculate reconstruction loss L rec as in Eqn.
Revised: Calculate reconstruction loss L rec as in Eqn.

Original: (22) ϕ
Revised: (22) ϕ

Original: ← ϕ − γ 2 ∇ ϕ L rec
Revised: ← ϕ − γ 2 ∇ ϕ L rec

Original: Optimization for θ
Revised: Optimization for θ

Original: Calculate L pre and L dis as in Eqn.
Revised: Calculate L pre and L dis as in Eqn.

Original: θ ← θ − γ 1 ∇ θ (cid:16) L pre ( ˆ Y θ ∗ , Y ) + α L dis (cid:17)
Revised: θ ← θ − γ 1 ∇ θ (cid:16) L pre ( ˆ Y θ ∗ , Y ) + α L dis (cid:17)

Original: F.6 Comparisons with Baselines
Revised: G More Related Works

Revised: Knowledge Distillation.

Original: There are mainly four different types of distilling strategies [17], namely response-based KD [22] (which uses output layer of the teacher model to teach student), feature-based KD [39, 62, 21] (which matches intermediate layers of teacher and student), relation-based KD [60, 61] (which aligns the relationship between different layers or samples), and graph-based KD [57, 59] (which considers the graph information or designed for GNN).
Revised: There are mainly four different types of distilling strategies [17], namely response-based KD [22] (which uses output layer of the teacher model to teach student), featurebased KD [41, 64, 21, 26] (which matches intermediate layers of teacher and student), relation-based KD [62, 63, 39, 60] (which aligns the relationship between different layers or samples), and graphbased KD [58, 61] (which considers the graph information or designed for GNN).

Original: For experiments, we choose representative method from each category as baselines.
Revised: For experiments, we choose representative method from each category as baselines.

Original: While these existing distillation strategies have shown remarkable success in distillation tasks such as model compression, they rarely (carefully) study the role of graph geometry in GNN iterations.
Revised: While these existing distillation strategies have shown remarkable success in distillation tasks such as model compression, they rarely (carefully) study the role of graph geometry in GNN iterations.

Original: For geometric knowledge transfer task, it is crucial to find a fundamental and principled way to track how graph topology affects the behavior of a specific GNN.
Revised: For geometric knowledge transfer task, it is crucial to find a fundamental and principled way to track how graph topology affects the behavior of a specific GNN.

Original: Therefore, we first probe the intersection between KD and geometric learning, propose NHK to characterize geometric knowledge and propose different variants of GKD that are shown to be effective especially in the geometric knowledge transfer setting. 
Revised: Therefore, we first probe the intersection between KD and geometric learning, propose NHK to characterize geometric knowledge and propose different variants of GKD that are shown to be effective especially in the geometric knowledge transfer setting. Generalization of GNNs.

Revised: Recent advances shed lights on the generalization ability of GNNs from various perspectives, e.g., the in-distribution generalization error [16], extrapolation capability [57], out-of-distribution generalization under distribution shifts [54].

Revised: Our work can be seen a specific embodiment for GNN generalization w.r.t.

Revised: topological domain transfer.

Revised: Furthermore, there are some recent studies exploring learning under the varied data space between training and inference [53], using GNNs as an encoding and reasoning tool.

Revised: GKD focuses on transferring across varied structural information and aims at compressing topological information for GNNs.