Original: Geometric Distillation for Graph Networks
Revised: Geometric Distillation for Graph Networks

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: We study a new paradigm of knowledge transfer in the context of geometric deeplearning, which aims at distilling knowledge from a teacher graph neural network(GNN) model trained on a large graph to a student GNN model operating on asmaller graph.
Revised: We study a new paradigm of knowledge transfer in the context of geometric deeplearning, which aims at distilling knowledge from a teacher graph neural network(GNN) model trained on a large graph to a student GNN model operating on asmaller graph.

Original: To this end, we revisit the connection between thermodynamicsand the behavior of GNN, based on which we propose Neural Heat Kernel (NHK)to encapsulate the geometric property of the underlying manifold concerning thearchitecture of GNN.
Revised: To this end, we revisit the connection between thermodynamicsand the behavior of GNN, based on which we propose Neural Heat Kernel (NHK)to encapsulate the geometric property of the underlying manifold concerning thearchitecture of GNN.

Original: A natural solution is derived by analysing and aligning NHKson teacher and student models, dubbed as Geometric Knowledge Distillation.
Revised: A natural solution is derived by analysing and aligning NHKson teacher and student models, dubbed as Geometric Knowledge Distillation.

Original: We develop non- and parametric instantiations and demonstrate their efﬁcacy invarious experimental settings for knowledge distillation regarding different typesof privileged topological information and teacher-student schemes.
Revised: We develop non- and parametric instantiations and demonstrate their efﬁcacy invarious experimental settings for knowledge distillation regarding different typesof privileged topological information and teacher-student schemes.

Original:  Introduction
Revised:  Introduction

Original: Modern graph neural networks (GNNs) [23, 39, 43] have shown remarkable performance in learningrepresentations for structured instances.
Revised: Modern graph neural networks (GNNs) [23, 39, 43] have shown remarkable performance in learningrepresentations for structured instances.

Original: From the perspective of geometric deep learning [4, 3, 30],much of the achievement of GNNs can be attributed to their successful implementation of thepermutation invariance property as geometric priors 1 into the architecture design.
Revised: From the perspective of geometric deep learning [4, 3, 30],much of the achievement of GNNs can be attributed to their successful implementation of thepermutation invariance property as geometric priors 1 into the architecture design.

Original: Nevertheless, inpractice, GNNs highly rely on graph topology, as essential input information, to explore the relationalknowledge implicit in interactions of instance pairs throughout the entire message passing process,termed as geometric knowledge in this paper.
Revised: Nevertheless, inpractice, GNNs highly rely on graph topology, as essential input information, to explore the relationalknowledge implicit in interactions of instance pairs throughout the entire message passing process,termed as geometric knowledge in this paper.

Original: As advances in generalized distillation [25, 37] revealthe possibility of encoding input features into model construction, natural questions arise as to:
Revised: As advances in generalized distillation [25, 37] revealthe possibility of encoding input features into model construction, natural questions arise as to:

Original: Is it possible, and if so, how can we encode graph topology as a special type of ‘geometric prior’ intoa GNN model, such that the model could precisely capture the underlying geometric knowledge evenwithout full graph topology as input?.
Revised: Is it possible, and if so, how can we encode graph topology as a special type of ‘geometric prior’ intoa GNN model, such that the model could precisely capture the underlying geometric knowledge evenwithout full graph topology as input?.

Original: In speciﬁc, we are interested in the following geometric knowledge transfer problem: a GNN model(with node-speciﬁc outputs for node-level prediction [19]) is exposed with a partial graph, which is asubset of the complete graph.
Revised: In speciﬁc, we are interested in the following geometric knowledge transfer problem: a GNN model(with node-speciﬁc outputs for node-level prediction [19]) is exposed with a partial graph, which is asubset of the complete graph.

Original: Formally speaking, we have notations:
Revised: Formally speaking, we have notations:

Original: G = {V , E} (partial graph) , ˜ G = { ˜ V , ˜ E} (complete graph) , where V ✓ ˜ V , E ✓ {V ⇥ V} \ ˜ E .
Revised: G = {V , E} (partial graph) , ˜ G = { ˜ V , ˜ E} (complete graph) , where V ✓ ˜ V , E ✓ {V ⇥ V} \ ˜ E .

Original: Our goal is to transfer or encode geometric knowledge extracted from ˜ G to the target GNN model thatis only aware of G .
Revised: Our goal is to transfer or encode geometric knowledge extracted from ˜ G to the target GNN model thatis only aware of G .

Original: Besides of academic interest, studying this problem is also of much practical value.
Revised: Besides of academic interest, studying this problem is also of much practical value.

Original: As a non-exhaustive list of applications: improving efﬁciency without compromising on effectivenessfor coarsened graphs [11, 20, 52], privacy constrained scenarios in social recommenders or federatedlearning where the complete graph is unavailable [26, 40, 51], promoting concentration on targetedcommunity to bring up economic beneﬁts [44]. 
Revised: As a non-exhaustive list of applications: improving efﬁciency without compromising on effectivenessfor coarsened graphs [11, 20, 52], privacy constrained scenarios in social recommenders or federatedlearning where the complete graph is unavailable [26, 40, 51], promoting concentration on targetedcommunity to bring up economic beneﬁts [44]. A B H P O M R V

Revised: W

Original: (cid:36) (cid:37) (cid:43) (cid:51) (cid:50) (cid:48) (cid:53) (cid:57) (cid:58) (cid:49) (cid:38) (cid:39) (cid:40) (cid:47) (cid:42) (cid:56) (cid:54) (cid:41) (cid:60) (cid:59) (cid:55) (cid:52) (cid:44) (cid:45) (cid:46) (cid:36) (cid:50) (cid:48) (cid:53) (cid:57) (cid:58) (cid:49) (cid:39) (cid:40) (cid:47) (cid:42) (cid:56) (cid:54) (cid:41) (cid:36) (cid:37) (cid:43) (cid:51) (cid:50) (cid:48) (cid:53) (cid:57) (cid:58) (cid:49) (cid:38) (cid:39) (cid:40) (cid:47) (cid:42) (cid:56) (cid:54) (cid:41) (cid:55) (cid:52) (cid:44) (cid:45) (cid:46) (cid:68)(cid:12)(cid:3)(cid:55)(cid:72)(cid:68)(cid:70)(cid:75)(cid:72)(cid:85) (cid:70)(cid:12)(cid:3)(cid:54)(cid:87)(cid:88)(cid:71)(cid:72)(cid:81)(cid:87) (cid:3) (cid:11)(cid:68)(cid:73)(cid:87)(cid:72)(cid:85)(cid:3)(cid:42)(cid:46)(cid:39)(cid:12) (cid:69)(cid:12)(cid:3)(cid:54)(cid:87)(cid:88)(cid:71)(cid:72)(cid:81)(cid:87) (cid:3) (cid:11)(cid:69)(cid:72)(cid:73)(cid:82)(cid:85)(cid:72)(cid:3)(cid:42)(cid:46)(cid:39)(cid:12)
Revised: N C D E L

Revised: G

Revised: U S F

Revised: Y X T Q

Revised: I J

Revised: K

Revised: A

Revised: O

Revised: M

Revised: R V

Revised: W

Revised: N

Revised: D E

Revised: L

Revised: G

Revised: U S F

Revised: A B

Revised: H P O

Revised: M R V

Revised: W

Revised: N

Revised: C

Revised: D E

Revised: L

Revised: G

Revised: U S F

Revised: T Q

Original:  Achieving this target is non-trivial for that we need to ﬁrst answer “what is geometric knowledgeand how to translate it into an explicit form", which requires in-depth investigation on the roleof graph topology throughout the progressive process of message passing.
Revised: I J K [Figure] Achieving this target is non-trivial for that we need to ﬁrst answer “what is geometric knowledgeand how to translate it into an explicit form", which requires in-depth investigation on the roleof graph topology throughout the progressive process of message passing.

Original: Therefore, we take athermodynamic view borrowed from physics and propose a new methodology built upon recentadvances revealing the connection between heat diffusion and architectures of GNNs [7, 41, 6].
Revised: Therefore, we take athermodynamic view borrowed from physics and propose a new methodology built upon recentadvances revealing the connection between heat diffusion and architectures of GNNs [7, 41, 6].

Original: Speciﬁcally, we interpret feature propagation as heat ﬂows on the underlying Riemmanian manifold,whose characteristics (that are dependent on graph topology and the GNN model) pave the way for aprincipled representation of the latent geometric knowledge.
Revised: Speciﬁcally, we interpret feature propagation as heat ﬂows on the underlying Riemmanian manifold,whose characteristics (that are dependent on graph topology and the GNN model) pave the way for aprincipled representation of the latent geometric knowledge.

Original: 1.1 Our Contributions
Revised: 1.1 Our Contributions

Original: New theoretical perspective for analyzing latent graph geometry.
Revised: New theoretical perspective for analyzing latent graph geometry.

Original: On top of the well-establishedconnection between heat equation and GNNs, we step further to inspect the implication of heat kernelfor existing GNNs, and propose a novel notion of Neural Heat Kernel (NHK) with rigorous proofof its existence.
Revised: On top of the well-establishedconnection between heat equation and GNNs, we step further to inspect the implication of heat kernelfor existing GNNs, and propose a novel notion of Neural Heat Kernel (NHK) with rigorous proofof its existence.

Original: Heat kernel intrinsically deﬁnes the unique solution to the heat equation and canbe a fundamental characterization for the geometric property of the underlying manifold [14, 15].
Revised: Heat kernel intrinsically deﬁnes the unique solution to the heat equation and canbe a fundamental characterization for the geometric property of the underlying manifold [14, 15].

Original: Likewise, NHK uncovers geometric property of the latent graph manifold for GNNs, and governs howinformation ﬂows between pairs of instances.
Revised: Likewise, NHK uncovers geometric property of the latent graph manifold for GNNs, and governs howinformation ﬂows between pairs of instances.

Original: It lends us a tractable mathematical tool to understandwhat geometric knowledge the GNN model has learned or extracted from the graph topology.
Revised: It lends us a tractable mathematical tool to understandwhat geometric knowledge the GNN model has learned or extracted from the graph topology.

Original: Flexible distillation framework with versatile instantiations.
Revised: Flexible distillation framework with versatile instantiations.

Original: Based on the insights mentionedabove, we treat NHK matrices as representation of the latent geometric knowledge, upon which webuild a ﬂexible and principled distillation framework dubbed as Geometric Knowledge Distillation(GKD), which aims at encoding and transferring geometric knowledge by aligning latent manifoldsbehind GNN models as illustrated in Fig.
Revised: Based on the insights mentionedabove, we treat NHK matrices as representation of the latent geometric knowledge, upon which webuild a ﬂexible and principled distillation framework dubbed as Geometric Knowledge Distillation(GKD), which aims at encoding and transferring geometric knowledge by aligning latent manifoldsbehind GNN models as illustrated in Fig.

Original: Moreover, we develop non-parametirc and parametricversions of GKD, in terms of different ways to approximate NHK computation.
Revised: Moreover, we develop non-parametirc and parametricversions of GKD, in terms of different ways to approximate NHK computation.

Original: Speciﬁcally, theformer derives explicit NHKs via assumptions on latent space, and the later learns NHK in a datadriven manner through the lens of variational inference, which will be shown to match two models byminimizing the KL-divergence of their distributions.
Revised: Speciﬁcally, theformer derives explicit NHKs via assumptions on latent space, and the later learns NHK in a datadriven manner through the lens of variational inference, which will be shown to match two models byminimizing the KL-divergence of their distributions.

Original: Promising results for geometric knowledge transfer and conventional KD purposes.
Revised: Promising results for geometric knowledge transfer and conventional KD purposes.

Original: We verifythe practical efﬁcacy of GKD in terms of different geometric knowledge types (i.e., edge-aware andnode-aware ones), and further show its effectiveness for conventional KD purposes (e.g., modelcompression, self-distillation).
Revised: We verifythe practical efﬁcacy of GKD in terms of different geometric knowledge types (i.e., edge-aware andnode-aware ones), and further show its effectiveness for conventional KD purposes (e.g., modelcompression, self-distillation).

Original: We highlight that our methods consistently exceed teacher model andeven rival with the oracle model that gives the performance upper bound in principle.
Revised: We highlight that our methods consistently exceed teacher model andeven rival with the oracle model that gives the performance upper bound in principle.

Original: 1.2 Links to Related Works
Revised: 1.2 Links to Related Works

Original: Geometric Deep Learning.
Revised: Geometric Deep Learning.

Original: The study of geometric deep learning [4, 30] provides fundamentalprinciples and methodology to generalize deep learning methods to non-Euclidean domains (e.g.,graphs and manifolds).
Revised: The study of geometric deep learning [4, 30] provides fundamentalprinciples and methodology to generalize deep learning methods to non-Euclidean domains (e.g.,graphs and manifolds).

Original: From this perspective, architectures for off-the-shelf GNNs [23, 39, 43, 16]have naturally incorporated the geometric prior knowledge for graphs such as permutation invariance .
Revised: From this perspective, architectures for off-the-shelf GNNs [23, 39, 43, 16]have naturally incorporated the geometric prior knowledge for graphs such as permutation invariance .

Original: Despite their remarkable success, they highly rely on the graph topology.
Revised: Despite their remarkable success, they highly rely on the graph topology.

Original: This work extends theidea of geometric deep learning by treating the global graph topology as a special type of priorknowledge, and attempts to encode it into GNNs themselves, such that the trained model wouldleverage information from the global graph topology even without explicitly taking it as input.
Revised: This work extends theidea of geometric deep learning by treating the global graph topology as a special type of priorknowledge, and attempts to encode it into GNNs themselves, such that the trained model wouldleverage information from the global graph topology even without explicitly taking it as input.

Original: Graph-Based Knowledge Distillation.
Revised: Graph-Based Knowledge Distillation.

Original: Knowledge distillation (KD)
Revised: Knowledge distillation (KD)

Original: [18, 1] uses the outputs of ateacher model as alternative supervised signals to teach a student model, with various new paradigmsincluding feature-based [33, 50] and relation-based [49] ones.
Revised: [18, 1] uses the outputs of ateacher model as alternative supervised signals to teach a student model, with various new paradigmsincluding feature-based [33, 50] and relation-based [49] ones.

Original: While some prior arts [8, 47, 40, 48,46] attempted to combine KD and GNNs, i.e., graph-based KD, they are nearly straight-forwardadaptations of KD without in-depth investigation on the role of graph topology, also restricted by a speciﬁc choice of GNN architecture or application scenarios, e.g., model compression.
Revised: While some prior arts [8, 47, 40, 48,46] attempted to combine KD and GNNs, i.e., graph-based KD, they are nearly straight-forwardadaptations of KD without in-depth investigation on the role of graph topology, also restricted by a speciﬁc choice of GNN architecture or application scenarios, e.g., model compression.

Original: In contrast, weﬁrst formalize the problem of geometric knowledge transfer, theoretically answer the question “howto represent graph geometric knowledge and encode it into GNN models", and propose geometricdistillation approach based on the theoretical results, which is shown to be effective in varioussettings.
Revised: In contrast, weﬁrst formalize the problem of geometric knowledge transfer, theoretically answer the question “howto represent graph geometric knowledge and encode it into GNN models", and propose geometricdistillation approach based on the theoretical results, which is shown to be effective in varioussettings.

Original:  Preliminaries
Revised:  Preliminaries

Original: We commence with a brief detour to heat equation on Riemannian manifolds, and its connection withmodern GNN architectures.
Revised: We commence with a brief detour to heat equation on Riemannian manifolds, and its connection withmodern GNN architectures.

Original: Moreover, we bring forth the notion of heat kernel to motivate this work.
Revised: Moreover, we bring forth the notion of heat kernel to motivate this work.

Original: Heat Equation on Manifolds.
Revised: Heat Equation on Manifolds.

Original: We are interested in heat equation deﬁned on a smooth k -dimensional
Revised: We are interested in heat equation deﬁned on a smooth k -dimensional

Original: Riemannian manifold M .
Revised: Riemannian manifold M .

Original: Suppose the manifold is associated with a scalar- or vector-valued functionx ( u, t ) :
Revised: Suppose the manifold is associated with a scalar- or vector-valued functionx ( u, t ) :

Original: M ⇥ [0 , 1 ) !
Revised: M ⇥ [0 , 1 ) !

Original: R d , quantifying a speciﬁc type of signals such as heat at a point u 2 M 88 and time t .
Revised: R d , quantifying a speciﬁc type of signals such as heat at a point u 2 M 88 and time t .

Original: Fourier’s law of heat conductivity describes the ﬂow of heat with respect to time andspace, via a partial differential equation (PDE) called heat equation [5], i.e.,
Revised: Fourier’s law of heat conductivity describes the ﬂow of heat with respect to time andspace, via a partial differential equation (PDE) called heat equation [5], i.e.,

Original: [Equation]
Revised: [Equation]

Original: where c > 0 is the thermal conductivity coefﬁcient, and  is the natural Laplace–Beltrami operatorassociated with M .
Revised: where c > 0 is the thermal conductivity coefﬁcient, and  is the natural Laplace–Beltrami operatorassociated with M .

Original: Rewriting  as the functional composition of the divergence operator r ⇤ and 92 gradient operator r , i.e.,  = r
Revised: Rewriting  as the functional composition of the divergence operator r ⇤ and 92 gradient operator r , i.e.,  = r

Original: ⇤  r , we can interpret the heat equation as: the variation of 93 temperature within an inﬁnitesimal time interval at a point is equivalent to the divergence between itsown temperature and the average temperature on an inﬁnitesimal sphere around it.
Revised: ⇤  r , we can interpret the heat equation as: the variation of 93 temperature within an inﬁnitesimal time interval at a point is equivalent to the divergence between itsown temperature and the average temperature on an inﬁnitesimal sphere around it.

Original: Implications on Graphs.
Revised: Implications on Graphs.

Original: A spatial discretisation of a continuous manifold yields a graph G ={V , E} , whose nodes can be thought of as embedded on the base manifold.
Revised: A spatial discretisation of a continuous manifold yields a graph G ={V , E} , whose nodes can be thought of as embedded on the base manifold.

Original: In fact, the heat equation 97 along with variants thereof (e.g., Schrödinger equation) have found widespread use in modeling graphdynamics [10, 21, 28].
Revised: In fact, the heat equation 97 along with variants thereof (e.g., Schrödinger equation) have found widespread use in modeling graphdynamics [10, 21, 28].

Original: More importantly, it has been recently revealed to be intimately related withthe architectures of modern GNNs [41, 7, 6]: suppose X (0) =
Revised: More importantly, it has been recently revealed to be intimately related withthe architectures of modern GNNs [41, 7, 6]: suppose X (0) =

Original: { x ( u, 0) } u 2 V 2 R n ⇥ d denotes theinitial condition for Eqn.
Revised: { x ( u, 0) } u 2 V 2 R n ⇥ d denotes theinitial condition for Eqn.

Original: (2) determined by input node features, then solving the heat equation undercertain deﬁnitions of r ⇤ and r (i.e., deﬁnition of  ) amounts to different architectures of GNNs.
Revised: (2) determined by input node features, then solving the heat equation undercertain deﬁnitions of r ⇤ and r (i.e., deﬁnition of  ) amounts to different architectures of GNNs.

Original: For instance:
Revised: For instance:

Original: Example 1.
Revised: Example 1.

Original: [41] Deﬁne the discretised counterpart of  as the graph Laplacian matrix L =e
Revised: [41] Deﬁne the discretised counterpart of  as the graph Laplacian matrix L =e

Original: [Equation]
Revised: [Equation]

Original: 
Revised:

Original: [Equation]
Revised: [Equation]

Original: e
Revised: e

Original: D  12 .
Revised: D  12 .

Original: Numerically solving Eqn.
Revised: Numerically solving Eqn.

Original: (2) using the forward Euler method with step size⌧ = 1 yields the formulation of Simple Graph Convolution (SGC) [43]
Revised: (2) using the forward Euler method with step size⌧ = 1 yields the formulation of Simple Graph Convolution (SGC) [43]

Original: [Equation]
Revised: [Equation]

Original: Example 2.
Revised: Example 2.

Original: [7] Deﬁne the gradient operator r ij as the difference of source and target node features,the divergence operator
Revised: [7] Deﬁne the gradient operator r ij as the difference of source and target node features,the divergence operator

Original: r
Revised: r

Original: ⇤ i as the sum of features of all edges for the node.
Revised: ⇤ i as the sum of features of all edges for the node.

Original: Numerically solving
Revised: Numerically solving

Original: Eqn.
Revised: Eqn.

Original: (2) using the explicit Euler scheme with step size ⌧ yields the following recursive formulation
Revised: (2) using the explicit Euler scheme with step size ⌧ yields the following recursive formulation

Original: [Equation]
Revised: [Equation]

Original: (4) where G is a diffusivity coefﬁcient matrix in place of c .
Revised: (4) where G is a diffusivity coefﬁcient matrix in place of c .

Original: Moreover, stacking a non-linear transformation layer after each step yields the formulation of Graph
Revised: Moreover, stacking a non-linear transformation layer after each step yields the formulation of Graph

Original: Convolution Networks (GCN) [23] for Eqn.
Revised: Convolution Networks (GCN) [23] for Eqn.

Original: (3), Graph Attention Networks (GAT) [39] with residualconnection for Eqn.
Revised: (3), Graph Attention Networks (GAT) [39] with residualconnection for Eqn.

Original: (4), and even more GNN architectures by virtue of the ﬂexibility of interpretionfor heat equation on graphs.
Revised: (4), and even more GNN architectures by virtue of the ﬂexibility of interpretionfor heat equation on graphs.

Original: Heat Kernels Intriguingly, it turns out that the initial value problem of heat equation on any manifold
Revised: Heat Kernels Intriguingly, it turns out that the initial value problem of heat equation on any manifold

Original: M has a smallest positive fundamental solution depending on the Laplace operator  , known as theheat kernel [2].
Revised: M has a smallest positive fundamental solution depending on the Laplace operator  , known as theheat kernel [2].

Original: It is denoted as a kernel function  ( x, y, t ) , such that
Revised: It is denoted as a kernel function  ( x, y, t ) , such that

Original: [Equation]
Revised: [Equation]

Original: where µ is a non-negative measure associated with M .
Revised: where µ is a non-negative measure associated with M .

Original: In physics, the heat kernel  ( x, y, t ) can 118 be interpreted as a transition density that describes the asymptotic behavior of a natural Brownian motion on the manifold.
Revised: In physics, the heat kernel  ( x, y, t ) can 118 be interpreted as a transition density that describes the asymptotic behavior of a natural Brownian motion on the manifold.

Original: Its formulation thus can be treated as a unique reﬂection or representation of 120 the geometry of the underlying manifold .
Revised: Its formulation thus can be treated as a unique reﬂection or representation of 120 the geometry of the underlying manifold .

Original: For example, if the manifold is a k -dimensional Euclidean
Revised: For example, if the manifold is a k -dimensional Euclidean

Original: Space R k or a Hyperbolic Space H k , the explicit formula of heat kernel is respectively given by, ( u i , u j , t ) =
Revised: Space R k or a Hyperbolic Space H k , the explicit formula of heat kernel is respectively given by, ( u i , u j , t ) =

Original: (4 ⇡ t ) k/ 2 exp ✓  ⇢4 t ◆ and  ( u i , u j , t ) =
Revised: (4 ⇡ t ) k/ 2 exp ✓  ⇢4 t ◆ and  ( u i , u j , t ) =

Original: (  1) m 2 m ⇡ m (4 ⇡ t ) 12 ✓ sinh ⇢ @ @⇢ ◆
Revised: (  1) m 2 m ⇡ m (4 ⇡ t ) 12 ✓ sinh ⇢ @ @⇢ ◆

Original: m e  m 2 t  ⇢ 2 4 t , (6) where ⇢ =
Revised: m e  m 2 t  ⇢ 2 4 t , (6) where ⇢ =

Original: d ( u i , u j ) denote geodesic distance.
Revised: d ( u i , u j ) denote geodesic distance.

Original: Heat kernel has also been adopted for graph-relatedapplications such as community detection [24], graph clustering [45].
Revised: Heat kernel has also been adopted for graph-relatedapplications such as community detection [24], graph clustering [45].

Original:  Extending Heat Kernel to GNNs
Revised:  Extending Heat Kernel to GNNs

Original: The starting point of this work is the development of neural heat kernel , built upon the previouslymentioned connection of GNNs and heat equation.
Revised: The starting point of this work is the development of neural heat kernel , built upon the previouslymentioned connection of GNNs and heat equation.

Original: As will be discussed later, this novel notion lendsus a thermodynamic perspective to the intrinsic geometric property of the latent graph manifoldembodied in GNNs, and hence paves the way for distilling geometric knowledge.
Revised: As will be discussed later, this novel notion lendsus a thermodynamic perspective to the intrinsic geometric property of the latent graph manifoldembodied in GNNs, and hence paves the way for distilling geometric knowledge.

Original: 3.1 Neural Heat Kernel
Revised: 3.1 Neural Heat Kernel

Original: Consider the graph signal X ( t ) at time t and node features H ( l ) at layer l as interchangeable notions.
Revised: Consider the graph signal X ( t ) at time t and node features H ( l ) at layer l as interchangeable notions.

Original: Consequently, feature propagation using one layer of GNN amounts to heat diffusion on the basemanifold M within a certain time interval ⌧ , leading to the equivalences of X ( t + ⌧ ) and H ( l +1) : 
Revised: Consequently, feature propagation using one layer of GNN amounts to heat diffusion on the basemanifold M within a certain time interval ⌧ , leading to the equivalences of X ( t + ⌧ ) and H ( l +1) : [Equation]

Original: [Equation]
Revised: [Equation]

Original: (7) where f ✓ denotes an arbitrary GNN model with parameter ✓ , and  ( f ✓ , G ) denotes a generalizationof Laplace-Beltrami operator deﬁned over the base manifold M associated with graph G and thearbitrary backbone GNN model f ✓ .
Revised: where f ✓ denotes an arbitrary GNN model with parameter ✓ , and  ( f ✓ , G ) denotes a generalizationof Laplace-Beltrami operator deﬁned over the base manifold M associated with graph G and thearbitrary backbone GNN model f ✓ .

Original: In light of this connection, we consider a natural generalization of heat kernel for GNNs, termed asneural heat kernel (NHK) to highlight its difference with heat kernel in the thermodynamic context.
Revised: In light of this connection, we consider a natural generalization of heat kernel for GNNs, termed asneural heat kernel (NHK) to highlight its difference with heat kernel in the thermodynamic context.

Original: In particular, a single-layer NHK is deﬁned as a positive deﬁnite symmetric kernel function denotedas  ( l ) ✓ ( v i , v j ) , where the sub-script ✓ implies that it is associated with the architecture and parametersof the backbone GNN, and the super-script ( l ) implies that it is speciﬁc to each layer, analogous tothe role of continuous time t in Eqn.
Revised: In particular, a single-layer NHK is deﬁned as a positive deﬁnite symmetric kernel function denotedas  ( l ) ✓ ( v i , v j ) , where the sub-script ✓ implies that it is associated with the architecture and parametersof the backbone GNN, and the super-script ( l ) implies that it is speciﬁc to each layer, analogous tothe role of continuous time t in Eqn.

Original: Theorem 1.
Revised: Theorem 1.

Original: (Existence of Single-Layer NHK)
Revised: (Existence of Single-Layer NHK)

Original: Suppose two expressions in Eqn.
Revised: Suppose two expressions in Eqn.

Original: (7) are equivalent,then for any graph G and GNN model f ✓ , there exist a unique single-layer NHK function  ( l ) ✓ ( · ) suchthat for any node v i 2 V and l > 0 ,
Revised: (7) are equivalent,then for any graph G and GNN model f ✓ , there exist a unique single-layer NHK function  ( l ) ✓ ( · ) suchthat for any node v i 2 V and l > 0 ,

Original: [Equation]
Revised: [Equation]

Original: where h ( l ) i 2 R d denotes the feature of node v i at l -th layer, and µ is a measure over vertices speciﬁedas the inverse of node degree
Revised: where h ( l ) i 2 R d denotes the feature of node v i at l -th layer, and µ is a measure over vertices speciﬁedas the inverse of node degree

Original: 1 /d i .
Revised: 1 /d i .

Original: To push further, we can generalize NHK across multiple layers of GNN, termed as a cross-layer NHK ✓ ( v i , v j , l 7! l + k ) (e.g., from l -th layer to ( l + k ) -th layer of GNN).
Revised: To push further, we can generalize NHK across multiple layers of GNN, termed as a cross-layer NHK ✓ ( v i , v j , l 7! l + k ) (e.g., from l -th layer to ( l + k ) -th layer of GNN).

Original: Its existence could be inducedrecursively by the semi-group identity property of NHK concerning consecutive GNN layers.
Revised: Its existence could be inducedrecursively by the semi-group identity property of NHK concerning consecutive GNN layers.

Original: Theorem 2.
Revised: Theorem 2.

Original: (Semigroup Identity Property of NHK)
Revised: (Semigroup Identity Property of NHK)

Original: The NHK satisﬁes the semigroup identityproperty: 8 v i , v j 2 V and l > 0 , there exists a cross-layer NHK across two consecutive layers
Revised: The NHK satisﬁes the semigroup identityproperty: 8 v i , v j 2 V and l > 0 , there exists a cross-layer NHK across two consecutive layers

Original: [Equation]
Revised: [Equation]

Original: This theorem indicates that stacks of multiple GNN layers also constitute a valid kernel, i.e.,
Revised: This theorem indicates that stacks of multiple GNN layers also constitute a valid kernel, i.e.,

Original: [Equation]
Revised: [Equation]

Original: Analogous to heat kernel as an unique characterization of the underlying space, NHK characterizes thegeometric property of the latent graph manifold for GNNs.
Revised: Analogous to heat kernel as an unique characterization of the underlying space, NHK characterizes thegeometric property of the latent graph manifold for GNNs.

Original: Additionally, NHK is dependent on GNNmodels through the deﬁnition of the associated Laplace-Beltrami operator  ( f ✓ , G ) , inheriting theexpressiveness of neural networks and varying through the course of training.
Revised: Additionally, NHK is dependent on GNNmodels through the deﬁnition of the associated Laplace-Beltrami operator  ( f ✓ , G ) , inheriting theexpressiveness of neural networks and varying through the course of training.

Original: Intuitively, NHK canbe thought of as a model-driven encoding for topological information, encapsulating the geometricknowledge learned by GNNs into a tractable functional form.
Revised: Intuitively, NHK canbe thought of as a model-driven encoding for topological information, encapsulating the geometricknowledge learned by GNNs into a tractable functional form.

Original: 3.2 Application in Geometric Distillation
Revised: 3.2 Application in Geometric Distillation

Original: Consider the problem of distilling geometric knowledge, which involves an intelligent teacher modelf ✓ ⇤ , which is exposed to and pre-trained over the (relatively) complete graph ˜ G =
Revised: Consider the problem of distilling geometric knowledge, which involves an intelligent teacher modelf ✓ ⇤ , which is exposed to and pre-trained over the (relatively) complete graph ˜ G =

Original: (˜ V , ˜ E ) , and astudent model f ✓ that is exposed to the partial graph G =
Revised: (˜ V , ˜ E ) , and astudent model f ✓ that is exposed to the partial graph G =

Original: ( V , E ) , where V ✓ ˜ V and E ✓ {V ⇥ V} \ ˜ E .
Revised: ( V , E ) , where V ✓ ˜ V and E ✓ {V ⇥ V} \ ˜ E .

Original: 163 Our target is to train a student model (with the help of teacher model) that operates on
Revised: 163 Our target is to train a student model (with the help of teacher model) that operates on

Original: G to be ascompetitive as models operating on ˜ G during inference.
Revised: G to be ascompetitive as models operating on ˜ G during inference.

Original: Since G is a sub-graph of ˜ G , they should lie inthe same space (i.e., latent manifold) governed by the underlying mechanism of data generation, andhence we expect student and teacher models to capture the same geometric property of this sharedspace.
Revised: Since G is a sub-graph of ˜ G , they should lie inthe same space (i.e., latent manifold) governed by the underlying mechanism of data generation, andhence we expect student and teacher models to capture the same geometric property of this sharedspace.

Original: This leads to the principle of Geometric Knowledge Distillation (GKD): transfer the geometricknowledge of the intelligent teacher to the student such that the student can propagate features as if itis aware of the complete graph topology (see the example in Fig. 1).
Revised: This leads to the principle of Geometric Knowledge Distillation (GKD): transfer the geometricknowledge of the intelligent teacher to the student such that the student can propagate features as if itis aware of the complete graph topology (see the example in Fig. 1).

Original: To this end, we resort to NHK matrices on the teacher (resp.
Revised: To this end, we resort to NHK matrices on the teacher (resp.

Original: student) model over the complete (resp.partial) graph as instantiations of their geometric knowledge, denoted as
Revised: student) model over the complete (resp.partial) graph as instantiations of their geometric knowledge, denoted as

Original: [Equation]
Revised: [Equation]

Original: written compactly as K ( l +1) ( G ) when k = 1 .
Revised: written compactly as K ( l +1) ( G ) when k = 1 .

Original: The NHK matrix is a positive semi-deﬁnite symmetricmatrix, and alike  , is dependent on the GNN model f ✓ and graph G .
Revised: The NHK matrix is a positive semi-deﬁnite symmetricmatrix, and alike  , is dependent on the GNN model f ✓ and graph G .

Original: Denote K ( l ) ✓ ⇤ , V ( ˜ G ) 2 R |V| ⇥ |V|as the sub-matrix of K ( l ) ✓ ⇤ ( ˜ G ) with row and column indices in V .
Revised: Denote K ( l ) ✓ ⇤ , V ( ˜ G ) 2 R |V| ⇥ |V|as the sub-matrix of K ( l ) ✓ ⇤ ( ˜ G ) with row and column indices in V .

Original: The distillation loss for GKD is
Revised: The distillation loss for GKD is

Original: [Equation]
Revised: [Equation]

Original: where d( · , · ) is a similarity measure, for which we choose Frobenius distance as implementation, i.e.,
Revised: where d( · , · ) is a similarity measure, for which we choose Frobenius distance as implementation, i.e.,

Original: [Equation]
Revised: [Equation]

Original: where W 2 R |V| ⇥ |V| is a weighting matrix to trade-off distillation loss with respect to different nodepairs depending on their connectivity.
Revised: where W 2 R |V| ⇥ |V| is a weighting matrix to trade-off distillation loss with respect to different nodepairs depending on their connectivity.

Original: For k = 1 , the loss can be re-written as L ( l +1) dis
Revised: For k = 1 , the loss can be re-written as L ( l +1) dis

Original: ( K ✓ ⇤ , V , K ✓ ) .
Revised: ( K ✓ ⇤ , V , K ✓ ) .

Original: Note that one can also specify different k for teacher and student models in Eqn.
Revised: Note that one can also specify different k for teacher and student models in Eqn.

Original: (11) in case when theteacher model is deeper.
Revised: (11) in case when theteacher model is deeper.

Original: We also study the impact of different similarity measures in Appendix F.1.
Revised: We also study the impact of different similarity measures in Appendix F.1.

Original:  Instantiations for Geometric Knowledge Distillation
Revised:  Instantiations for Geometric Knowledge Distillation

Original: Unfortunately, deriving explicit formulas for NHKs is prohibitively challenging due to introduction ofnon-linearity.
Revised: Unfortunately, deriving explicit formulas for NHKs is prohibitively challenging due to introduction ofnon-linearity.

Original: To circumvent it, we propose two types of instantiations for GKD, i.e., non-parametricand parametric.
Revised: To circumvent it, we propose two types of instantiations for GKD, i.e., non-parametricand parametric.

Original: The former considers explicit NHKs by making assumptions on the underlying space,and the latter resorts to variational inference techniques to learn NHK in a data-driven manner.
Revised: The former considers explicit NHKs by making assumptions on the underlying space,and the latter resorts to variational inference techniques to learn NHK in a data-driven manner.

Original: 4.1 Non-Parametric Geometric Distillation
Revised: 4.1 Non-Parametric Geometric Distillation

Original: Deterministic Kernel.
Revised: Deterministic Kernel.

Original: One instantiation of NHK is a Gauss-Weierstrass kernel in the form of
Revised: One instantiation of NHK is a Gauss-Weierstrass kernel in the form of

Original: Eqn.
Revised: Eqn.

Original: (6), by assuming the underlying space is a Euclidean space.
Revised: (6), by assuming the underlying space is a Euclidean space.

Original: Since the distillation loss in
Revised: Since the distillation loss in

Original: Eqn.
Revised: Eqn.

Original: (11) is a homogeneous function, we can remove its scaling factor and deﬁne NHK as(Gauss-Weierstrass
Revised: (11) is a homogeneous function, we can remove its scaling factor and deﬁne NHK as(Gauss-Weierstrass

Original: [Equation]
Revised: [Equation]

Original: where T denotes the estimation of the accumulated time interval.
Revised: where T denotes the estimation of the accumulated time interval.

Original: Alternatively, we can use Sigmoidkernel and deﬁne non-parametric NHK as:
Revised: Alternatively, we can use Sigmoidkernel and deﬁne non-parametric NHK as:

Original: [Equation]
Revised: [Equation]

Original: where a, b are positive constants depending on l and k .
Revised: where a, b are positive constants depending on l and k .

Original: It is a natural and intuitive choice as similaritymeasurement and empirically found as-well effective, albeit does not correspond to any namedmanifold to our knowledge.
Revised: It is a natural and intuitive choice as similaritymeasurement and empirically found as-well effective, albeit does not correspond to any namedmanifold to our knowledge.

Original: Randomized Kernel.
Revised: Randomized Kernel.

Original: We can also deﬁne Randomized kernel based on the following theorem.
Revised: We can also deﬁne Randomized kernel based on the following theorem.

Original: Theorem 3.
Revised: Theorem 3.

Original: (Expansion of NHK) Let { ' k 0 } 1 k 0 =
Revised: (Expansion of NHK) Let { ' k 0 } 1 k 0 =

Original: be orthonormal basis of eigenfunctions of  ( f ✓ , G ) with eigenvalues 0 <  0   1   2  . .
Revised: be orthonormal basis of eigenfunctions of  ( f ✓ , G ) with eigenvalues 0 <  0   1   2  . .

Original: . , NHK allows the expansion:
Revised: . , NHK allows the expansion:

Original: [Equation]
Revised: [Equation]

Original: Based on this result, we resort to the approximation of NHK by deﬁning a randomized kernel in asimilar form as Eqn.
Revised: Based on this result, we resort to the approximation of NHK by deﬁning a randomized kernel in asimilar form as Eqn.

Original: (15), leading to the following formulation of randomized NHK:
Revised: (15), leading to the following formulation of randomized NHK:

Original: [Equation]
Revised: [Equation]

Original: where  ( W k 0 h i ) is used to proximate ' k 0 ( v i ) , W k 

Revised: where  ( W k 0 h i ) is used to proximate ' k 0 ( v i ) , W k

Original: =
Revised: =

Original: [  1 ,k 0 ,  2 ,k 0 , · · · ,  s,k 0 ] > is a transformation 200 matrix,  ⇠ N ( 0 , I d ) is a d -dimensional random variable from Gaussian distribution.
Revised: [  1 ,k 0 ,  2 ,k 0 , · · · ,  s,k 0 ] > is a transformation 200 matrix,  ⇠ N ( 0 , I d ) is a d -dimensional random variable from Gaussian distribution.

Original: In fact, undercertain choice of activation function  , Eqn.
Revised: In fact, undercertain choice of activation function  , Eqn.

Original: (16) could approximate a diversity of kernels [32, 9].
Revised: (16) could approximate a diversity of kernels [32, 9].

Original: Thisdesign essentially enforces the alignment of teacher and student for arbitrary underlying manifold.
Revised: Thisdesign essentially enforces the alignment of teacher and student for arbitrary underlying manifold.

Original: Training Scheme.
Revised: Training Scheme.

Original: We follow the standard training paradigm in KD literature [18, 13]: the teacheris pre-trained by a supervised prediction loss involving all labeled nodes in ˜ V .
Revised: We follow the standard training paradigm in KD literature [18, 13]: the teacheris pre-trained by a supervised prediction loss involving all labeled nodes in ˜ V .

Original: After teacher iswell-trained, we ﬁx ✓ ⇤ and train the student model according to
Revised: After teacher iswell-trained, we ﬁx ✓ ⇤ and train the student model according to

Original: [Equation]
Revised: [Equation]

Original: where Y denotes ground-truth labels of labeled nodes in V , and ˆ Y ✓ denotes the predictions of studentmodel f ✓ on G , L dis is the distillation loss deﬁned by Eqn.
Revised: where Y denotes ground-truth labels of labeled nodes in V , and ˆ Y ✓ denotes the predictions of studentmodel f ✓ on G , L dis is the distillation loss deﬁned by Eqn.

Original: (11), L denotes the total number of layers.
Revised: (11), L denotes the total number of layers.

Original: 4.2 Parametric Geometric Distillation
Revised: 4.2 Parametric Geometric Distillation

Original: Inheriting the similar spirit of auto-encoding Bayes [22], we introduce a variational inverse-NHK thatis independently parameterized, denoted as  †  , whose existence is guaranteed by the invertibility of
Revised: Inheriting the similar spirit of auto-encoding Bayes [22], we introduce a variational inverse-NHK thatis independently parameterized, denoted as  †  , whose existence is guaranteed by the invertibility of

Original: NHK matrices.
Revised: NHK matrices.

Original: Together with  ✓ , they deﬁne a symmetric form characterizing feature propagation:
Revised: Together with  ✓ , they deﬁne a symmetric form characterizing feature propagation:

Original: [Equation]
Revised: [Equation]

Original: In practice, we follow existing kernel learning approaches [42] and use deep architectures toparameterize the variational inverse-NHK as
Revised: In practice, we follow existing kernel learning approaches [42] and use deep architectures toparameterize the variational inverse-NHK as

Original: [Equation]
Revised: [Equation]

Original: where g  : R d !
Revised: where g  : R d !

Original: R s is the associated learnable non-linear mapping.
Revised: R s is the associated learnable non-linear mapping.

Original: Given a pre-trained teachermodel, distilling geometric knowledge boils down to 1) establishing equivalence of Eqn.
Revised: Given a pre-trained teachermodel, distilling geometric knowledge boils down to 1) establishing equivalence of Eqn.

Original: (18) and
Revised: (18) and

Original: Eqn. (19), and 2) matching pseudo-inverse NHK matrices for teacher and student models (respectivelydenoted as K † ✓ ⇤ , V and K † ✓ with clear meanings), leading to the training scheme as follows.
Revised: Eqn. (19), and 2) matching pseudo-inverse NHK matrices for teacher and student models (respectivelydenoted as K † ✓ ⇤ , V and K † ✓ with clear meanings), leading to the training scheme as follows.

Original: Training Scheme.
Revised: Training Scheme.

Original: Based on Eqn.
Revised: Based on Eqn.

Original: (19), we can deﬁne a reconstruction loss with respect to theteacher model (similar applies to the student model) as
Revised: (19), we can deﬁne a reconstruction loss with respect to theteacher model (similar applies to the student model) as

Original: [Equation]
Revised: [Equation]

Original: where ˜ D denotes the degree matrix over ˜ G .
Revised: where ˜ D denotes the degree matrix over ˜ G .

Original: Then, minimizing the reconstruction loss with ﬁxed GNNmodel parameter ✓ amounts to optimizing the variational parameter  , and minimizing predictionand distillation losses given ﬁxed  amounts to optimizing the student model parameter ✓ :
Revised: Then, minimizing the reconstruction loss with ﬁxed GNNmodel parameter ✓ amounts to optimizing the variational parameter  , and minimizing predictionand distillation losses given ﬁxed  amounts to optimizing the student model parameter ✓ :

Original: [Equation]
Revised: [Equation]

Original: Applying two steps iteratively adds up to an EM-like algorithm for training the student model.
Revised: Applying two steps iteratively adds up to an EM-like algorithm for training the student model.

Original: Inpractice, we set l + k as the last layer, and l as the ﬁrst layer.
Revised: Inpractice, we set l + k as the last layer, and l as the ﬁrst layer.

Original: Justiﬁcation.
Revised: Justiﬁcation.

Original: We justify parametric GKD from a variational inference perspective.
Revised: We justify parametric GKD from a variational inference perspective.

Original: From
Revised: From

Original: Eqn.
Revised: Eqn.

Original: (18), the forward GNN model f ✓ deﬁnes a model distribution p ✓ ( H ( l ) , H ( l + k ) , K ) =
Revised: (18), the forward GNN model f ✓ deﬁnes a model distribution p ✓ ( H ( l ) , H ( l + k ) , K ) =

Original: p ✓ ( H ( l ) )
Revised: p ✓ ( H ( l ) )

Original: p ✓ ( K | H ( l ) )
Revised: p ✓ ( K | H ( l ) )

Original: p ✓ ( H ( l + k ) | K , H ( l ) ) , where p ✓ ( K | H ( l ) ) is intractable, hindering the proceedingdistillation.
Revised: p ✓ ( H ( l + k ) | K , H ( l ) ) , where p ✓ ( K | H ( l ) ) is intractable, hindering the proceedingdistillation.

Original: In this light, the variational inverse-NHK model  †  is proposed with a variational distribution q
Revised: In this light, the variational inverse-NHK model  †  is proposed with a variational distribution q

Original:  ( H ( l ) , H ( l + k ) , K ) =
Revised:  ( H ( l ) , H ( l + k ) , K ) =

Original: q  ( H ( l + k ) )
Revised: q  ( H ( l + k ) )

Original: q  ( K | H ( l + k ) )
Revised: q  ( K | H ( l + k ) )

Original: q  ( H ( l ) | H ( l + k ) , K ) , which has a tractableposterior q  ( K | H ( l + k ) ) .
Revised: q  ( H ( l ) | H ( l + k ) , K ) , which has a tractableposterior q  ( K | H ( l + k ) ) .

Original: Now, we justify our training scheme with iterative optimization for Eqn.
Revised: Now, we justify our training scheme with iterative optimization for Eqn.

Original: (22)and (23) by the following proposition and refer the reader to Appendix D for detailed reasoning.
Revised: (22)and (23) by the following proposition and refer the reader to Appendix D for detailed reasoning.

Original: Proposition 1.
Revised: Proposition 1.

Original: The optimization in Eqn.
Revised: The optimization in Eqn.

Original: (22) and (23) essentially minimizes the following Kullback–Leibler (KL) divergence,
Revised: (22) and (23) essentially minimizes the following Kullback–Leibler (KL) divergence,

Original: [Equation]
Revised: [Equation]

Original: and hence attempts to establish equivalence between two latent variable models p ✓ and q  .
Revised: and hence attempts to establish equivalence between two latent variable models p ✓ and q  .

Original:  Experiments
Revised:  Experiments

Original: We conduct experiments to validate the efﬁcacy of our method on graph-structured data in terms ofvarious types of privileged geometric knowledge, combinations of teacher-student GNN architecturesand potential application scenarios.
Revised: We conduct experiments to validate the efﬁcacy of our method on graph-structured data in terms ofvarious types of privileged geometric knowledge, combinations of teacher-student GNN architecturesand potential application scenarios.

Original: We use three benchmark datasets Cora [27], Citeseer [35],
Revised: We use three benchmark datasets Cora [27], Citeseer [35],

Original: Pubmed [31], and two larger datasets, i.e., OGBN-Proteins and OGBN-Arxiv [19], for node classiﬁcation tasks.
Revised: Pubmed [31], and two larger datasets, i.e., OGBN-Proteins and OGBN-Arxiv [19], for node classiﬁcation tasks.

Original: More details are given in Appendix E.
Revised: More details are given in Appendix E.

Original: Implementation and Competitors We consider the following variants of the proposed GKD. 1)
Revised: Implementation and Competitors We consider the following variants of the proposed GKD. 1)

Original: GKD-G : non-parametric Gaussian NHK; 2) GKD-S : non-parametric Sigmoid NHK; 3) GKD-R : randomized NHK; 3) VGKD : parametric variational NHK.
Revised: GKD-G : non-parametric Gaussian NHK; 2) GKD-S : non-parametric Sigmoid NHK; 3) GKD-R : randomized NHK; 3) VGKD : parametric variational NHK.

Original: We choose KD methods that is representativein its own category for comparison, including KD [18], FitNets [33], FSP [49], TinyGNN [46], LSP [48].
Revised: We choose KD methods that is representativein its own category for comparison, including KD [18], FitNets [33], FSP [49], TinyGNN [46], LSP [48].

Original: Implementation details and descriptions of baselines are deferred to Appendix.
Revised: Implementation details and descriptions of baselines are deferred to Appendix.

Original: We also report theperformances of teacher and student model trained with the standard classiﬁcation loss, short as
Revised: We also report theperformances of teacher and student model trained with the standard classiﬁcation loss, short as

Original: Teacher and Student .
Revised: Teacher and Student .

Original: The teacher model is trained using the complete graph ˜ G , and, to calibrate withall other methods, tested using the partial graph G .
Revised: The teacher model is trained using the complete graph ˜ G , and, to calibrate withall other methods, tested using the partial graph G .

Original: Besides, we consider an Oracle model whichis both trained and tested on ˜ G , which naturally takes an advantaged place given more informationduring inference.
Revised: Besides, we consider an Oracle model whichis both trained and tested on ˜ G , which naturally takes an advantaged place given more informationduring inference.

Original: Since our method is compatible with the vanilla KD paradigm [18], we report theperformance delivered by their combinations (i.e., GKD+KD and VGKD+KD).
Revised: Since our method is compatible with the vanilla KD paradigm [18], we report theperformance delivered by their combinations (i.e., GKD+KD and VGKD+KD).

Original: Experiment Settings We investigate on various experimental settings according to different typesof privileged geometric knowledge.
Revised: Experiment Settings We investigate on various experimental settings according to different typesof privileged geometric knowledge.

Original: In the case of edge-aware geometric knowledge , the teachermodel has access to additional edge information, i.e., E ⇢ ˜ E and V =
Revised: In the case of edge-aware geometric knowledge , the teachermodel has access to additional edge information, i.e., E ⇢ ˜ E and V =

Original: ˜ V .
Revised: ˜ V .

Original: In the case of node-awaregeometric knowledge , the teacher model has access to additional node information, i.e., V ⇢ ˜ Vand E =
Revised: In the case of node-awaregeometric knowledge , the teacher model has access to additional node information, i.e., V ⇢ ˜ Vand E =

Original: ˜
Revised: ˜

Original: E \ {V ⇥ V} .
Revised: E \ {V ⇥ V} .

Original: We also consider other conventional or novel settings including model 257 compression, self-distillation, teaching MLP, which will be illustrated in detail.
Revised: We also consider other conventional or novel settings including model 257 compression, self-distillation, teaching MLP, which will be illustrated in detail.

Original: The backbone f ✓is set as 3 -layer GCN [23] for both student and teacher models, unless otherwise stated.
Revised: The backbone f ✓is set as 3 -layer GCN [23] for both student and teacher models, unless otherwise stated.

Original: All theexperiments are repeated ﬁve times with random initialization.
Revised: All theexperiments are repeated ﬁve times with random initialization.

Original: 5.1 Edge-Aware Geometric Knowledge
Revised: 5.1 Edge-Aware Geometric Knowledge

Original: We report results for the edge-aware geometric knowledge setting, where the teacher model hasaccess to auxiliary edges.
Revised: We report results for the edge-aware geometric knowledge setting, where the teacher model hasaccess to auxiliary edges.

Original: To quantify the privileged information, we set the quantity ( | ˜ E|  |E| ) / | ˜ E| , 263 called privileged information ratio (PIR), as 0 .
Revised: To quantify the privileged information, we set the quantity ( | ˜ E|  |E| ) / | ˜ E| , 263 called privileged information ratio (PIR), as 0 .

Original: As shown in Tab. 1, all variants of GKD outperformother KD baselines by a large margin on both datasets, and signiﬁcantly exceeds both Student and
Revised: As shown in Tab. 1, all variants of GKD outperformother KD baselines by a large margin on both datasets, and signiﬁcantly exceeds both Student and

Original: Teacher models.
Revised: Teacher models.

Original: Further, GKD and its variants rival, if not surpass, the Oracle model.
Revised: Further, GKD and its variants rival, if not surpass, the Oracle model.

Original: In other words,the student model trained using GKD could use far less graph topological information to achieve veryclose performance to competitors that are aware of the full graph topology during inference.
Revised: In other words,the student model trained using GKD could use far less graph topological information to achieve veryclose performance to competitors that are aware of the full graph topology during inference.

Original: Thisimplies that our GKD framework could indeed effectively transfer topological information.
Revised: Thisimplies that our GKD framework could indeed effectively transfer topological information.

Original: With regard to comparison among the variants of GKD, the parametric VGKD performs better than itsnon-parametric counterparts in most cases, and GKD-R is the most effective non-parametric methodin general.
Revised: With regard to comparison among the variants of GKD, the parametric VGKD performs better than itsnon-parametric counterparts in most cases, and GKD-R is the most effective non-parametric methodin general.

Original: Despite these results, GKD-G and GKD-S are also effective while being simpler and more
Revised: Despite these results, GKD-G and GKD-S are also effective while being simpler and more

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: easy-to-implement.
Revised: easy-to-implement.

Original: We presume that the performance variation of different GKD realizations stemfrom the different geometric property governed by the feature of datasets.
Revised: We presume that the performance variation of different GKD realizations stemfrom the different geometric property governed by the feature of datasets.

Original: Thus, it is useful to choosea proper implementation of GKD that is suitable for the application.
Revised: Thus, it is useful to choosea proper implementation of GKD that is suitable for the application.

Original: 5.2 Node-Aware Geometric Knowledge
Revised: 5.2 Node-Aware Geometric Knowledge

Original: We further investigate on the node-aware geometric knowledge setting where the teacher model hasaccess to more labeled nodes and their relations with the rest nodes.
Revised: We further investigate on the node-aware geometric knowledge setting where the teacher model hasaccess to more labeled nodes and their relations with the rest nodes.

Original: We set the PIR w.r.t.
Revised: We set the PIR w.r.t.

Original: labelednodes, deﬁned as ( | ˜ V train |  |V train | ) / |
Revised: labelednodes, deﬁned as ( | ˜ V train |  |V train | ) / |

Original: ˜ V train | , to 0 .
Revised: ˜ V train | , to 0 .

Original: A unique challenge of this setting compared 279 to the edge-ware counterpart is that, apart from graph topological information, the student model hasless labeled training samples, putting it into disadvantage.
Revised: A unique challenge of this setting compared 279 to the edge-ware counterpart is that, apart from graph topological information, the student model hasless labeled training samples, putting it into disadvantage.

Original: As we can see from Tab. 2.
Revised: As we can see from Tab. 2.

Original: The proposed
Revised: The proposed

Original: GKD and its variants again consistently outperform KD baselines throughout all the cases, surpassesboth Student and Teacher models, and are even as competitive as Oracle.
Revised: GKD and its variants again consistently outperform KD baselines throughout all the cases, surpassesboth Student and Teacher models, and are even as competitive as Oracle.

Original: 5.3 Results on Larger Datasets
Revised: 5.3 Results on Larger Datasets

Original: Table 3 presents results on two large graphs, i.e., OGBN-Proteins and OGBN-Arxiv.
Revised: Table 3 presents results on two large graphs, i.e., OGBN-Proteins and OGBN-Arxiv.

Original: We followthe data split in original paper [19], and use the same PIR setting as citation networks, and choosethe best variant of GKD to report in the table.
Revised: We followthe data split in original paper [19], and use the same PIR setting as citation networks, and choosethe best variant of GKD to report in the table.

Original: As shown in the table, again, we found our methodsconsistently outperform Teacher and Student models, and are close to the performance of the Oraclemodel, which suggests the effectiveness of GKD in large graphs.
Revised: As shown in the table, again, we found our methodsconsistently outperform Teacher and Student models, and are close to the performance of the Oraclemodel, which suggests the effectiveness of GKD in large graphs.

Original: 5.4 Performance Variation with Privileged Ratio
Revised: 5.4 Performance Variation with Privileged Ratio

Original: The results with respect to varying privileged (edge-aware and node-ware) information ratio are givenrespectively in Fig.
Revised: The results with respect to varying privileged (edge-aware and node-ware) information ratio are givenrespectively in Fig.

Original: 2 and Fig.
Revised: 2 and Fig.

Original: The performance of Oracle model is invariant as it is trained andtested on the same (complete) graph.
Revised: The performance of Oracle model is invariant as it is trained andtested on the same (complete) graph.

Original: In general, for Teacher model, Student model, and vanilla KD,their performance drops dramatically with increasing PIR quantifying the information loss, withvanilla KD’s performance lie in-between.
Revised: In general, for Teacher model, Student model, and vanilla KD,their performance drops dramatically with increasing PIR quantifying the information loss, withvanilla KD’s performance lie in-between.

Original: In contrast, our method is signiﬁcantly more robust, onlyshowing slight performance deterioration, exceeding the Teacher model, and approaching the Oraclemodel.
Revised: In contrast, our method is signiﬁcantly more robust, onlyshowing slight performance deterioration, exceeding the Teacher model, and approaching the Oraclemodel.

Original: Besides, we ﬁnd an interesting phenomenon that in the edge-aware setting on Pubmed dataset,the performance of Teacher model is the worst.
Revised: Besides, we ﬁnd an interesting phenomenon that in the edge-aware setting on Pubmed dataset,the performance of Teacher model is the worst.

Original: This is reasonable since the Teacher is trained usingfully observed graph, and may perform poorly once the graph topology, which it relies on for accurateresults, becomes incomplete at test time.
Revised: This is reasonable since the Teacher is trained usingfully observed graph, and may perform poorly once the graph topology, which it relies on for accurateresults, becomes incomplete at test time.

Original: 5.5 Other Settings
Revised: 5.5 Other Settings

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Revised: (cid:17)(cid:15)(cid:16)(cid:17)(cid:15)

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: Model Compression We report the perfor-
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: mance of GKD in the conventional model com-
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: pression setting [18] where the teacher and stu- [Table] dent are different-sized GNN models.
Revised: Model Compression We report the performance of GKD in the conventional model compression setting [18] where the teacher and student are different-sized GNN models.

Original: In thissetting, we train and test both teacher and student models on the complete graph.
Revised: In thissetting, we train and test both teacher and student models on the complete graph.

Original: We use
Revised: We use

Original: GCN as the backbone for teacher with a rel- atively large hidden size 64 , and use SGC or
Revised: GCN as the backbone for teacher with a relatively large hidden size 64 , and use SGC or

Original: GCN (with hidden size 8 and 16 ) as the student model.
Revised: GCN (with hidden size 8 and 16 ) as the studentmodel.

Original: We report the best result among all variants of GKD.
Revised: We report the best result among all variants of GKD.

Original: As shown in the ﬁrst section of
Revised: As shown in the ﬁrst section of

Original: Tab. 4, our method achieves notable improve- ments over the student model, rendering it as a useful KD approach for model compression.[Table]
Revised: Tab. 4, our method achieves notable improvements over the student model, rendering it as auseful KD approach for model compression.

Original: Self-Distillation We further report results for

Original: [Table] the setting of self-distillation [12, 29], which [Table] is a special case of KD when the teacher’s andstudent’s architectures are identical, often usedfor reﬁning their own performance.
Revised: Self-Distillation We further report results forthe setting of self-distillation [12, 29], whichis a special case of KD when the teacher’s andstudent’s architectures are identical, often usedfor reﬁning their own performance.

Original: The resultsas shown in Tab.
Revised: The resultsas shown in Tab.

Original: 4 validate that GKD could alsobe used to effectively boost GNN’s own performance.
Revised: 4 validate that GKD could alsobe used to effectively boost GNN’s own performance.

Original: GNNs Teach MLPs Finally, we study a special setting where any instance relation is unavailable attest time, i.e., E = ; , and examine whether our method could compress the geometric knowledge inthis setting when GNN degrades to MLP.
Revised: GNNs Teach MLPs Finally, we study a special setting where any instance relation is unavailable attest time, i.e., E = ; , and examine whether our method could compress the geometric knowledge inthis setting when GNN degrades to MLP.

Original: The results are shown in Tab. 4.
Revised: The results are shown in Tab. 4.

Original: Notably, the performanceof conventional GNN model degrades dramatically when the topological information is not availableat the test time, while our method turns out to still effective.
Revised: Notably, the performanceof conventional GNN model degrades dramatically when the topological information is not availableat the test time, while our method turns out to still effective.

Original: It again veriﬁes that our method couldcompress the geometric knowledge into the GNN, and also echoes recent advances in the domain ofcomputer vision, which suggest that MLP could achieve competitive performance with CNN [36].
Revised: It again veriﬁes that our method couldcompress the geometric knowledge into the GNN, and also echoes recent advances in the domain ofcomputer vision, which suggest that MLP could achieve competitive performance with CNN [36].

Original:  Conclusion
Revised:  Conclusion

Original: This paper formalizes the problem of graph topological knowledge transfer for GNNs.
Revised: This paper formalizes the problem of graph topological knowledge transfer for GNNs.

Original: We investigate on the implication of heat kernel in GNNs and propose the novel notion of neural heat kernel.
Revised: We investigateon the implication of heat kernel in GNNs and propose the novel notion of neural heat kernel.

Original: We leverage it to characterize the geometric property of the underlying manifold for graphs, and propose the framework of geometric knowledge distillation to transfer geometric knowledge from a teacher
Revised: Weleverage it to characterize the geometric property of the underlying manifold for graphs, and proposethe framework of geometric knowledge distillation to transfer geometric knowledge from a teacher

Original: GNN model to a student GNN.
Revised: GNN model to a student GNN.

Original: Experimental results validate the effectiveness of our approach in various practical settings.
Revised: Experimental results validate the effectiveness of our approach in various practical settings.

Original: The source code will be made public available
Revised: The source code will be made public available