Original: Geometric Distillation for Graph Networks
Revised: Geometric Distillation for Graph Networks

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: We study a new paradigm of knowledge transfer in the context of geometric deeplearning, which aims at distilling knowledge from a teacher graph neural network(GNN) model trained on a large graph to a student GNN model operating on asmaller graph.
Revised: We study a new paradigm of knowledge transfer in the context of geometric deeplearning, which aims at distilling knowledge from a teacher graph neural network(GNN) model trained on a large graph to a student GNN model operating on asmaller graph.

Original: To this end, we revisit the connection between thermodynamicsand the behavior of GNN, based on which we propose Neural Heat Kernel (NHK)to encapsulate the geometric property of the underlying manifold concerning thearchitecture of GNN.
Revised: To this end, we revisit the connection between thermodynamicsand the behavior of GNN, based on which we propose Neural Heat Kernel (NHK)to encapsulate the geometric property of the underlying manifold concerning thearchitecture of GNN.

Original: A natural solution is derived by analysing and aligning NHKson teacher and student models, dubbed as Geometric Knowledge Distillation.
Revised: A natural solution is derived by analysing and aligning NHKson teacher and student models, dubbed as Geometric Knowledge Distillation.

Original: We develop non- and parametric instantiations and demonstrate their efficacy invarious experimental settings for knowledge distillation regarding different typesof privileged topological information and teacher-student schemes
Revised: We develop non- and parametric instantiations and demonstrate their efﬁcacy invarious experimental settings for knowledge distillation regarding different typesof privileged topological information and teacher-student schemes.

Original:  Introduction
Revised:  Introduction

Original: Modern graph neural networks (GNNs) [26, 46, 50] have shown remarkable performance in learningrepresentations for structured instances.
Revised: Modern graph neural networks (GNNs) [23, 39, 43] have shown remarkable performance in learningrepresentations for structured instances.

Original: From the perspective of geometric deep learning [5, 4, 35],much of the achievement of GNNs can be attributed to their successful implementation of thepermutation invariance property as geometric priors 1 into the architecture design.
Revised: From the perspective of geometric deep learning [4, 3, 30],much of the achievement of GNNs can be attributed to their successful implementation of thepermutation invariance property as geometric priors 1 into the architecture design.

Original: Nevertheless, inpractice, GNNs highly rely on graph topology, as essential input information, to explore the relationalknowledge implicit in interactions of instance pairs throughout the entire message passing process,termed as geometric knowledge in this paper.
Revised: Nevertheless, inpractice, GNNs highly rely on graph topology, as essential input information, to explore the relationalknowledge implicit in interactions of instance pairs throughout the entire message passing process,termed as geometric knowledge in this paper.

Original: As advances in generalized distillation [30, 44] revealthe possibility of encoding input features into model construction, natural questions arise as to:
Revised: As advances in generalized distillation [25, 37] revealthe possibility of encoding input features into model construction, natural questions arise as to:

Original: Is it possible, and if so, how can we encode graph topology as a special type of ‘geometric prior’ intoa GNN model, such that the model could precisely capture the underlying geometric knowledge evenwithout full graph topology as input?.
Revised: Is it possible, and if so, how can we encode graph topology as a special type of ‘geometric prior’ intoa GNN model, such that the model could precisely capture the underlying geometric knowledge evenwithout full graph topology as input?.

Original: In specific, we are interested in the following geometric knowledge transfer problem: a GNN model(with node-specific outputs for node-level prediction [22]) is exposed with a partial graph, which is asubset of the complete graph.
Revised: In speciﬁc, we are interested in the following geometric knowledge transfer problem: a GNN model(with node-speciﬁc outputs for node-level prediction [19]) is exposed with a partial graph, which is asubset of the complete graph.

Original: Formally speaking, we have notations:
Revised: Formally speaking, we have notations:

Original: [Equation] } (partial graph) , ˜ G = { ˜ V , ˜ E} (complete graph) , where V ⊆ ˜ V , E ⊆ {V × V} ∩ ˜ E .
Revised: G = {V , E} (partial graph) , ˜ G = { ˜ V , ˜ E} (complete graph) , where V ✓ ˜ V , E ✓ {V ⇥ V} \ ˜ E .

Original: Our goal is to transfer or encode geometric knowledge extracted from ˜ G to the target GNN model thatis only aware of G .
Revised: Our goal is to transfer or encode geometric knowledge extracted from ˜ G to the target GNN model thatis only aware of G .

Original: Besides of academic interest, studying this problem is also of much practical value.
Revised: Besides of academic interest, studying this problem is also of much practical value.

Original: As a non-exhaustive list of applications: improving efficiency without compromising on effectivenessfor coarsened graphs [14, 23, 60], privacy constrained scenarios in social recommenders or federatedlearning where the complete graph is unavailable [31, 47, 59], promoting concentration on targetedcommunity to bring up economic benefits [51]. Achieving this target is non-trivial for that we need to first find a principled and fundamental way toencapsulate the geometric knowledge extracted by GNN model, which requires in-depth investigation A B H P O
Revised: As a non-exhaustive list of applications: improving efﬁciency without compromising on effectivenessfor coarsened graphs [11, 20, 52], privacy constrained scenarios in social recommenders or federatedlearning where the complete graph is unavailable [26, 40, 51], promoting concentration on targetedcommunity to bring up economic beneﬁts [44].

Original: M R V

Original: W

Original: N C D E L
Revised: (cid:36) (cid:37) (cid:43) (cid:51) (cid:50) (cid:48) (cid:53) (cid:57) (cid:58) (cid:49) (cid:38) (cid:39) (cid:40) (cid:47) (cid:42) (cid:56) (cid:54) (cid:41) (cid:60) (cid:59) (cid:55) (cid:52) (cid:44) (cid:45) (cid:46) (cid:36) (cid:50) (cid:48) (cid:53) (cid:57) (cid:58) (cid:49) (cid:39) (cid:40) (cid:47) (cid:42) (cid:56) (cid:54) (cid:41) (cid:36) (cid:37) (cid:43) (cid:51) (cid:50) (cid:48) (cid:53) (cid:57) (cid:58) (cid:49) (cid:38) (cid:39) (cid:40) (cid:47) (cid:42) (cid:56) (cid:54) (cid:41) (cid:55) (cid:52) (cid:44) (cid:45) (cid:46) (cid:68)(cid:12)(cid:3)(cid:55)(cid:72)(cid:68)(cid:70)(cid:75)(cid:72)(cid:85) (cid:70)(cid:12)(cid:3)(cid:54)(cid:87)(cid:88)(cid:71)(cid:72)(cid:81)(cid:87) (cid:3) (cid:11)(cid:68)(cid:73)(cid:87)(cid:72)(cid:85)(cid:3)(cid:42)(cid:46)(cid:39)(cid:12) (cid:69)(cid:12)(cid:3)(cid:54)(cid:87)(cid:88)(cid:71)(cid:72)(cid:81)(cid:87) (cid:3) (cid:11)(cid:69)(cid:72)(cid:73)(cid:82)(cid:85)(cid:72)(cid:3)(cid:42)(cid:46)(cid:39)(cid:12)

Original: G U S F Y X T Q I J
Revised: Achieving this target is non-trivial for that we need to ﬁrst answer “what is geometric knowledgeand how to translate it into an explicit form", which requires in-depth investigation on the roleof graph topology throughout the progressive process of message passing.

Original: K

Original: A

Original: O

Original: M

Original: R V

Original: W

Original: N

Original: D E

Original: L

Original: G

Original: U S F

Original: A B

Original: H P O M R V W N
Revised: Therefore, we take athermodynamic view borrowed from physics and propose a new methodology built upon recentadvances revealing the connection between heat diffusion and architectures of GNNs [7, 41, 6].

Original: C

Original: D E

Original: L

Original: G

Original: U S F

Original: T Q

Original: I J

Original: K

Original: [Figure] on the role of graph topology throughout the progressive process of message passing. Therefore,we take a thermodynamic view borrowed from physics and propose a new methodology built uponrecent advances revealing the connection between heat diffusion and architectures of GNNs [8, 48, 7]. Specifically, we interpret feature propagation as heat flows on the underlying Riemmanian manifold,whose characteristics (that are dependent on graph topology and the GNN model) pave the way for aprincipled representation of the latent geometric knowledge.
Revised: Speciﬁcally, we interpret feature propagation as heat ﬂows on the underlying Riemmanian manifold,whose characteristics (that are dependent on graph topology and the GNN model) pave the way for aprincipled representation of the latent geometric knowledge.

Original: 1.1 Our Contributions
Revised: 1.1 Our Contributions

Original: New theoretical perspective for analyzing latent graph geometry.
Revised: New theoretical perspective for analyzing latent graph geometry.

Original: On top of the connectionbetween heat equation and GNNs, we step further to inspect the implication of heat kernel for GNNs, and propose a novel notion of Neural Heat Kernel (NHK) with rigorous proof of its existence.
Revised: On top of the well-establishedconnection between heat equation and GNNs, we step further to inspect the implication of heat kernelfor existing GNNs, and propose a novel notion of Neural Heat Kernel (NHK) with rigorous proofof its existence.

Original: Heat kernel intrinsically defines the unique solution to the heat equation and can be a fundamentalcharacterization for the geometric property of the underlying manifold [17, 18].
Revised: Heat kernel intrinsically deﬁnes the unique solution to the heat equation and canbe a fundamental characterization for the geometric property of the underlying manifold [14, 15].

Original: Likewise, NHKuncovers geometric property of the latent graph manifold for GNNs, and governs how informationflows between pairs of instances, which lends us a mathematical tool to encapsulate geometricknowledge extracted from GNN model and enables geometric knowledge transfer.
Revised: Likewise, NHK uncovers geometric property of the latent graph manifold for GNNs, and governs howinformation ﬂows between pairs of instances.

Original: We also highlightthis result alone may be of interest in the field of geometric deep learning, and is valuable forunderstanding GNNs.
Revised: It lends us a tractable mathematical tool to understandwhat geometric knowledge the GNN model has learned or extracted from the graph topology.

Original: Flexible distillation framework with versatile instantiations.
Revised: Flexible distillation framework with versatile instantiations.

Original: Based on the insights mentionedabove, we treat NHK matrices as representation of the latent geometric knowledge, upon which webuild a flexible and principled distillation framework dubbed as Geometric Knowledge Distillation(GKD), which aims at encoding and transferring geometric knowledge by aligning latent manifoldsbehind GNN models as illustrated in Fig.
Revised: Based on the insights mentionedabove, we treat NHK matrices as representation of the latent geometric knowledge, upon which webuild a ﬂexible and principled distillation framework dubbed as Geometric Knowledge Distillation(GKD), which aims at encoding and transferring geometric knowledge by aligning latent manifoldsbehind GNN models as illustrated in Fig.

Original: Moreover, we develop non-parametirc and parametricversions of GKD, in terms of different ways to approximate NHK computation.
Revised: Moreover, we develop non-parametirc and parametricversions of GKD, in terms of different ways to approximate NHK computation.

Original: Specifically, theformer derives explicit NHKs via assumptions on latent space, and the later learns NHK in a datadriven manner through the lens of variational inference, which will be shown to match two models byminimizing the KL-divergence of their distributions.
Revised: Speciﬁcally, theformer derives explicit NHKs via assumptions on latent space, and the later learns NHK in a datadriven manner through the lens of variational inference, which will be shown to match two models byminimizing the KL-divergence of their distributions.

Original: Promising results for geometric knowledge transfer and conventional KD purposes.
Revised: Promising results for geometric knowledge transfer and conventional KD purposes.

Original: We verifythe efficacy of GKD in terms of different geometric knowledge types (i.e., edge-aware and node-awareones), and further show its effectiveness for conventional KD purposes (e.g., model compression,self-distillation) for broader applicability.
Revised: We verifythe practical efﬁcacy of GKD in terms of different geometric knowledge types (i.e., edge-aware andnode-aware ones), and further show its effectiveness for conventional KD purposes (e.g., modelcompression, self-distillation).

Original: We highlight that our methods consistently exceed teachermodel and rival with the oracle model that gives the performance upper bound in principle.
Revised: We highlight that our methods consistently exceed teacher model andeven rival with the oracle model that gives the performance upper bound in principle.

Original: 1.2 Links to Related Works
Revised: 1.2 Links to Related Works

Original: Geometric Deep Learning.
Revised: Geometric Deep Learning.

Original: The study of geometric deep learning [5, 35] provides fundamentalprinciples and methodology to generalize deep learning methods to non-Euclidean domains (e.g.,graphs and manifolds).
Revised: The study of geometric deep learning [4, 30] provides fundamentalprinciples and methodology to generalize deep learning methods to non-Euclidean domains (e.g.,graphs and manifolds).

Original: From this perspective, architectures for off-the-shelf GNNs [26, 46, 50, 19]have naturally incorporated the geometric prior knowledge for graphs such as permutation invariance .
Revised: From this perspective, architectures for off-the-shelf GNNs [23, 39, 43, 16]have naturally incorporated the geometric prior knowledge for graphs such as permutation invariance .

Original: Despite their remarkable success, they highly rely on the graph topology.
Revised: Despite their remarkable success, they highly rely on the graph topology.

Original: This work extends theidea of geometric deep learning by treating the global graph topology as a special type of priorknowledge, and attempts to encode it into GNNs themselves, such that the trained model wouldleverage information from the global graph topology even without explicitly taking it as input.
Revised: This work extends theidea of geometric deep learning by treating the global graph topology as a special type of priorknowledge, and attempts to encode it into GNNs themselves, such that the trained model wouldleverage information from the global graph topology even without explicitly taking it as input.

Original: Graph-Based Knowledge Distillation.
Revised: Graph-Based Knowledge Distillation.

Original: Knowledge distillation (KD)
Revised: Knowledge distillation (KD)

Original: [21, 1] uses the outputs of ateacher model as alternative supervised signals to teach a student model, with various new paradigmsincluding feature-based [38, 58] and relation-based [56] ones.
Revised: [18, 1] uses the outputs of ateacher model as alternative supervised signals to teach a student model, with various new paradigmsincluding feature-based [33, 50] and relation-based [49] ones.

Original: While some prior arts [9, 54, 47, 55,53] attempted to combine KD and GNNs, i.e., graph-based KD, they are nearly straight-forwardadaptations of KD without in-depth investigation on the role of graph topology, also restricted by a specific choice of GNN architecture or application scenarios.
Revised: While some prior arts [8, 47, 40, 48,46] attempted to combine KD and GNNs, i.e., graph-based KD, they are nearly straight-forwardadaptations of KD without in-depth investigation on the role of graph topology, also restricted by a speciﬁc choice of GNN architecture or application scenarios, e.g., model compression.

Original: In contrast, we first formalize theproblem of geometric knowledge transfer, theoretically answer the question “how to represent graphgeometric knowledge and encode it into GNN models", and propose geometric distillation approachbased on the theoretical results. We also add introduction and comparison with other distillingmethods to Appendix E.5.
Revised: In contrast, weﬁrst formalize the problem of geometric knowledge transfer, theoretically answer the question “howto represent graph geometric knowledge and encode it into GNN models", and propose geometricdistillation approach based on the theoretical results, which is shown to be effective in varioussettings.

Original:  Preliminaries
Revised:  Preliminaries

Original: We commence with a brief detour to heat equation on Riemannian manifolds, and its connection withmodern GNN architectures.
Revised: We commence with a brief detour to heat equation on Riemannian manifolds, and its connection withmodern GNN architectures.

Original: Moreover, we bring forth the notion of heat kernel to motivate this work.
Revised: Moreover, we bring forth the notion of heat kernel to motivate this work.

Original: Heat Equation on Manifolds.
Revised: Heat Equation on Manifolds.

Original: We are interested in heat equation defined on a smooth k -dimensional
Revised: We are interested in heat equation deﬁned on a smooth k -dimensional

Original: Riemannian manifold M .
Revised: Riemannian manifold M .

Original: Suppose the manifold is associated with a scalar- or vector-valued functionx ( u, t ) : M × [0 , ∞ ) 
Revised: Suppose the manifold is associated with a scalar- or vector-valued functionx ( u, t ) : M ⇥ [0 , 1 ) !

Original: → R d , quantifying a specific type of signals such as heat at a point u ∈ Mand time t .
Revised: R d , quantifying a speciﬁc type of signals such as heat at a point u 2 M 88 and time t .

Original: Fourier’s law of heat conductivity describes the flow of heat with respect to time andspace, via a partial differential equation (PDE) called heat equation [6], i.e.,
Revised: Fourier’s law of heat conductivity describes the ﬂow of heat with respect to time andspace, via a partial differential equation (PDE) called heat equation [5], i.e.,

Original: [Equation]
Revised: [Equation]

Original: where c > 0 is the thermal conductivity coefficient, and ∆ is the natural Laplace–Beltrami operatorassociated with M .
Revised: where c > 0 is the thermal conductivity coefﬁcient, and  is the natural Laplace–Beltrami operatorassociated with M .

Original: Rewriting ∆ as the functional composition of the divergence operator ∇ ∗ andgradient operator ∇ , i.e., ∆ = ∇ ∗ ◦ ∇ , we can interpret the heat equation as: the variation oftemperature within an infinitesimal time interval at a point is equivalent to the divergence between itsown temperature and the average temperature on an infinitesimal sphere around it.
Revised: Rewriting  as the functional composition of the divergence operator r ⇤ and 92 gradient operator r , i.e.,  = r ⇤  r , we can interpret the heat equation as: the variation of 93 temperature within an inﬁnitesimal time interval at a point is equivalent to the divergence between itsown temperature and the average temperature on an inﬁnitesimal sphere around it.

Original: Implications on Graphs.
Revised: Implications on Graphs.

Original: A spatial discretisation of a continuous manifold yields a graph G ={V , E} , whose nodes can be thought of as embedded on the base manifold.
Revised: A spatial discretisation of a continuous manifold yields a graph G ={V , E} , whose nodes can be thought of as embedded on the base manifold.

Original: In fact, the heat equationalong with variants thereof (e.g., Schrödinger equation) have found widespread use in modeling graphdynamics [11, 24, 33].
Revised: In fact, the heat equation 97 along with variants thereof (e.g., Schrödinger equation) have found widespread use in modeling graphdynamics [10, 21, 28].

Original: More importantly, it has been recently revealed to be intimately related withthe architectures of modern GNNs [48, 8, 7]: suppose X (0) =
Revised: More importantly, it has been recently revealed to be intimately related withthe architectures of modern GNNs [41, 7, 6]: suppose X (0) =

Original: { x ( u, 0) } u ∈V ∈ R n × d denotes theinitial condition for Eqn.
Revised: { x ( u, 0) } u 2 V 2 R n ⇥ d denotes theinitial condition for Eqn.

Original: (2) determined by input node features, then solving the heat equation undercertain definitions of ∇ ∗ and ∇ (i.e., definition of ∆ ) amounts to different architectures of GNNs.
Revised: (2) determined by input node features, then solving the heat equation undercertain deﬁnitions of r ⇤ and r (i.e., deﬁnition of  ) amounts to different architectures of GNNs.

Original: For instance:
Revised: For instance:

Original: Example 1.
Revised: Example 1.

Original: [48] Define the discretised counterpart of ∆ as the graph Laplacian matrix L =(cid:101)
Revised: [41] Deﬁne the discretised counterpart of  as the graph Laplacian matrix L =e [Equation]

Original:  D − 12 ( (cid:101) D − (cid:101) A ) (cid:101) D − 12 .
Revised:  [Equation] e D  12 .

Original: Numerically solving Eqn.
Revised: Numerically solving Eqn.

Original: (2) using the forward Euler method with step sizeτ = 1 yields the formulation of Simple Graph Convolution (SGC) [50], where Θ denote learnabletransformation matrix
Revised: (2) using the forward Euler method with step size⌧ = 1 yields the formulation of Simple Graph Convolution (SGC) [43]

Original: [Equation]
Revised: [Equation]

Original: Example 2.
Revised: Example 2.

Original: [8] Define the gradient operator ∇ ij as the difference of source and target node features,the divergence operator ∇ ∗ i as the sum of features of all edges for the node.
Revised: [7] Deﬁne the gradient operator r ij as the difference of source and target node features,the divergence operator r ⇤ i as the sum of features of all edges for the node.

Original: Numerically solving
Revised: Numerically solving

Original: Eqn.
Revised: Eqn.

Original: (2) using the explicit Euler scheme with step size τ yields the following recursive formulation
Revised: (2) using the explicit Euler scheme with step size ⌧ yields the following recursive formulation

Original: [Equation][Equation]
Revised: [Equation]

Original: ) where G is a diffusivity coefficient matrix in place of c .
Revised: (4) where G is a diffusivity coefﬁcient matrix in place of c .

Original: Moreover, stacking a non-linear transformation layer after each step yields the formulation of Graph
Revised: Moreover, stacking a non-linear transformation layer after each step yields the formulation of Graph

Original: Convolution Networks (GCN) [26] for Eqn.
Revised: Convolution Networks (GCN) [23] for Eqn.

Original: (3), Graph Attention Networks (GAT) [46] with residualconnection for Eqn.
Revised: (3), Graph Attention Networks (GAT) [39] with residualconnection for Eqn.

Original: (4), and even more GNN architectures by virtue of the flexibility of interpretionfor heat equation on graphs.
Revised: (4), and even more GNN architectures by virtue of the ﬂexibility of interpretionfor heat equation on graphs.

Original: Heat Kernels Intriguingly, it turns out that the initial value problem of heat equation on any manifold
Revised: Heat Kernels Intriguingly, it turns out that the initial value problem of heat equation on any manifold

Original: M has a smallest positive fundamental solution depending on the Laplace operator ∆ , known as theheat kernel [2].
Revised: M has a smallest positive fundamental solution depending on the Laplace operator  , known as theheat kernel [2].

Original: It is denoted as a kernel function κ ( x, y, t ) , such that
Revised: It is denoted as a kernel function  ( x, y, t ) , such that

Original: [Equation]
Revised: [Equation]

Original: where µ is a non-negative measure associated with M .
Revised: where µ is a non-negative measure associated with M .

Original: In physics, the heat kernel κ ( x, y, t ) canbe interpreted as a transition density that describes the asymptotic behavior of a natural Brownian motion on the manifold.
Revised: In physics, the heat kernel  ( x, y, t ) can 118 be interpreted as a transition density that describes the asymptotic behavior of a natural Brownian motion on the manifold.

Original: Its formulation thus can be treated as a unique reflection or representation of 123 the geometry of the underlying manifold .
Revised: Its formulation thus can be treated as a unique reﬂection or representation of 120 the geometry of the underlying manifold .

Original: For example, if the manifold is a k -dimensional Euclidean
Revised: For example, if the manifold is a k -dimensional Euclidean

Original: Space R k or a Hyperbolic Space H k , the explicit formula of heat kernel is respectively given by,κ ( u i , u j , t ) =
Revised: Space R k or a Hyperbolic Space H k , the explicit formula of heat kernel is respectively given by, ( u i , u j , t ) =

Original: (4 πt ) k/ 2 exp (cid:18) − ρ4 t (cid:19) and κ ( u i , u j , t ) = ( − 1) m 2 m π m (4 πt )(cid:18) sinh ρ ∂ ∂ρ (cid:19) m 
Revised: (4 ⇡ t ) k/ 2 exp ✓  ⇢4 t ◆ and  ( u i , u j , t ) = (  1) m 2 m ⇡ m (4 ⇡ t ) 12 ✓ sinh ⇢ @ @⇢ ◆ m e  m 2 t  ⇢ 2 4 t , (6) where ⇢ =

Original: e − m 2 t − ρ 2 4 t , (6) where ρ = d ( u i , u j ) denote geodesic distance.
Revised: d ( u i , u j ) denote geodesic distance.

Original: Heat kernel has also been adopted for graph-relatedapplications such as community detection [29], graph clustering [52].
Revised: Heat kernel has also been adopted for graph-relatedapplications such as community detection [24], graph clustering [45].

Original:  Extending Heat Kernel to GNNs
Revised:  Extending Heat Kernel to GNNs

Original: The starting point of this work is the development of neural heat kernel , built upon the previouslymentioned connection of GNNs and heat equation.
Revised: The starting point of this work is the development of neural heat kernel , built upon the previouslymentioned connection of GNNs and heat equation.

Original: As will be discussed later, this novel notion lendsus a thermodynamic perspective to the intrinsic geometric property of the latent graph manifoldembodied in GNNs, and hence paves the way for distilling geometric knowledge.
Revised: As will be discussed later, this novel notion lendsus a thermodynamic perspective to the intrinsic geometric property of the latent graph manifoldembodied in GNNs, and hence paves the way for distilling geometric knowledge.

Original: 3.1 Neural Heat Kernel
Revised: 3.1 Neural Heat Kernel

Original: Consider the graph signal X ( t ) at time t and node features H ( l ) at layer l as interchangeable notions.
Revised: Consider the graph signal X ( t ) at time t and node features H ( l ) at layer l as interchangeable notions.

Original: Consequently, feature propagation using one layer of GNN amounts to heat diffusion on the basemanifold M within a certain time interval τ , leading to the equivalences of X ( t + τ ) and H ( l +1) :
Revised: Consequently, feature propagation using one layer of GNN amounts to heat diffusion on the basemanifold M within a certain time interval ⌧ , leading to the equivalences of X ( t + ⌧ ) and H ( l +1) :

Original: [Equation]
Revised: [Equation]

Original:  where f θ denotes an arbitrary GNN model with parameter θ , and ∆( f θ , G ) denotes a generalizationof Laplace-Beltrami operator defined over the base manifold M associated with graph G and thearbitrary backbone GNN model f θ .
Revised: (7) where f ✓ denotes an arbitrary GNN model with parameter ✓ , and  ( f ✓ , G ) denotes a generalizationof Laplace-Beltrami operator deﬁned over the base manifold M associated with graph G and thearbitrary backbone GNN model f ✓ .

Original: In light of this connection, we consider a natural generalization of heat kernel for GNNs, termed asneural heat kernel (NHK) to highlight its difference with heat kernel in the thermodynamic context.
Revised: In light of this connection, we consider a natural generalization of heat kernel for GNNs, termed asneural heat kernel (NHK) to highlight its difference with heat kernel in the thermodynamic context.

Original: In particular, a single-layer NHK is defined as a positive definite symmetric kernel function denotedas κ ( l ) θ ( v i , v j ) , where the sub-script θ implies that it is associated with the architecture and parametersof the backbone GNN, and the super-script ( l ) implies that it is specific to each layer, analogous tothe role of continuous time t in Eqn.
Revised: In particular, a single-layer NHK is deﬁned as a positive deﬁnite symmetric kernel function denotedas  ( l ) ✓ ( v i , v j ) , where the sub-script ✓ implies that it is associated with the architecture and parametersof the backbone GNN, and the super-script ( l ) implies that it is speciﬁc to each layer, analogous tothe role of continuous time t in Eqn.

Original: Theorem 1.
Revised: Theorem 1.

Original: (Existence of Single-Layer NHK)
Revised: (Existence of Single-Layer NHK)

Original: Suppose two expressions in Eqn.
Revised: Suppose two expressions in Eqn.

Original: (7) are equivalent(see Appendix G for more discussions), then for any graph G and GNN model f θ , there exist a uniquesingle-layer NHK function κ ( l ) θ ( · ) such that for any node v i ∈ V and l > 0 ,[Equation] µ
Revised: (7) are equivalent,then for any graph G and GNN model f ✓ , there exist a unique single-layer NHK function  ( l ) ✓ ( · ) suchthat for any node v i 2 V and l > 0 ,

Original: [Equation]
Revised: [Equation]

Original: where h ( l ) i ∈ R d denotes the feature of node v i at l -th layer, and µ is a measure over vertices specifiedas the inverse of node degree
Revised: where h ( l ) i 2 R d denotes the feature of node v i at l -th layer, and µ is a measure over vertices speciﬁedas the inverse of node degree

Original: 1 /d i .
Revised: 1 /d i .

Original: To push further, we can generalize NHK across multiple layers of GNN, termed as a cross-layer NHKκ θ ( v i , v j , l (cid:55)→ l + k ) (e.g., from l -th layer to ( l + k ) -th layer of GNN).
Revised: To push further, we can generalize NHK across multiple layers of GNN, termed as a cross-layer NHK ✓ ( v i , v j , l 7! l + k ) (e.g., from l -th layer to ( l + k ) -th layer of GNN).

Original: Its existence could be inducedrecursively by the semi-group identity property of NHK concerning consecutive GNN layers.
Revised: Its existence could be inducedrecursively by the semi-group identity property of NHK concerning consecutive GNN layers.

Original: Theorem 2.
Revised: Theorem 2.

Original: (Semigroup Identity Property of NHK)
Revised: (Semigroup Identity Property of NHK)

Original: The NHK satisfies the semigroup identityproperty: ∀ v i , v j ∈ V and l > 0 , there exists a cross-layer NHK across two consecutive layersκ θ ( v i , v j , l (cid:55)→ l + 2) = (cid:88) v k ∈V κ (
Revised: The NHK satisﬁes the semigroup identityproperty: 8 v i , v j 2 V and l > 0 , there exists a cross-layer NHK across two consecutive layers

Original: [Equation]
Revised: [Equation]

Original: This theorem indicates that stacks of multiple GNN layers also constitute a valid kernel, i.e.,
Revised: This theorem indicates that stacks of multiple GNN layers also constitute a valid kernel, i.e.,

Original: [Equation]
Revised: [Equation]

Original: Analogous to heat kernel as an unique characterization of the underlying space, NHK characterizes thegeometric property of the latent graph manifold for GNNs.
Revised: Analogous to heat kernel as an unique characterization of the underlying space, NHK characterizes thegeometric property of the latent graph manifold for GNNs.

Original: Additionally, NHK is dependent on GNNmodels through the definition of the associated Laplace-Beltrami operator ∆( f θ , G ) , inheriting theexpressiveness of neural networks and varying through the course of training.
Revised: Additionally, NHK is dependent on GNNmodels through the deﬁnition of the associated Laplace-Beltrami operator  ( f ✓ , G ) , inheriting theexpressiveness of neural networks and varying through the course of training.

Original: Intuitively, NHK canbe thought of as a model-driven encoding for topological information, encapsulating the geometricknowledge learned by GNNs into a tractable functional form.
Revised: Intuitively, NHK canbe thought of as a model-driven encoding for topological information, encapsulating the geometricknowledge learned by GNNs into a tractable functional form.

Original: 3.2 Application in Geometric Distillation
Revised: 3.2 Application in Geometric Distillation

Original: Consider the problem of distilling geometric knowledge, which involves an intelligent teacher modelf θ ∗ , which is exposed to and pre-trained over the (relatively) complete graph ˜ G =
Revised: Consider the problem of distilling geometric knowledge, which involves an intelligent teacher modelf ✓ ⇤ , which is exposed to and pre-trained over the (relatively) complete graph ˜ G =

Original: (˜ V , ˜ E ) , and astudent model f θ that is exposed to the partial graph G =
Revised: (˜ V , ˜ E ) , and astudent model f ✓ that is exposed to the partial graph G =

Original: ( V , E ) , where V ⊆ ˜ V and E ⊆ {V ×V}∩ ˜ E .
Revised: ( V , E ) , where V ✓ ˜ V and E ✓ {V ⇥ V} \ ˜ E .

Original:  Our target is to train a student model (with the help of teacher model) that operates on G to be ascompetitive as models operating on ˜ G during inference.
Revised: 163 Our target is to train a student model (with the help of teacher model) that operates on G to be ascompetitive as models operating on ˜ G during inference.

Original: Since G is a sub-graph of ˜ G , they should lie inthe same space (i.e., latent manifold) governed by the underlying mechanism of data generation, andhence we expect student and teacher models to capture the same geometric property of this sharedspace.
Revised: Since G is a sub-graph of ˜ G , they should lie inthe same space (i.e., latent manifold) governed by the underlying mechanism of data generation, andhence we expect student and teacher models to capture the same geometric property of this sharedspace.

Original: This leads to the principle of Geometric Knowledge Distillation (GKD): transfer the geometricknowledge of the intelligent teacher to the student such that the student can propagate features as if itis aware of the complete graph topology (see the example in Fig. 1).
Revised: This leads to the principle of Geometric Knowledge Distillation (GKD): transfer the geometricknowledge of the intelligent teacher to the student such that the student can propagate features as if itis aware of the complete graph topology (see the example in Fig. 1).

Original: To this end, we resort to NHK matrices on the teacher (resp.
Revised: To this end, we resort to NHK matrices on the teacher (resp.

Original: student) model over the complete (resp.partial) graph as instantiations of their geometric knowledge, denoted as
Revised: student) model over the complete (resp.partial) graph as instantiations of their geometric knowledge, denoted as

Original: [Equation]
Revised: [Equation]

Original: written compactly as K ( l +1) ( G ) when k = 1 .
Revised: written compactly as K ( l +1) ( G ) when k = 1 .

Original: The NHK matrix is a positive semi-definite symmetricmatrix, and alike κ , is dependent on the GNN model f θ and graph G .
Revised: The NHK matrix is a positive semi-deﬁnite symmetricmatrix, and alike  , is dependent on the GNN model f ✓ and graph G .

Original: Denote K ( l ) θ ∗ , V ( ˜ G ) ∈ R |V|×|V|as the sub-matrix of K ( l ) θ ∗ ( ˜ G ) with row and column indices in V .
Revised: Denote K ( l ) ✓ ⇤ , V ( ˜ G ) 2 R |V| ⇥ |V|as the sub-matrix of K ( l ) ✓ ⇤ ( ˜ G ) with row and column indices in V .

Original: The distillation loss for GKD is[Equation] dis
Revised: The distillation loss for GKD is

Original: [Equation][Equation]
Revised: [Equation]

Original: where d( · , · ) is a similarity measure, for which we choose Frobenius distance as implementation, i.e.,
Revised: where d( · , · ) is a similarity measure, for which we choose Frobenius distance as implementation, i.e.,

Original: [Equation]
Revised: [Equation]

Original: where W ∈ R |V|×|V| is a weighting matrix to trade-off distillation loss with respect to different nodepairs depending on their connectivity.
Revised: where W 2 R |V| ⇥ |V| is a weighting matrix to trade-off distillation loss with respect to different nodepairs depending on their connectivity.

Original: For k = 1 , the loss can be re-written as L ( l +1) dis
Revised: For k = 1 , the loss can be re-written as L ( l +1) dis

Original: ( K θ ∗ , V , K θ ) .
Revised: ( K ✓ ⇤ , V , K ✓ ) .

Original: Note that one can also specify different k for teacher and student models in Eqn.
Revised: Note that one can also specify different k for teacher and student models in Eqn.

Original: (11) in case when theteacher model is deeper.
Revised: (11) in case when theteacher model is deeper.

Original: We also study the impact of different similarity measures in Appendix F.1.
Revised: We also study the impact of different similarity measures in Appendix F.1.

Original:  Instantiations for Geometric Knowledge Distillation
Revised:  Instantiations for Geometric Knowledge Distillation

Original: Unfortunately, deriving explicit formulas for NHKs is prohibitively challenging due to introduction ofnon-linearity.
Revised: Unfortunately, deriving explicit formulas for NHKs is prohibitively challenging due to introduction ofnon-linearity.

Original: To circumvent it, we propose two types of instantiations for GKD, i.e., non-parametricand parametric.
Revised: To circumvent it, we propose two types of instantiations for GKD, i.e., non-parametricand parametric.

Original: The former considers explicit NHKs by making assumptions on the underlying space,and the latter resorts to variational inference techniques to learn NHK in a data-driven manner.
Revised: The former considers explicit NHKs by making assumptions on the underlying space,and the latter resorts to variational inference techniques to learn NHK in a data-driven manner.

Original: 4.1 Non-Parametric Geometric Distillation
Revised: 4.1 Non-Parametric Geometric Distillation

Original: Deterministic Kernel.
Revised: Deterministic Kernel.

Original: One instantiation of NHK is a Gauss-Weierstrass kernel in the form of
Revised: One instantiation of NHK is a Gauss-Weierstrass kernel in the form of

Original: Eqn.
Revised: Eqn.

Original: (6), by assuming the underlying space is a Euclidean space.
Revised: (6), by assuming the underlying space is a Euclidean space.

Original: Since the distillation loss in
Revised: Since the distillation loss in

Original: Eqn.
Revised: Eqn.

Original: (11) is a homogeneous function, we can remove its scaling factor and define NHK as
Revised: (11) is a homogeneous function, we can remove its scaling factor and deﬁne NHK as(Gauss-Weierstrass

Original: [Equation]
Revised: [Equation]

Original: where T denotes the estimation of the accumulated time interval.
Revised: where T denotes the estimation of the accumulated time interval.

Original: Alternatively, we can use Sigmoidkernel and define non-parametric NHK as:
Revised: Alternatively, we can use Sigmoidkernel and deﬁne non-parametric NHK as:

Original: [Equation]
Revised: [Equation]

Original: where a, b are positive constants depending on l and k .
Revised: where a, b are positive constants depending on l and k .

Original: It is a natural and intuitive choice as similaritymeasurement and empirically found as-well effective, albeit does not correspond to any namedmanifold to our knowledge.
Revised: It is a natural and intuitive choice as similaritymeasurement and empirically found as-well effective, albeit does not correspond to any namedmanifold to our knowledge.

Original: Randomized Kernel.
Revised: Randomized Kernel.

Original: We can also define Randomized kernel based on the following theorem.
Revised: We can also deﬁne Randomized kernel based on the following theorem.

Original: Theorem 3.
Revised: Theorem 3.

Original: (Expansion of NHK) Let { φ k ′ } ∞
Revised: (Expansion of NHK) Let { ' k 0 } 1 k 0 =

Original: k ′ =0 be orthonormal basis of eigenfunctions of− ∆( f θ , G ) with eigenvalues 0 < λ 0 ≤ λ 1 ≤ λ 2 ≤ . . .
Revised: be orthonormal basis of eigenfunctions of  ( f ✓ , G ) with eigenvalues 0 <  0   1   2  . .

Original:  , NHK allows the expansion:
Revised: . , NHK allows the expansion:

Original: [Equation]
Revised: [Equation]

Original: Based on this result, we resort to the approximation of NHK by defining a randomized kernel in asimilar form as Eqn.
Revised: Based on this result, we resort to the approximation of NHK by deﬁning a randomized kernel in asimilar form as Eqn.

Original: (15), leading to the following formulation of randomized NHK:
Revised: (15), leading to the following formulation of randomized NHK:

Original: [Equation]
Revised: [Equation]

Original: where σ ( W k ′ h i ) is used to proximate φ k ′ ( v i ) , {+W k 
+}
Revised: where  ( W k 0 h i ) is used to proximate ' k 0 ( v i ) , {+W k 
+}

Original: W k ′ =
Revised: =

Original: [ ϕ 1 ,k ′ , ϕ 2 ,k ′ , · · · , ϕ s,k ′ ] ⊤ is a transformationmatrix, ϕ ∼ N ( 0 , I d ) is a d -dimensional random variable from Gaussian distribution.
Revised: [  1 ,k 0 ,  2 ,k 0 , · · · ,  s,k 0 ] > is a transformation 200 matrix,  ⇠ N ( 0 , I d ) is a d -dimensional random variable from Gaussian distribution.

Original: In fact, undercertain choice of activation function σ , Eqn.
Revised: In fact, undercertain choice of activation function  , Eqn.

Original: (16) could approximate a diversity of kernels [37,10].
Revised: (16) could approximate a diversity of kernels [32, 9].

Original: This design essentially enforces the alignment of teacher and student for arbitrary underlyingmanifold.
Revised: Thisdesign essentially enforces the alignment of teacher and student for arbitrary underlying manifold.

Original: Training Scheme.
Revised: Training Scheme.

Original: We follow the standard training paradigm in KD literature [21, 16]: the teacheris pre-trained by a supervised prediction loss involving all labeled nodes in ˜ V .
Revised: We follow the standard training paradigm in KD literature [18, 13]: the teacheris pre-trained by a supervised prediction loss involving all labeled nodes in ˜ V .

Original: After teacher iswell-trained, we fix θ ∗ and train the student model according to
Revised: After teacher iswell-trained, we ﬁx ✓ ⇤ and train the student model according to

Original: [Equation]
Revised: [Equation]

Original: (17) where Y denotes ground-truth labels of labeled nodes in V , and ˆ Y θ denotes the predictions of studentmodel f θ on G , L dis is the distillation loss defined by Eqn.
Revised: where Y denotes ground-truth labels of labeled nodes in V , and ˆ Y ✓ denotes the predictions of studentmodel f ✓ on G , L dis is the distillation loss deﬁned by Eqn.

Original: (11), L denotes the total number of layers.
Revised: (11), L denotes the total number of layers.

Original: 4.2 Parametric Geometric Distillation
Revised: 4.2 Parametric Geometric Distillation

Original: Inheriting the similar spirit of auto-encoding Bayes [25], we introduce a variational inverse-NHK thatis independently parameterized, denoted as κ † ϕ , whose existence is guaranteed by the invertibility of
Revised: Inheriting the similar spirit of auto-encoding Bayes [22], we introduce a variational inverse-NHK thatis independently parameterized, denoted as  †  , whose existence is guaranteed by the invertibility of

Original: NHK matrices.
Revised: NHK matrices.

Original: Together with κ θ , they define a symmetric form characterizing feature propagation:
Revised: Together with  ✓ , they deﬁne a symmetric form characterizing feature propagation:

Original: [Equation]
Revised: [Equation]

Original: In practice, we follow existing kernel learning approaches [49] and use deep architectures toparameterize the variational inverse-NHK as
Revised: In practice, we follow existing kernel learning approaches [42] and use deep architectures toparameterize the variational inverse-NHK as

Original: [Equation]
Revised: [Equation]

Original: where g ϕ : R d → R s is the associated learnable non-linear mapping.
Revised: where g  : R d ! R s is the associated learnable non-linear mapping.

Original: Given a pre-trained teachermodel, distilling geometric knowledge boils down to 1) establishing equivalence of Eqn.
Revised: Given a pre-trained teachermodel, distilling geometric knowledge boils down to 1) establishing equivalence of Eqn.

Original: (18) and
Revised: (18) and

Original: Eqn. (19), and 2) matching pseudo-inverse NHK matrices for teacher and student models (respectivelydenoted as K † θ ∗ , V and K † θ with clear meanings), leading to the training scheme as follows.
Revised: Eqn. (19), and 2) matching pseudo-inverse NHK matrices for teacher and student models (respectivelydenoted as K † ✓ ⇤ , V and K † ✓ with clear meanings), leading to the training scheme as follows.

Original: Training Scheme.
Revised: Training Scheme.

Original: Based on Eqn.
Revised: Based on Eqn.

Original: (19), we can define a reconstruction loss with respect to theteacher model (similar applies to the student model) as
Revised: (19), we can deﬁne a reconstruction loss with respect to theteacher model (similar applies to the student model) as

Original: [Equation][Equation]
Revised: [Equation]

Original: where ˜ D denotes the degree matrix over ˜ G .
Revised: where ˜ D denotes the degree matrix over ˜ G .

Original: Then, minimizing the reconstruction loss with fixed GNNmodel parameter θ amounts to optimizing the variational parameter ϕ , and minimizing predictionand distillation losses given fixed ϕ amounts to optimizing the student model parameter θ :
Revised: Then, minimizing the reconstruction loss with ﬁxed GNNmodel parameter ✓ amounts to optimizing the variational parameter  , and minimizing predictionand distillation losses given ﬁxed  amounts to optimizing the student model parameter ✓ :

Original: [Equation][Equation]
Revised: [Equation]

Original: Applying two steps iteratively adds up to an EM-like algorithm for training the student model.
Revised: Applying two steps iteratively adds up to an EM-like algorithm for training the student model.

Original: Inpractice, we set l + k as the last layer, and l as the first layer to use as much information as possible.
Revised: Inpractice, we set l + k as the last layer, and l as the ﬁrst layer.

Original: Justification.
Revised: Justiﬁcation.

Original: We justify parametric GKD from a variational inference perspective.
Revised: We justify parametric GKD from a variational inference perspective.

Original: From
Revised: From

Original: Eqn.
Revised: Eqn.

Original: (18), the forward GNN model f θ defines a model distribution p θ ( H ( l ) , H ( l + k ) , K ) 
Revised: (18), the forward GNN model f ✓ deﬁnes a model distribution p ✓ ( H ( l ) , H ( l + k ) , K ) =

Original: =
Revised: p ✓ ( H ( l ) )

Original: p θ ( H ( l ) ) p θ ( K | H ( l ) )p
Revised: p ✓ ( K | H ( l ) )

Original: θ ( H ( l + k ) | K , H ( l ) ) , where p θ ( K | H ( l ) ) is intractable, hindering the proceedingdistillation.
Revised: p ✓ ( H ( l + k ) | K , H ( l ) ) , where p ✓ ( K | H ( l ) ) is intractable, hindering the proceedingdistillation.

Original: In this light, the variational inverse-NHK model κ † ϕ is proposed with a variational distribution qϕ ( H ( l ) , H ( l + k ) , K ) =
Revised: In this light, the variational inverse-NHK model  †  is proposed with a variational distribution q

Original: q ϕ ( H ( l+ k ) ) q ϕ ( K | H ( l + k )  ) q ϕ ( H ( l ) |
Revised:  ( H ( l ) , H ( l + k ) , K ) =

Original:  H 
Revised: q  ( H ( l + k ) ) q  ( K | H ( l + k ) )

Original:  ( l + k ) , K ) , which has a tractableposterior q ϕ ( K | H ( l + k ) ) .
Revised: q  ( H ( l ) | H ( l + k ) , K ) , which has a tractableposterior q  ( K | H ( l + k ) ) .

Original: Now, we justify our training scheme with iterative optimization for Eqn.
Revised: Now, we justify our training scheme with iterative optimization for Eqn.

Original: (22)and (23) by the following proposition and refer the reader to Appendix D for detailed reasoning.
Revised: (22)and (23) by the following proposition and refer the reader to Appendix D for detailed reasoning.

Original: Proposition 1.
Revised: Proposition 1.

Original: The optimization in Eqn.
Revised: The optimization in Eqn.

Original: (22) and (23) essentially minimizes the following Kullback–Leibler (KL) divergence,
Revised: (22) and (23) essentially minimizes the following Kullback–Leibler (KL) divergence,

Original: [Equation][Equation]
Revised: [Equation]

Original: and hence attempts to establish equivalence between two latent variable models p θ and q ϕ .
Revised: and hence attempts to establish equivalence between two latent variable models p ✓ and q  .

Original:  Experiments
Revised:  Experiments

Original: We conduct experiments to validate the efficacy of our method on graph-structured data in terms ofvarious types of privileged geometric knowledge, combinations of teacher-student GNN architecturesand potential application scenarios.
Revised: We conduct experiments to validate the efﬁcacy of our method on graph-structured data in terms ofvarious types of privileged geometric knowledge, combinations of teacher-student GNN architecturesand potential application scenarios.

Original: We use three benchmark datasets Cora [32], Citeseer [40],
Revised: We use three benchmark datasets Cora [27], Citeseer [35],

Original: Pubmed [36], and two larger datasets, i.e., OGBN-Proteins and OGBN-Arxiv [22], for node classification tasks.
Revised: Pubmed [31], and two larger datasets, i.e., OGBN-Proteins and OGBN-Arxiv [19], for node classiﬁcation tasks.

Original: More implementation details, experimental results, and discussions on scalability arerespectively given in Appendix E, F, and H.
Revised: More details are given in Appendix E.

Original: Implementation and Competitors We consider the following variants of the proposed GKD. 1)
Revised: Implementation and Competitors We consider the following variants of the proposed GKD. 1)

Original: GKD-G : non-parametric Gaussian NHK; 2) GKD-S : non-parametric Sigmoid NHK; 3) GKD-R : randomized NHK; 3) VGKD : parametric variational NHK.
Revised: GKD-G : non-parametric Gaussian NHK; 2) GKD-S : non-parametric Sigmoid NHK; 3) GKD-R : randomized NHK; 3) VGKD : parametric variational NHK.

Original: We choose KD methods that is representativein its own category for comparison, including KD [21], FitNets [38], FSP [56], TinyGNN [53], LSP [55].
Revised: We choose KD methods that is representativein its own category for comparison, including KD [18], FitNets [33], FSP [49], TinyGNN [46], LSP [48].

Original: Implementation details and descriptions of baselines are deferred to Appendix.
Revised: Implementation details and descriptions of baselines are deferred to Appendix.

Original: We also report theperformances of teacher and student model trained with the standard classification loss, short as
Revised: We also report theperformances of teacher and student model trained with the standard classiﬁcation loss, short as

Original: Teacher and Student .
Revised: Teacher and Student .

Original: The teacher model is trained using the complete graph ˜ G , and, to calibrate withall other methods, tested using the partial graph G .
Revised: The teacher model is trained using the complete graph ˜ G , and, to calibrate withall other methods, tested using the partial graph G .

Original: Besides, we consider an Oracle model whichis both trained and tested on ˜ G , which naturally takes an advantaged place given more informationduring inference.
Revised: Besides, we consider an Oracle model whichis both trained and tested on ˜ G , which naturally takes an advantaged place given more informationduring inference.

Original: Since our method is compatible with the vanilla KD paradigm [21], we report theperformance delivered by their combinations (i.e., GKD+KD and VGKD+KD).
Revised: Since our method is compatible with the vanilla KD paradigm [18], we report theperformance delivered by their combinations (i.e., GKD+KD and VGKD+KD).

Original: Experiment Settings We investigate on various experimental settings according to different typesof privileged geometric knowledge.
Revised: Experiment Settings We investigate on various experimental settings according to different typesof privileged geometric knowledge.

Original: In the case of edge-aware geometric knowledge , the teachermodel has access to additional edge information, i.e., E ⊂ ˜ E and V =˜ V .
Revised: In the case of edge-aware geometric knowledge , the teachermodel has access to additional edge information, i.e., E ⇢ ˜ E and V =

Revised: ˜ V .

Original: In the case of node-awaregeometric knowledge , the teacher model has access to additional node information, i.e., V ⊂ ˜
Revised: In the case of node-awaregeometric knowledge , the teacher model has access to additional node information, i.e., V ⇢ ˜ Vand E = ˜

Original: Vand E = ˜ E ∩ {V × V} .
Revised: E \ {V ⇥ V} .

Original: We also consider other conventional or novel settings including modelcompression, self-distillation, teaching MLP, which will be illustrated in detail.
Revised: We also consider other conventional or novel settings including model 257 compression, self-distillation, teaching MLP, which will be illustrated in detail.

Original: The backbone f θis set as 3 -layer GCN [26] for both student and teacher models, unless otherwise stated.
Revised: The backbone f ✓is set as 3 -layer GCN [23] for both student and teacher models, unless otherwise stated.

Original: All theexperiments are repeated five times with random initialization.
Revised: All theexperiments are repeated ﬁve times with random initialization.

Original: 5.1 Edge-Aware Geometric Knowledge
Revised: 5.1 Edge-Aware Geometric Knowledge

Original: We report results for the edge-aware geometric knowledge setting, where the teacher model hasaccess to auxiliary edges.
Revised: We report results for the edge-aware geometric knowledge setting, where the teacher model hasaccess to auxiliary edges.

Original: To quantify the privileged information, we set the quantity ( | ˜ E| − |E| ) / | ˜ E| ,called privileged information ratio (PIR), as 0 .
Revised: To quantify the privileged information, we set the quantity ( | ˜ E|  |E| ) / | ˜ E| , 263 called privileged information ratio (PIR), as 0 .

Original: As shown in Tab. 1, all variants of GKD outperformother KD baselines by a large margin on both datasets, and significantly exceeds both Student and
Revised: As shown in Tab. 1, all variants of GKD outperformother KD baselines by a large margin on both datasets, and signiﬁcantly exceeds both Student and

Original: Teacher models.
Revised: Teacher models.

Original: Further, GKD and its variants rival, if not surpass, the Oracle model.
Revised: Further, GKD and its variants rival, if not surpass, the Oracle model.

Original: In other words,the student model trained using GKD could use far less graph topological information to achieve veryclose performance to competitors that are aware of the full graph topology during inference.
Revised: In other words,the student model trained using GKD could use far less graph topological information to achieve veryclose performance to competitors that are aware of the full graph topology during inference.

Original: Thisimplies that our GKD framework could indeed effectively transfer topological information.
Revised: Thisimplies that our GKD framework could indeed effectively transfer topological information.

Original: With regard to comparison among the variants of GKD, the parametric VGKD performs better than itsnon-parametric counterparts in most cases, and GKD-R is the most effective non-parametric method
Revised: With regard to comparison among the variants of GKD, the parametric VGKD performs better than itsnon-parametric counterparts in most cases, and GKD-R is the most effective non-parametric methodin general. Despite these results, GKD-G and GKD-S are also effective while being simpler and more

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: in general.
Revised: easy-to-implement.

Original: Despite these results, GKD-G and GKD-S are also effective while being simpler and moreeasy-to-implement. We presume that the performance variation of different GKD realizations stemfrom the different geometric property governed by the feature of datasets.
Revised: We presume that the performance variation of different GKD realizations stemfrom the different geometric property governed by the feature of datasets.

Original: Thus, it is useful to choosea proper implementation of GKD that is suitable for the application.
Revised: Thus, it is useful to choosea proper implementation of GKD that is suitable for the application.

Original: 5.2 Node-Aware Geometric Knowledge
Revised: 5.2 Node-Aware Geometric Knowledge

Original: We further investigate on the node-aware geometric knowledge setting where the teacher model hasaccess to more labeled nodes and their relations with the rest nodes.
Revised: We further investigate on the node-aware geometric knowledge setting where the teacher model hasaccess to more labeled nodes and their relations with the rest nodes.

Original: We set the PIR w.r.t.
Revised: We set the PIR w.r.t.

Original: labelednodes, defined as ( | ˜ V train | − |V train | ) / |
Revised: labelednodes, deﬁned as ( | ˜ V train |  |V train | ) / |

Original: ˜ V train | , to 0 .
Revised: ˜ V train | , to 0 .

Original: A unique challenge of this setting comparedto the edge-ware counterpart is that, apart from graph topological information, the student model hasless labeled training samples, putting it into disadvantage.
Revised: A unique challenge of this setting compared 279 to the edge-ware counterpart is that, apart from graph topological information, the student model hasless labeled training samples, putting it into disadvantage.

Original: As we can see from Tab. 2.
Revised: As we can see from Tab. 2.

Original: The proposed
Revised: The proposed

Original: GKD and its variants again consistently outperform KD baselines throughout all the cases, surpassesboth Student and Teacher models, and are even as competitive as Oracle.
Revised: GKD and its variants again consistently outperform KD baselines throughout all the cases, surpassesboth Student and Teacher models, and are even as competitive as Oracle.

Original: 5.3 Results on Larger Datasets
Revised: 5.3 Results on Larger Datasets

Original: Table 3 presents results on two large graphs, i.e., OGBN-Proteins and OGBN-Arxiv.
Revised: Table 3 presents results on two large graphs, i.e., OGBN-Proteins and OGBN-Arxiv.

Original: We followthe data split in original paper [22], and use the same PIR setting as citation networks, and choosethe best variant of GKD to report in the table.
Revised: We followthe data split in original paper [19], and use the same PIR setting as citation networks, and choosethe best variant of GKD to report in the table.

Original: As shown in the table, again, we found our methodsconsistently outperform Teacher and Student models, and are close to the performance of the Oraclemodel, which suggests the effectiveness of GKD in large graphs.More details regarding how ourmethods scale to larger graphs are deferred to Appendix H.
Revised: As shown in the table, again, we found our methodsconsistently outperform Teacher and Student models, and are close to the performance of the Oraclemodel, which suggests the effectiveness of GKD in large graphs.

Original: 5.4 Performance Variation with Privileged Ratio
Revised: 5.4 Performance Variation with Privileged Ratio

Original: The results with respect to varying privileged (edge-aware and node-ware) information ratio are givenrespectively in Fig.
Revised: The results with respect to varying privileged (edge-aware and node-ware) information ratio are givenrespectively in Fig.

Original: 2 and Fig.
Revised: 2 and Fig.

Original: The performance of Oracle model is invariant as it is trained andtested on the same (complete) graph.
Revised: The performance of Oracle model is invariant as it is trained andtested on the same (complete) graph.

Original: In general, for Teacher model, Student model, and vanilla KD,their performance drops dramatically with increasing PIR quantifying the information loss, withvanilla KD’s performance lie in-between.
Revised: In general, for Teacher model, Student model, and vanilla KD,their performance drops dramatically with increasing PIR quantifying the information loss, withvanilla KD’s performance lie in-between.

Original: In contrast, our method is significantly more robust, onlyshowing slight performance deterioration, exceeding the Teacher model, and approaching the Oraclemodel.
Revised: In contrast, our method is signiﬁcantly more robust, onlyshowing slight performance deterioration, exceeding the Teacher model, and approaching the Oraclemodel.

Original: Besides, we find an interesting phenomenon that in the edge-aware setting on Pubmed dataset,the performance of Teacher model is the worst.
Revised: Besides, we ﬁnd an interesting phenomenon that in the edge-aware setting on Pubmed dataset,the performance of Teacher model is the worst.

Original: This is reasonable since the Teacher is trained usingfully observed graph, and may perform poorly once theprivileged part of graph topology, which itrelies on for accurate results, becomes unavailable at test time.[Figure]
Revised: This is reasonable since the Teacher is trained usingfully observed graph, and may perform poorly once the graph topology, which it relies on for accurateresults, becomes incomplete at test time.

Original: [Figure]
Revised: 5.5 Other Settings

Original: [Figure]
Revised: [Table]

Original: [Figure] [Figure][Figure]
Revised: [Figure] [Figure]

Original: T e s t A cc u r a c y TeacherStudentOracle KDGKD
Revised: [Table]

Original: [Figure][Figure]
Revised: [Figure]

Original: T e s t

Original: A cc u r a c y

Original: TeacherStudentOracle

Original: [Figure] [Figure] [Figure]
Revised: [Figure] [Table]

Original: T e s t A cc u
Revised: Model Compression We report the perfor-

Original: [Figure] [Figure]
Revised: [Table]

Original: u
Revised: mance of GKD in the conventional model com-

Original: [Figure]
Revised: [Table]

Original: 5.5 Other Settings
Revised: pression setting [18] where the teacher and stu-

Original: [Figure] [Table]
Revised: [Table]

Original: Model Compression We report the performance of GKD in the conventional model compression setting [21] where the teacher and student are different-sized GNN models. In thissetting, we train and test both teacher and student models on the complete graph.
Revised: dent are different-sized GNN models. In thissetting, we train and test both teacher and student models on the complete graph.

Original: We use
Revised: We use

Original: GCN as the backbone for teacher with a rel-[Table] atively large hidden size 64 , and use SGC or
Revised: GCN as the backbone for teacher with a rel- atively large hidden size 64 , and use SGC or

Original: GCN (with hidden size 8 and 16 ) as the studentmodel.
Revised: GCN (with hidden size 8 and 16 ) as the student model.

Original: We report the best result among all variants of GKD.
Revised: We report the best result among all variants of GKD.

Original: As shown in the first section of
Revised: As shown in the ﬁrst section of

Original: Tab. 4, our method achieves notable improvements over the student model, rendering it as auseful KD approach for model compression. 
Revised: Tab. 4, our method achieves notable improve- ments over the student model, rendering it as a useful KD approach for model compression. [Table] Self-Distillation We further report results for [Table]

Original: Self-Distillation We further report results forthe setting of self-distillation [15, 34], whichis a special case of KD when the teacher’s andstudent’s architectures are identical, often usedfor refining their own performance.
Revised: the setting of self-distillation [12, 29], which [Table] is a special case of KD when the teacher’s andstudent’s architectures are identical, often usedfor reﬁning their own performance.

Original: The resultsas shown in Tab.
Revised: The resultsas shown in Tab.

Original: 4 validate that GKD couldalso be used to effectively boost GNN’s ownperformance.
Revised: 4 validate that GKD could alsobe used to effectively boost GNN’s own performance.

Original: GNNs Teach MLPs Finally, we study a special setting where any instance relation is unavailable attest time, i.e., E = ∅ , and examine whether our method could compress the geometric knowledge inthis setting when GNN degrades to MLP.
Revised: GNNs Teach MLPs Finally, we study a special setting where any instance relation is unavailable attest time, i.e., E = ; , and examine whether our method could compress the geometric knowledge inthis setting when GNN degrades to MLP.

Original: The results are shown in Tab. 4.
Revised: The results are shown in Tab. 4.

Original: Notably, the performanceof conventional GNN model degrades dramatically when the topological information is not availableat the test time, while our method turns out to still effective.
Revised: Notably, the performanceof conventional GNN model degrades dramatically when the topological information is not availableat the test time, while our method turns out to still effective.

Original: It again verifies that our method couldcompress the geometric knowledge into the GNN, and also echoes recent advances in the domain ofcomputer vision, which suggest that MLP could achieve competitive performance with CNN [42].
Revised: It again veriﬁes that our method couldcompress the geometric knowledge into the GNN, and also echoes recent advances in the domain ofcomputer vision, which suggest that MLP could achieve competitive performance with CNN [36].

Original:  Conclusion
Revised:  Conclusion

Original: This paper formalizes the problem of graph topological knowledge transfer for GNNs.
Revised: This paper formalizes the problem of graph topological knowledge transfer for GNNs.

Original: We investigateon the implication of heat kernel in GNNs and propose the novel notion of neural heat kernel.
Revised: We investigate on the implication of heat kernel in GNNs and propose the novel notion of neural heat kernel.

Original: Weleverage it to characterize the geometric property of the underlying manifold for graphs, and proposethe framework of geometric knowledge distillation to transfer geometric knowledge from a teacherGNN model to a student GNN. Experimental results validate the effectiveness of our approach invarious practical settings. The source code will be made public available.
Revised: We leverage it to characterize the geometric property of the underlying manifold for graphs, and propose the framework of geometric knowledge distillation to transfer geometric knowledge from a teacher

Original: See limitations and potentialsocial impact in Appendix I.

Original: A Proof for Theorem 1

Original: Suppose ( M , µ ) is the base manifold of dimension n with respect to graph G and GNN model f θ ,and ∆ µ ( G , f θ ) is the associated weighted Laplace operator with respect to G and f θ (in the followingshortened as ∆ µ ).

Original: We leverage a Sobolev inequality on manifolds as Lemma 1.

Original: Lemma 1.

Original: [39] Let x be a function from the local Sobolev space W 2 σ loc ( M ) for a positive integerσ > n/ 4 .

Original: Then, for any relatively compact open set Ω ⊂ M and any set K ⋐ Ω , there is a constant

Original: C such that

Original: [Equation]

Original: [Equation]

Original: where the norm ∥ · ∥ W 2 σ is defined as

Original: [Equation]

Original: [Equation]

Original: Suppose σ is the smallest integer larger than n/ 4 , and P t = e − t L is the heat kernel semigroup, where

Original: L = − ∆ µ | W 2 0 is the Dirichlet Laplace operator for the base manifold regarding graph G and GNNmodel f θ , we have the following lemma.

Original: Lemma 2.

Original: For any function x ∈ L 2 ( M ) , t > 0 , and set K ⋐ M , it holds that

Original: [Equation]

Original: (27) where C is a constant depending on K, g , µ, n .

Original: Proof.

Original: Suppose { E λ } is the spectral resolution of the Dirichlet operator L for the base manifold.

Original: Consider the function Φ( λ ) =

Original: λ k e − tλ , where t > 0 and k ∈ Z + .

Original: We have

Original: [Equation]

Original: Since Φ( λ ) is bounded on [0 , + ∞ ) , L k e − t L is also bounded, and hence we have L k (cid:0) e − t L x (cid:1) ∈

Original: L 2 ( M ) .

Original: By noticing that the function λ (cid:55)→ λ k e − tλ takes its maximal value at λ = k/t , we have, forany x ∈ L 2 ,

Original: [Equation]

Original: [Equation]

Original: Using the definition of ∥ · ∥ W 2 σ in Eqn.

Original: (26) and the result of Eqn.

Original: (29), we obtain

Original: [Equation]

Original: Substituting P t x into Lemma.

Original: 1 and using the result of Eqn.

Original: 30 yields Eqn. 27, and thus completesthe proof.

Original: Consider the following equivalent expressions describing heat diffusion and feature propagationrespectively,

Original: [Equation]

Original: where τ is a constant dependent on f θ and l .

Original: We can rewrite the single-layer NHK as a function p τ,v isuch that for any v j ∈ V (or equivalently v j ∈ M )

Original: [Equation]

Original: We define g : V → R d as a function that outputs the feature of a node in l -th layer, which is clearly

Original: Lebesgue integrable by thinking of nodes embedded on the manifold, i.e., g ∈ L 2 ( M ) .

Original: Proof for the existence of unique single-layer NHK boils down to proof that for any v i ∈ M , l > 0(i.e.

Original: , τ > 0 ), there exists a unique function p τ,v

Original: i ∈

Original: L 2 ( M ) such that, for all g θ ,

Original: [Equation]

Original: Fix a relatively compact set K ⋐ M .

Original: By Lemma 2, for all τ > 0 and g θ ∈

Original: L 2 ( M ) , the function

Original: P τ g θ ( v i ) admits the estimate

Original: [Equation]

Original: Therefore, for fixed l and GNN model f θ , the mapping g θ (cid:55)→ P

Original: τ

Original: g

Original: θ is a bounded linear functional on

Original: L 2 ( M ) .

Original: By the Riesz representation theorem, there exists a function p τ,v

Original: i ∈

Original: L 2 ( M ) such that

Original: [Equation]

Original: 35) where ( · , · ) L 2 denotes inner product in L 2 , whence Eqn.

Original: (33) follows.

Original: The uniqueness of p τ,v i isevident from Eqn.

Original: (33) since for any point v i ∈ M there is a compact set K containing v i , thefunction p τ,v is defined for all τ > 0 and v ∈ M .

Original: B Proof for Theorem

Original: Lemma 3.

Original: For all v i , v j ∈ M and t > 0 , the inner product ( p s,x , p t − s,y ) does not depend ons ∈ (0 , t ] .

Original: Proof.

Original: Using P t + s = P s P t (by the definition of P t ), Eqn. (35), and the symmetry of P t , we obtainthat for all v ∈ M , t, s > 0 , and g θ ∈

Original: L 2 ( M ) , it holds that

Original: [Equation]

Original: Lemma 3 implies that, for all v i , v j ∈ M and 0 < s ≤ tp t ( v i

Original: [Equation]

Original: v k

Original: [Equation]

Original: = ( p s,v i

Original: [Equation]

Original: p t − s

Original: [Equation]

Original: v k )

Original: [Equation]

Original: Hence, it holds that

Original: [Equation]

Original: By lettings = min { τ ( l +1) , τ ( l +2) } ,

Original: [Equation]

Original: [Equation]

Original: we can rewrite Eqn.

Original: (38) as the expression of semigroup identity property of layer-wise NHK

Original: [Equation]

Original: The proof also generalizes to the cross-layer case, inducing cross-layer NHK κ θ ( v i , v j , l (cid:55)→ l + k ) .

Original: C Proof for Theorem 3

Original: Let { φ k } ∞ k =1 be an orthonormal basis of eigenfunctions of L with an increasing sequence of nonnegative eigenvalues { λ k } ∞ k =1 , where λ k → + ∞ .

Original: In consideration of

Original: [Equation]

Original: we have the following expansion of p t,v i by referring to results in literature [45]

Original: [Equation]

Original: Let T be the accumulated time interval from l -th layer to ( l + k ) -th layer, in consideration of theequivalence shown in Eqn.

Original: (32), we could write Eqn.

Original: (42) as

Original: [Equation]

Original: completing the proof.

Original: D Proof for Proposition 1

Original: In the following, we use X , Y , K to denote H ( l ) , H ( l + k ) , K ( G , l (cid:55)→ l + k ) for brevity.

Original: By definitionin Section 4.2 we have a forward GNN model f θ with the joint distribution of latent variablesp

Original: [Equation]

Original: X , Y , K )

Original: [Equation]

Original: p

Original: [Equation]

Original: [Equation]

Original: (cid:124) (cid:123)(cid:122) (cid:125)

Original: Intractable p

Original: [Equation]

Original: ( Y | X

Original: [Equation]

Original: K

Original: [Equation]

Original: (44) and a variational inverse-NHK model κ † ϕ with the joint distribution of latent variablesq ϕ (

Original: [Equation]

Original: , Y , K ) =

Original: q ϕ ( Y ) q

Original: ϕ

Original: ( K | Y ) (cid:124) (cid:123)(cid:122) (cid:125)

Original: Tractable

Original: [Equation]

Original: Based on these equations

Original: [Equation]

Original: we can write the KL-divergence between joint distributions of p θ and q ϕ as

Original: [Equation]

Original: Reconstruction Loss

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: The first term C is a constant entropy with respect to q ϕ .

Original: The second term is the KL-divergencebetween the variational posterior q ϕ

Original: ( X | Y , K ) and the prior p θ ( X ) , which corresponds to the reconstruction loss in Eqn.

Original: (22) that attempts to learn a NHK that faithfully reflects the latent heat diffusionprocess describing the GNN feature propagation.

Original: The third term is negative log-likelihood, whichcorresponds to the prediction loss in Eqn.

Original: (23) that attempts to fit the dataset.

Original: Therefore, minimizingthis KL-divergence amounts to the iterative optimization scheme of Eqn.

Original: (22) and (23).

Original: E Implementation Details

Original: We present implementation details for our experiments for reproducibility.

Original: We implement our modelas well as the baselines with Python 3.8, Pytorch 1.8 and Pytorch Geometric 2.0.

Original: All experimentsare conducted on NVIDIA V100 with 16 GB memory.

Original: All parameters are initialized with Xavierinitialization procedure.

Original: We train the model by Adam optimizer.

Original: Both teacher and student models aretrained from scratch.

Original: For the main results reported in Tab. 1 and 2, we choose the backbone model asa 3 -layer GCN with hidden size 32 .

Original: In case that the graph is too large, we use mini-batch training(draw a mini-batch of nodes from the vertex set V ) for computing the distillation loss.

Original: We use thewhole vertex set on Cora and Citeseer datasets, and choose the largest batch-size under the memoryconstraint for Pubmed dataset and other large datasets.

Original: More details can be found in Appendix H.

Original: E.1 Dataset Description

Original: We choose three benchmark citation network datasets, i.e., Cora, Citeseer and Pubmed, and twolarge-scale network datasets, i.e., OGB-Arxiv and OGB-Proteins for node classification.

Original: For citationnetworks, we randomly split them into train/valid/test data according to the ratio 2:1:1, in order tobalance (i.e., 1:1:1:1) node number of each category (student/ privilege/ valid/ test) in the node-awaresetting.

Original: For two large-scale datasets, we follow the original splitting [22] for evaluation.

Original: The statisticsof these datasets are summarized in Tab. 5.

Original: [Table]

Original: E.2 Implementation of non-parametric GKD

Original: For Gauss-Weierstrass NHK in Eqn.

Original: (13), we treat the accumulated time interval T as a hyperparameter that is consistent for all layers.

Original: For sigmoid NHK in Eqn.

Original: [-(13), we let a = 1 , b = 
-]
Revised: [-(13), we let a = 1 , b = 
-]

Original: andthe NHK is simple dot-product with activation.

Original: For randomized NHK in Eqn.

Original: (16), we let s = 2 d forthe random transformation matrix W , λ max > 0 and m are hyper-parameters, and we use tanh ( · )as activation function for σ ( · ) .

Original: Algorithm

Original: Input: Complete graph ˜ G with labels ˜

Original: Y , partial graph G with labels Y , learning rate γ , initial parameters θ ∗ , θ .

Original: Training Teacher GNN : while Not converged do

Original: Teacher model conducts feature propagation on ˜ G

Original: Calculate L pre ( ˆ Y θ ∗ , ˜ Y ) as in Eqn.

Original: (17) θ ∗ ← θ ∗ − γ ∇ θ ∗ L pre ( ˆ Y θ , ˜ Y )

Original: Save teacher model as f θ ∗ Training Student GNN :
Revised: GNN model to a student GNN.

Original: Load teacher model f θ ∗ while Not converged do

Original: Teacher model conducts feature propagation on ˜ G

Original: Student model conducts feature propagation on G

Original: Calculate L pre and L dis as in Eqn.

Original: (17) θ ←

Original: [Equation]

Original: 1 L ( l ) dis

Original: E.3 Implementation of parametric GKD

Original: For the case of parametric GKD, we realize the non-linear mapping g ϕ as a neural network one-layerfeed-forward neural network with tanh ( · ) activation, and set s = 2 d , the same as the non-parametricsetting.

Original: In case that the teacher model has larger hidden size than the student model, we followstandard approaches in feature-based knowledge distillation methods [20] and use independentmappings for teacher and student models that are customized for their own hidden sizes.

Original: We usean EM-style algorithm for training the student model as in Eqn.

Original: (22) and Eqn.

Original: For variants of

Original: GKD+KD and VGKD+KD, we consider additional standard label-based distillation loss in [21] withrespect to labeled nodes, inducing an additional hyper-parameter α kd that controls its importance.

Original: Training Teacher GNN : while Not converged do

Original: Teacher model conducts feature propagation on ˜ G

Original: Calculate L pre ( ˆ Y θ ∗ , ˜ Y ) as in Eqn.

Original: (17) θ ∗ ← θ ∗ − γ 1 ∇ θ ∗ L pre ( ˆ Y θ , ˜ Y )

Original: Save teacher model as f θ ∗

Original: Training Student GNN :

Original: Load teacher model f θ ∗ while Not converged do

Original: Teacher model conducts feature propagation on ˜ G

Original: Optimization for ϕ

Original: Student model conducts feature propagation on G

Original: Calculate reconstruction loss L rec as in Eqn.

Original: (22) ϕ

Original: ← ϕ − γ 2 ∇ ϕ L rec

Original: Optimization for θ

Original: Student model conducts feature propagation on G

Original: Calculate L pre and L dis as in Eqn.

Original: (23) θ ←

Original: [Equation]

Original: , Y ) + α L dis

Original: E.4 Descriptions and Implementations of Baselines

Original: KD [21]: is the seminal work of knowledge distillation, which uses the predictions of the teachermodel as soft labels to teach the student model.

Original: FitNets [38]: is the representative work of feature-based knowledge distillation, using the intermediatelayers of training instances to teach the student model.

Original: FSP [56]: is a relation-based knowledge distillation method, using the Gram matrix between twointermediate layers to explore the relationships between different feature maps.

Original: TinyGNN [53]: is a knowledge distillation method that adopts a neighbor distillation strategy tolearn local structure knowledge from a deeper GNN implicitly.

Original: LSP [55]: is a knowledge distillation method for graph convolutional networks, which aims to matchthe local distribution (similarity with adjacent nodes) of teacher and student models.

Original: Note that some of these baselines (FitNets and FSP) are originally designed specifically for computervision tasks, we adapt them to the setting of graph neural networks with slight modifications.

Original: While

Original: FSP is designed for graph neural networks, its original formulation of distillation loss does notaccount for the difference of graph topology for teacher and student models.

Original: To make it compatiblewith the settings considered in this paper, we fix the definition of local structures with respect toeither student or teacher graph topology, and choose the best result for report.

Original: We refer to thehyper-parameter settings in their papers and also finetune them on different datasets.

Original: E.5 Comparisons with Baselines

Original: There are mainly four different types of distilling strategies, namely response-based KD [21] (whichuses output layer of the teacher model to teach student), feature-based KD [38, 58, 20] (whichmatches intermediate layers of teacher and student), relation-based KD [56, 57] (which aligns therelationship between different layers or samples), and graph-based KD [53, 55] (which considersthe graph information or designed for GNN).

Original: For experiments, we choose representative methodfrom each category as baselines.

Original: While these existing distillation strategies have shown remarkablesuccess in distillation tasks such as model compression, they rarely (carefully) study the role of graphgeometry in GNN iterations.

Original: For geometric knowledge transfer task, it is crucial to find a fundamentaland principled way to track how graph topology affects the behavior of a specific GNN.

Original: Therefore,we first probe the intersection between KD and geometric learning, propose NHK to characterizegeometric knowledge and propose different variants of GKD that are shown to be effective especiallyin the geometric knowledge transfer setting.

Original: F More Experimental ResultsF.1 Different Similarity Measures

Original: We further investigate on the impact of different similarity measures in the definition of geometricknowledge distillation loss function Eqn.

Original: Specifically, we consider the following three similaritymeasures including, L2-Distance, L1-Distance and Dot-Product, whose corresponding matrix distanceare respectively defined as:

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: The results are shown in Tab. 6,7 and 8.

Original: We can see that in most cases L2-Distance performs betterthan other similarity measurements, and thus we use its variant as formulation of our definition ofmatrix similarity measurement.

Original: Specifically, when δ = 1 , Eqn.

Original: (12) degrades to L2-Distance.

Original: Also,

Original: L1-Distance in some cases exceeds L2-Distance, which renders it as an alternative approach forsimilarity measurement in other scenarios.

Original: In contrast, Dot-Product only works in some rare casesand thus is out of our consideration in practice.

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: F.2 Hyper-Parameter Analysis

Original: For parameter tuning, we adopt grid search method to search for hyper-parameters on validationset.

Original: Since our model is only sensitive to α (which will be discussed next), one can use other moreefficient searching strategies instead of grid search (e.g., coordinate descent, Bayesian optimization)to achieve very similar results.

Original: The descriptions for several hyper-parameters and their search spacesare shown in Tab. 9.

Original: Specifically, we investigate on the impact of hyper-parameter α , namely the weight of the geometricdistillation loss, on the model performance.

Original: The results are shown in Tab. 10.

Original: As we can see from thetable, for all variants of GKD, the test accuracy increases and then falls with the hyper-parameteralpha changing from low to high values, achieving the peak accuracy within [1 , 1000] .

Original: Amongall variants, VGKD has the best peak performance, while GKD-R is the most stable one whoseperformance does not vary dramatically with increasing α .

Original: F.3 More Results on Social Networks

Original: We supplement more experiment results on Reddit, a social network, as is shown in Table 11 inedge-aware setting.

Original: We found that our method still performs better than Teacher and close to Oracle.

Original: [Table]

Original: F.4 Evaluation for Online Distillation

Original: Online distillation in dynamic graphs is also an interesting application scenario where our methodmight be useful.

Original: As an exploration in this direction, we provide new results for online distillation(where teacher and student are simultaneously trained) on Cora in Table 13.

Original: We can see thatour model’s performance is similar to that in the offline setting except for taking more time toconverge (which is an intrinsic limitation of online distillation).

Original: As a pioneering in the directionof geometric knowledge distillation, we stay focused in our paper for a general methodology andstandard evaluation settings in the experiments.

Original: We leave more investigation on online distillation,which is promising yet orthogonal to the main body of our work, for future works.

Original: [Table]

Original: [Table]

Original: G More Discussions on the Equivalence of Eqn.7The

Original: equivalence of two equations in Eqn.

Original: 7 is based on recent works [8, 7, 48, 13, 12] that builtconnection between heat equation and GNN.

Original: The main result of these works is that by treating nodefeatures H as signal X (corresponding to x ( u, t ) in heat equation Eqn. 2) on the graph, solving theheat equation with Euler scheme yields the formulation of a GNN layer.

Original: In other words, the GNNcan be seen as the discretisations of the continuous diffusion process described by the heat equation.

Original: Correspondingly in Eqn. 2, the left equation is a general GNN layer (corresponding to discretizeddiffusion process), and the right equation is directly derived from Eqn.

Original: 2 (corresponding to continuousdiffusion process).

Original: Moreover, different definitions of Laplace-Beltrami operator ∆ yield different GNNs.

Original: To be morespecific, the simplest definition [8, 48] of ∆ is by letting the gradient operator ∇ denote assigningeach edge the difference of adjacent nodes’ features, and the divergence operator ∇ ∗ denote thesummation of edge features obtained from last step.

Original: In this case, setting τ in Eqn.

Original: 2 as 1 yields

Original: SGC [50], and since it is being pointed out to be a potential cause of over-smoothing (which is oneexample to show the thermodynamic and geometric perspective is helpful for understanding GNNs),the authors in [48] further set smaller τ which yields DGC.

Original: Reinterpreting ∇ ∗ as weighted sum ofedge features yields GAT [8, 46] and any other GNN following the message passing scheme:

Original: MessagePassing

Original: [Equation]

Original: where N u denotes neighbored nodes of u , and s denotes a parametric or non-parametric similarityscore.

Original: The neighborhood summation could also be modified by changing the definition of ∇ ,which yields GNNs based on learned structures (e.g., GDC [28]) and those with residual links (e.g.,

Original: APPNP [27]).

Original: Further considering different discretization schemes (e.g., implicit scheme, multistep schemes) yields different variants of GRAND [8, 7, 41] that are found more powerful than itssimplification (presumably because it better matches the continuous diffusion process).

Original: Unfortunately, not all GNNs have a simple form of ∆ , and for some of them, whether there existssuch ∆ is an open research question.

Original: Therefore, we write the operator as ∆( f θ , G ) to associateit with model f θ and use equivalence in Eqn. 2 as an analytical assumption.

Original: The thermodynamicand geometric perspective used in the paper is fundamental and useful for studying the geometryproperty of GNNs, which has also been adopted by other works [3, 43, 48, 41, 8, 7].

Original: For example,

Original: [3] attempts to explain heterophily and oversmoothing problems in GNNs by connecting GNNsto the heat equation defined by the sheaf Laplace-Beltrami operator ∆ F and propose new GNNarchitectures based on the theory.

Original: Similarly, [41, 8, 7] propose new GNN architectures / rewiringmethods from the same thermodynamic perspective, [48] explains over-smoothing issue of SGC basedon the same equivalence of GNN and solve of heat equation, and [43] explains the over-squashingissue from a geometric perspective by analyzing the curvature.

Original: A common trait of these works is todraw analogies between GNNs and differential geometry / diffusion process to obtain meaningfultheoretical results (ours: NHK as a characterization of GNN’s underlying geometry) that are used toguide implementation (ours: GKD for geometric knowledge transfer).

Original: H Scalability AnalysisH.1 How Does Our Method Scale to Large Graphs?

Original: Suppose the size of partial graph (i.e., the small one) is n , then the computation complexity fordistillation loss (in both non-parametric and parametric cases) is O ( dn 2 ) , and the extra spaceconsumption is O ( n 2 ) .

Original: Directly using whole graph is impractical when the partial graph is large.

Original: Therefore, as mentioned inappendix E.2, we recommend using mini-batch training for large graphs, which has already beenadopted for training two large-scale datasets in our experiment.

Original: It could also be seamlessly integratedwith the original mini-batch method (by sampling ego-graphs) used by large graphs without furthermodification.

Original: Our method could be run in most GPUs within a reasonable amount of time.

Original: Moreover,we want to remark the following points:

Original: • The distillation loss for mini-batch training is not biased as long as nodes are evenly sampled,and thus will not harm the performance.

Original: We support this by showing GKD’s performance

Original: with decreasing batch size using mini-batch training in Table.

Original: Notably, the distilled student model is efficient for inference because of sparser graph

Original: structure and perhaps smaller model size, which is an important factor for those who resort

Original: to KD.

Original: • Even in the case where we somehow must use the whole large graph for calculating the loss

Original: (which is extremely rare or doesn’t exist), there are ways to reduce the complexity (e.g.,

Original: pre-computing teacher’s NHK matrix, using low-rank approximation).

Original: [Table]

Original: H.2 Convergence Speed

Original: We also supplement more comparisons with other peer KD methods in terms of convergence epochsin Table 15.

Original: As we can see from the table:

Original: • The EM-style algorithm used for training VGKD does not cost too much extra epochs

Original: for convergence than the non-parametric GKD.

Original: The result is intuitive by noting that

Original: the

Original: learnable mapping g ϕ used in Eqn.

Original: (22) is an independent module since it only "calibrates"

Original: the distillation loss and does not affect GNN’s feed-forward computation.

Original: It is worth

Original: mentioning that the model needs not to wait for Eqn.

Original: (22) to converge to achieve good

Original: performance (though this step is necessary for further improvement) considering the fact

Original: that even using random mapping function (i.e., GKD-R) is effective.

Original: • Both proposed models can converge within reasonable number of epochs (less than 200).

Original: Such a result is important since we usually use a fixed epoch number (e

Original: g. 200) as the default

Original: setting for training as recommended in [26].

Original: In this case, the training time is proportional tothe time per epoch, which will be discussed in detail next.

Original: [Table]

Original: • In broader context, existing KD methods [16, 21, 38, 56] naturally cost more time than

Original: [Table]

Original: the non-KD counterpart (the teacher and student models), which is even amplified in our

Original: [Table]

Original: geometric knowledge transfer setting because of the inconsistency of graph topology that

Original: [Table]

Original: makes it harder for student to mimic the behavior of teacher.

Original: Similarly, in our work, it is

Original: [Table]

Original: normal that the proposed GKD costs more time than the pure teacher/student models due

Original: [Table]

Original: to the additional loss term for transferring geometric knowledge.

Original: Such extra time is often

Original: [Table]

Original: acceptable and unimportant in this area due to that the more important factors of interest are

Original: [Table]

Original: the inference efficiency and effectiveness of student model.

Original: [Table]

Original: We can see that the convergence speed of our models are on par with other KD baselines, while

Original: [Table]

Original: outperforming them significantly in terms of testing performance (as shown in Tab. [Table] H.3 Training Time We report the the training time per epoch of different methods in Table 16.
Revised: Experimental results validate the effectiveness of our approach in various practical settings.

Original: Remarkably, our parametric version in the worst case takes less than two times of the training time ofvanilla KD, and is even more efficient than the relation-based KD method FSP that aims to matchflow of solution procedure (FSP) matrices.

Original: In practice, the whole training in citation networks (in onerun) can be done within a minute, which is very efficient.

Original: Also, our models successfully scale to large OGB datasets with comparable time as other models andyielding significantly better performance, which again verifies the efficacy of our models.

Original: [Table]

Original: I Limitations and Potential Impacts

Original: We thoroughly discuss the limitations of the present work that we could identify at current stage fromthe following two aspects.

Original: Theoretical Analysis .

Original: While the equivalence that connects GNNs and heat diffusion on manifold weuse for theoretical analysis only strictly holds for some specific GNN instantiations including SGC,

Original: APPNP, GAT and more others, as illustrated in Appendix G, it is also adopted by many of other worksas meaningful interpretations and insights [3, 48, 41, 8, 7].

Original: In practice, this is fundamental and usefulfor studying the geometry property of GNNs and provides a principled perspective for understandingthe behaviors of common GNN models based on which more complex GNN architectures (e.g.,[41, 3]) are developed.

Original: Furthermore, the methodology introduced by this paper, i.e., the geometricknowledge distillation approaches with the non-parametric and parametric versions, is applicable forarbitrary off-the-shelf GNN architectures beyond the above-mentioned GNN instantiations.

Original: Whileour experiments mainly focus on GCN, SGC and GAT as basic representatives for evaluation purpose, we believe the methods can potentially combine with more powerful models for further performanceimprovement in the problem of our interest and be extended to other promising applications involvinggeometric knowledge transfer, e.g., cross-graph domain generalization.

Original: We leave these interestingexplorations for future works.

Original: [Table]

Original: Time Cost and Complexity .

Original: The computation complexity of GKD is O ( dn 2 ) and the parametricinstantiation may take some extra time to converge compared to the pure non-KD counterpart (pureteacher and student models), as specifically shown in Appendix H. However, such extra training timeis acceptable and relatively unimportant given two facts: 1) the total training time on citation networks(in one run) is within one minute, which is quite efficient and convenient for model development; 2)with slight sacrifice on time cost, the advantage brought by our model is significant, i.e., allowing forgeometric knowledge transfer between two models, which is non-trivial to achieve. Moreover, withmini-batch training, our model can successfully scale to large graphs, i.e., the OGB datasets, withoutcompromising the performance on testing data, as shown in our experiments in Appendix H. Therealso exists other potential ways, such as low-rank approximation, as illustrated in Appendix H.1,which can be used for further accelerating the current models. As a pioneering work in geometricknowledge distillation, we stay focused on introduction of new methodology and leave such anorthogonal improvement for future investigation. Finally, our work contributes from both theoretical and practical sides, and we do not foresee anydirect negative societal impacts.
Revised: The source code will be made public available