Original: Capturing Graphs with Hypo-Elliptic Diffusions
Revised: Capturing Graphs with Hypo-Elliptic Diffusions

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Convolutional layers within graph neural networks operate by aggregating information about local neighbourhood structures; one common way to encode suchsubstructures is through random walks.
Revised: Convolutional layers within graph neural networks operate by aggregating information about local neighbourhood structures; one common way to encode suchsubstructures is through random walks.

Original: The distribution of these random walksevolves according to a diffusion equation deﬁned using the graph Laplacian.
Revised: The distribution of these random walksevolves according to a diffusion equation deﬁned using the graph Laplacian.

Original: Weextend this approach by leveraging classic mathematical results about hypo-ellipticdiffusions.
Revised: Weextend this approach by leveraging classic mathematical results about hypo-ellipticdiffusions.

Original: This results in a novel tensor-valued graph operator, which we callthe hypo-elliptic graph Laplacian.
Revised: This results in a novel tensor-valued graph operator, which we callthe hypo-elliptic graph Laplacian.

Original: We provide theoretical guarantees and efﬁcientlow-rank approximation algorithms.
Revised: We provide theoretical guarantees and efﬁcientlow-rank approximation algorithms.

Original: In particular, this gives a structured approachto capture long-range dependencies on graphs that is robust to pooling.
Revised: In particular, this gives a structured approachto capture long-range dependencies on graphs that is robust to pooling.

Original: Besides theattractive theoretical properties, our experiments show that this method competeswith graph transformers on datasets requiring long-range reasoning but scales onlylinearly in the number of edges as opposed to quadratically in nodes
Revised: Besides theattractive theoretical properties, our experiments show that this method competeswith graph transformers on datasets requiring long-range reasoning but scales onlylinearly in the number of edges as opposed to quadratically in nodes

Original:  Introduction
Revised:  Introduction

Original: Obtaining a latent description of the non-Euclidean structure of a graph is central to many applications.
Revised: Obtaining a latent description of the non-Euclidean structure of a graph is central to many applications.

Original: One common approach is to construct a set of features for each node that represents the localneighborhood of this node; pooling these node features then provides a latent description of the wholegraph.
Revised: One common approach is to construct a set of features for each node that represents the localneighborhood of this node; pooling these node features then provides a latent description of the wholegraph.

Original: A classic way to arrive at such node features is by random walks: at the given node one startsa random walk, and extracts a summary of the local neighbourhood from its sample trajectories.
Revised: A classic way to arrive at such node features is by random walks: at the given node one startsa random walk, and extracts a summary of the local neighbourhood from its sample trajectories.

Original: Werevisit this random walk construction and are inspired by two classical mathematical results:
Revised: Werevisit this random walk construction and are inspired by two classical mathematical results:

Original: Hypo-elliptic Laplacian.
Revised: Hypo-elliptic Laplacian.

Original: In the Euclidean case of Brownian motion B = ( B t ) t ≥ 0 evolving in R n ,the quantity u ( t, x ) =
Revised: In the Euclidean case of Brownian motion B = ( B t ) t ≥ 0 evolving in R n ,the quantity u ( t, x ) =

Original: E [ f ( B t ) | B 0 = x ] solves the heat equation ∂ t u = ∆ u on [0 , ∞ )
Revised: E [ f ( B t ) | B 0 = x ] solves the heat equation ∂ t u = ∆ u on [0 , ∞ )

Original: × R n ,u (0 , x ) = f
Revised: × R n ,u (0 , x ) = f

Original: ( x ) .
Revised: ( x ) .

Original: Seminal work of Gaveau [23] in the 1970s shows that if one replaces f ( B t )in the expectation by a functional of the whole trajectory, F ( B s : s ∈ [0 , t ]) , then a pathdependent heat equation can be derived where the classical Laplacian ∆ is replaced by thehypo-elliptic Laplacian.
Revised: Seminal work of Gaveau [22] in the 1970s shows that if one replaces f ( B t )in the expectation by a functional of the whole trajectory, F ( B s : s ∈ [0 , t ]) , then a pathdependent heat equation can be derived where the classical Laplacian ∆ is replaced by theso-called hypo-elliptic Laplacian.

Original: Free Algebras.
Revised: Free Algebras.

Original: A simple way to capture a sequence – for us, a sequence of nodes visited by a randomwalk on a graph – is to associate with each sequence element an element in an algebra 1and multiply these algebra elements together.
Revised: A simple way to capture a sequence – for us, a sequence of nodes visited by a randomwalk on a graph – is to associate with each sequence element an element in an algebra 1and multiply these algebra elements together.

Original: If the algebra multiplication is commutative,the sequential structure is lost but if it is non-commutative, this captures the order in thesequence.
Revised: If the algebra multiplication is commutative,the sequential structure is lost but if it is non-commutative, this captures the order in thesequence.

Original: In fact, by using the free associative algebra, this can be done faithfully and linearfunctionals of this algebra correspond to functionals on the space of sequences.
Revised: In fact, by using the free associative algebra, this can be done faithfully and linearfunctionals of this algebra correspond to functionals on the space of sequences.

Original: We leverage these ideas from the Euclidean case of R d to the non-Euclidean case of graphs.
Revised: We leverage these ideas from the Euclidean case of R d to the non-Euclidean case of graphs.

Original: Inparticular, we construct features for a given node by sampling from a random walk started at this node, but instead of averaging over the end points, we average over path-dependent functions.
Revised: Inparticular, we construct features for a given node by sampling from a random walk started at thisnode, but instead of averaging over the end points, we average over path-dependent functions.

Original: Putinformally, instead of asking a random walker that started at a node, "What do you see now?"
Revised: Put informally, instead of asking a random walker that started at a node, "What do you see now?"

Original: after ksteps, we ask "What have you seen along your way?" .
Revised: after ksteps, we ask "What have you seen along your way?" .

Original: The above notions from mathematics about thehypo-elliptic Laplacian and the free algebra allow us to formalize this in the form of a generalizedgraph diffusion equation and we develop algorithms that make this a scalable method.
Revised: The above notions from mathematics about thehypo-elliptic Laplacian and the free algebra allow us to formalize this in the form of a generalizedgraph diffusion equation and we develop algorithms that make this a scalable method.

Original: Related Work.
Revised: Related Work.

Original: From the ML literature, [49, 27] popularized the combination of deep learningarchitectures to capture random walk histories.
Revised: From the ML literature, [46, 25] popularized the combination of deep learningarchitectures to capture random walk histories.

Original: Such ideas have been incorporated, sometimes implicitly, into graph neural networks (GNN) [53, 9, 54, 18, 29, 5, 33] that in turn build on convolutionalapproaches [37, 38, 27], as well as their combination with attention or message passing [45, 65, 24],and more recent improvements [72, 46, 43, 15] that provide and improve on theoretical guarantees.
Revised: Such ideas have been incorporated, sometimes implicitly, into graph neural networks (GNN) [49, 9, 50, 17, 27, 5, 31] that in turn build on convolutionalapproaches [35, 36, 25], as well as their combination with attention or message passing [43, 59, 23],and more recent improvements [66, 44, 41, 14] that provide and improve on theoretical guarantees.

Original: Another classic approach are graph kernels, see [7] for a recent survey; in particular, the seminalpaper [36] explored the connection between diffusion equations and random walkers in a kernellearning context.
Revised: Another classic approach are graph kernels, see [7] for a recent survey; in particular, the seminalpaper [34] explored the connection between diffusion equations and random walkers in a kernellearning context.

Original: More recently, [14] proposed sequence kernels to capture the random walk history.
Revised: More recently, [13] proposed sequence kernels to capture the random walk history.

Original: Furthermore, [17] uses the signature kernel maximum mean discrepancy (MMD) [16] as a metricfor trees which implicitly relies on the algebra of tensors that we use, and [48] aggregates randomwalk histories to derive a kernel for graphs.
Revised: Furthermore, [16] uses the signature kernel maximum mean discrepancy (MMD) [15] as a metric fortrees which implicitly relies on the algebra of tensors that we use, and [45] aggregates random walkhistories to derive a kernel for graphs.

Original: Moreover, the concept of network motifs [44, 55] relatesto similar ideas that describe a graph by node sequences.Further, the Bethe Hessian [52] has beensuccessfully used in spectral clustering and shares the same goal of capturing pathdependence viaa "deformed Laplacians", although the mathematical approach is very different to ours.
Revised: Moreover, the concept of network motifs [42, 51] relates tosimilar ideas that describe a graph by node sequences.

Original: Directlyrelated to our approach is the topic of learning diffusion models [35, 13, 60, 20, 11] on graphs.
Revised: Directly related to our approach is the topic oflearning diffusion models [33, 12, 54, 19, 10] on graphs.

Original: Whilesimilar ideas on random walks and diffusion for graph learning have been developed by differentcommunities, our proposed method leverages these perspectives by capturing random walk historiesthrough a novel diffusion operation.
Revised: While similar ideas on random walks anddiffusion for graph learning have been developed by different communities, our proposed methodleverages these perspectives by capturing random walk histories through a novel diffusion operation.

Original: Our main mathematical inﬂuence is the seminal work of Gaveau [23] from the 1970s that showshow Brownian motion can be lifted into a Markov process evolving in a free algebra to capturepath-dependence.
Revised: Our main mathematical inﬂuence is the seminal work of Gaveau [22] from the 1970s that showshow Brownian motion can be lifted into a Markov process evolving in a free algebra to capturepath-dependence.

Original: This leads to a heat equation governed by the hypo-elliptic Laplacian.
Revised: This leads to a heat equation governed by the hypo-elliptic Laplacian.

Original: Theseinsights had a large inﬂuence in PDE theory, see [51, 31], but it seems that their discrete counterparton graphs has received no attention despite the well-developed literature on random walks on graphsand general non-Euclidean objects, [68, 19, 26, 63].
Revised: Theseinsights had a large inﬂuence in PDE theory, see [48, 29], but it seems that their discrete counterparton graphs has received no attention despite the well-developed literature on random walks on graphsand general non-Euclidean objects, [62, 18, 24, 57].

Original: A key challenge to go from theory to applicationis handling the computational complexity.
Revised: A key challenge to go from theory to applicationis to handle the computational complexity.

Original: To do so, we build on ideas from [62] to design effectivealgorithms for the hypo-elliptic graph diffusion.
Revised: To do so, we build on ideas from [56] to design effectivealgorithms for the hypo-elliptic graph diffusion.

Original: Contribution and Outline.
Revised: Contribution and Outline.

Original: We introduce the hypo-elliptic graph Laplacian which allows to effectively capture random walk histories through a generalized diffusion model
Revised: We introduce the hypo-elliptic graph Laplacian which allows to effectively capture random walk histories through a generalized diffusion model.

Original: • In Section 3, we introduce the hypo-elliptic variants of standard graph matrices such as
Revised: • In Section 3, we introduce the hypo-elliptic variants of standard graph matrices such as

Original: the adjacency matrix and (normalized) graph Laplacians.
Revised: the adjacency matrix and (normalized) graph Laplacians.

Original: These hypo-elliptic variants are
Revised: These hypo-elliptic variants are

Original: formulated in terms of tensor-valued matrices rather than scalar-valued matrices, and can be
Revised: formulated in terms of tensor-valued matrices rather than scalar-valued matrices, and can be

Original: manipulated using linear algebra in the same manner as the classical setting.
Revised: manipulated using linear algebra in the same manner as the classical setting.

Original: • The hypo-elliptic Laplacian leads to a corresponding diffusion model, and in Theorem 1,weshow that the solution to this generalized diffusion equation summarizes the microscopic
Revised: • The hypo-elliptic Laplacian leads to a corresponding diffusion model, and in Theorem 1,

Revised: we

Revised: show that the solution to this generalized diffusion equation summarizes the microscopic

Original: picture of the entire history of random walks and not just their location after k steps.
Revised: picture of the entire history of random walks and not just their location after k steps.

Original: • This solution provides a rich description of the local neighbourhood about a node, which
Revised: • This solution provides a rich description of the local neighbourhood about a node, which

Original: can either be used directly as node features or be pooled over the graph to obtain a latent
Revised: can either be used directly as node features or be pooled over the graph to obtain a latent

Original: description of the graph.
Revised: description of the graph.

Original: Theorem 2 shows that these node features characterize random
Revised: Theorem 2 shows that these node features characterize random

Original: walks on the graph, and we provide an analogous statement for graph features in Appendix E.
Revised: walks on the graph, and we provide an analogous statement for graph features in Appendix E.•

Original: • One can solve the hypo-elliptic graph diffusion equation directly with linearalgebra, but this
Revised: In principle, one can solve the hypo-elliptic graph diffusion equation directly with linear

Original:  is computationally prohibitive and Theorem 3 provides an efﬁcient low-rank approximation.
Revised: algebra, but this is computationally prohibitive and Theorem 3 provides an efﬁcient low-rank approximation.

Original: • Finally, Section 5 provides experiments and benchmarks.
Revised: • Finally, Section 5 provides experiments and benchmarks.

Original: A particular focus is to test theability of our model to capture long-range interactions between nodes and the robustness ofpooling operations which makes it less susceptible to the "over-squashing" phenomenon [2].
Revised: A particular focus is to test the ability of our model to capture long-range interactions between nodes and the robustness of pooling operations which makes it less susceptible to the "over-squashing" phenomenon [2].

Original:  Sequence Features by Non-Commutative Multiplication.
Revised:  Sequence Features by Non-Commutative Multiplication.

Original: We deﬁne the space of sequences in R d by
Revised: We deﬁne the space of sequences in R d by

Original: [Equation]
Revised: Seq ( R d ) := ∞ (cid:91) k

Original:  where elements are sequences denoted by x = ( x 0 , x 1 , . . 
Revised: = ( R d ) k +1 , where elements are sequences denoted by x = ( x 0 , x 1 , . . .

Original: , x k ) ∈ ( R d )
Revised: , x k ) ∈ ( R d )

Original: k +1 .
Revised: k +1 .

Original: Assume we are given aninjective map, which we call the algebra lifting ,ϕ [Equation] R d → H. from R d into an algebra H .
Revised: Assume we are given aninjective map, which we call the algebra lifting ,ϕ : R d → H. from R d into an algebra H .

Original: We can use this to deﬁne a sequence feature map 

Revised: We can use this to deﬁne a sequence feature map

Original: [Equation]
Revised: [Equation]

Original: where δ 0 x = x 0 and δ i x := x i − x i − 1 for i ≥ 1 are used to denote the increments of a sequencex =
Revised: where δ 0 x = x 0 and δ i x := x i − x i − 1 for i ≥ 1 are used to denote the increments of a sequencex =

Original: ( x 0 , . . .
Revised: ( x 0 , . . .

Original: , x k ) .
Revised: , x k ) .

Original: This map associates to any sequence x ∈ Seq ( R d ) an element of the algebra H .
Revised: This map associates to any sequence x ∈ Seq ( R d ) an element of the algebra H .

Original: Ifthe multiplication in H is commutative, then the map (cid:101) ϕ would have no information about the order ofincrements, i.e. ϕ ( δ 0 x ) · · · ϕ ( δ k x ) =
Revised: Ifthe multiplication in H is commutative, then the map (cid:101) ϕ would have no information about the order ofincrements, i.e. ϕ ( δ 0 x ) · · · ϕ ( δ k x ) =

Original: ϕ
Revised: ϕ

Original: ( δ π (0) x ) · · · ϕ ( δ π ( k ) x ) for any permutation π of { 0 , . . .
Revised: ( δ π (0) x ) · · · ϕ ( δ π ( k ) x ) for any permutation π of { 0 , . . .

Original: , k } .
Revised: , k } .

Original: However, if the multiplication in H is "non-commutative enough" we expect (cid:101) ϕ to be injective.
Revised: However, if the multiplication in H is "non-commutative enough" we expect (cid:101) ϕ to be injective.

Original: A Free Construction.
Revised: A Free Construction.

Original: Many choices for H are possible, but intuitively it makes sense to use the"most general object" for H .
Revised: Many choices for H are possible, but intuitively it makes sense to use the"most general object" for H .

Original: The mathematically rigorous approach is to use the free algebra over
Revised: The mathematically rigorous approach is to use the free algebra over

Original: R d and we give a summary in Appendix B. Despite this abstract motivation, the algebra H has aconcrete form: it is realized as a sequence of tensors in R d of increasing degree, and is deﬁned by
Revised: R d and we give a summary in Appendix B. Despite this abstract motivation, the algebra H has aconcrete form: it is realized as a sequence of tensors in R d of increasing degree, and is deﬁned by

Original: [Equation]
Revised: [Equation]

Original: where by convention ( R d ) ⊗ 

Revised: where by convention ( R d ) ⊗

Original: = R , and we describe the norm (cid:107) v (cid:107) in the paragraph below.
Revised: = R , and we describe the norm (cid:107) v (cid:107) in the paragraph below.

Original: Forexample, if v = ( v m ) m ≥ 0 ∈ H , then v 0 is a scalar, v 1 is a vector, v 2 ∈ ( R d ) ⊗ 2 is a d × d matrix,and so on.
Revised: Forexample, if v = ( v m ) m ≥ 0 ∈ H , then v 0 is a scalar, v 1 is a vector, v 2 ∈ ( R d ) ⊗ 2 is a d × d matrix,and so on.

Original: The vector space structure of H is given by addition and scalar multiplication according to
Revised: The vector space structure of H is given by addition and scalar multiplication according tov + w :=

Original: [Equation]
Revised: [Equation]

Original:  and
Revised: v m + w m ) m ≥ 0 ∈ H and

Original: [Equation]
Revised: [Equation]

Original: for λ ∈ R ,
Revised: for λ ∈ R ,

Original: and the algebra structure is given by
Revised: and the algebra structure is given by

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: An Inner Product.
Revised: An Inner Product.

Original: If e 1 , . .
Revised: If e 1 , . .

Original: , e d is a basis of R d , then every tensor v m ∈ ( R d ) ⊗ m can be written as
Revised: , e d is a basis of R d , then every tensor v m ∈ ( R d ) ⊗ m can be written as

Original: [Equation]
Revised: [Equation]

Original: This allows us to deﬁne an inner product (cid:104)· , ·(cid:105) m on ( R d ) ⊗ m by extending(cid:104) 
Revised: This allows us to deﬁne an inner product (cid:104)· , ·(cid:105) m on ( R d ) ⊗ m by extending(cid:104) e i 1 ⊗ · · · ⊗

Original: [Equation]
Revised: [Equation]

Original: (cid:26)
Revised: ⊗ · · · ⊗

Original: [Equation][Equation]
Revised: [Equation]

Original: i m
Revised: j m (cid:105) m

Original: [Equation]
Revised: [Equation]

Original: j m
Revised: (cid:26) 1 : i

Original: [Equation]
Revised: [Equation]

Original:  : otherwise.
Revised: , i m = j m , 0 : otherwise.

Original: (3) to ( R d ) ⊗ m by linearity.
Revised: (3) to ( R d ) ⊗ m by linearity.

Original: This gives us an inner product on H ,(cid:104) v , w (cid:105) :
Revised: This gives us an inner product on H ,(cid:104) v , w (cid:105) :

Original: = (cid:88)m
Revised: = (cid:88)

Original: [Equation]
Revised: [Equation]

Original: (cid:104) v m , w m (cid:105) m such that H is a Hilbert space; in particular we get a norm (cid:107) v (cid:107) :
Revised: w m (cid:105) m such that H is a Hilbert space; in particular we get a norm (cid:107) v (cid:107) :

Original: = (cid:112) (cid:104) v , v (cid:105) .
Revised: = (cid:112) (cid:104) v , v (cid:105) .

Original: To sum up, the space
Revised: To sum up, the space

Original: H has a rich structure: it has a vector space structure, it has an algebra structure (a noncommutative product), and it is a Hilbert space (an inner product between elements of H gives a scalar).
Revised: H has a rich structure: it has a vector space structure, it has an algebra structure (a noncommutativeproduct), and it is a Hilbert space (an inner product between elements of H gives a scalar).

Original: Characterizing Random Walks.
Revised: Characterizing Random Walks.

Original: From Equation (1), we have constructed a map (cid:101) ϕ that maps asequence x ∈ Seq ( R d ) of arbitrary length into the space H (see Appendix C for further details).
Revised: We have constructed a map (cid:101) ϕ that maps a sequence x ∈ Seq ( R d )of arbitrary length into the space H .

Original: Ouraim is to apply this to the sequence of node attributes corresponding to random walks on a graph.
Revised: Our aim is to apply this to the sequence of node attributescorresponding to random walks on a graph.

Original: Therefore, the expectation of (cid:101) ϕ should be able to characterize the distribution of the random walk.
Revised: Therefore, the expectation of (cid:101) ϕ should be able tocharacterize the distribution of the random walk.

Original: Formally the map (cid:101) ϕ is characteristic if the map µ (cid:55)→ E x ∼ µ [ (cid:101) ϕ
Revised: Formally the map (cid:101) ϕ is characteristic if the mapµ (cid:55)→ E x ∼ µ [ (cid:101) ϕ

Original: ( x )] from the space of probabilitymeasures on Seq ( R d ) into H is injective.
Revised: ( x )] from the space of probability measures on Seq ( R d ) into H is injective.

Original: Indeed, if the chosen lifting ϕ satisﬁes some mild conditionsthis holds for (cid:101) ϕ ; see Appendix C and [16, 62].
Revised: Indeed, ifthe chosen lifting ϕ satisﬁes some mild conditions this holds for (cid:101) ϕ ; see Appendix C and [15, 56].

Original: Linear Functionals.
Revised: Linear Functionals.

Original: The quantity E x ∼ µ [ (cid:101) ϕ
Revised: The quantity E x ∼ µ [ (cid:101) ϕ

Original: ( x )] characterizes the probability measure µ but isvalued in the inﬁnite-dimensional Hilbert space H .
Revised: ( x )] characterizes the probability measure µ but isvalued in the inﬁnite-dimensional Hilbert space H .

Original: Using the inner product, we can instead consider(cid:104) (cid:96) , E x ∼ µ [ (cid:101) ϕ
Revised: Using the inner product, we can instead consider(cid:104) (cid:96) , E x ∼ µ [ (cid:101) ϕ

Original: ( x )] (cid:105) for (cid:96) =
Revised: ( x )] (cid:105) for (cid:96) =

Original: ( (cid:96) 0 , (cid:96) 1 , (cid:96) 2 , . . .
Revised: ( (cid:96) 0 , (cid:96) 1 , (cid:96) 2 , . . .

Original: , (cid:96) M , 0 , . . . )
Revised: , (cid:96) M , 0 , . . . )

Original: ∈ H and M ≥(4) which is equivalent to knowing E x ∼ µ [ (cid:101) ϕ
Revised: ∈ H and M ≥(4) which is equivalent to knowing E x ∼ µ [ (cid:101) ϕ

Original: ( x )] ; i.e. the set (4) characterizes µ .
Revised: ( x )] ; i.e. the set (4) characterizes µ .

Original: This is analogous to howone can use either the moment generating function of a real-valued random variable or its sequenceof moments to characterize its distribution; the former is one inﬁnite-dimensional object (a function),the latter is a inﬁnite sequence of scalars.
Revised: This is analogous to howone can use either the moment generating function of a real-valued random variable or its sequenceof moments to characterize its distribution; the former is one inﬁnite-dimensional object (a function),the latter is a inﬁnite sequence of scalars.

Original: We extend a key insight from [62] in Section 4: a linearfunctional (cid:104) (cid:96) , E x ∼ µ [ (cid:101) ϕ
Revised: We extend a key insight from [56] in Section 4: a linearfunctional (cid:104) (cid:96) , E x ∼ µ [ (cid:101) ϕ

Original: ( x )] (cid:105) can be efﬁciently approximated without directly computing E x ∼ µ [ (cid:101) ϕ
Revised: ( x )] (cid:105) can be efﬁciently approximated without directly computing E x ∼ µ [ (cid:101) ϕ

Original: ( x )]or storing large tensors.
Revised: ( x )]or storing large tensors.

Original: The Tensor Exponential.
Revised: The Tensor Exponential.

Original: While we will continue to keep ϕ arbitrary for our main results (see [62]and Appendix G for other choices), we will use the tensor exponential exp ⊗ : R d → H , deﬁned by [Equation]
Revised: While we will continue to keep ϕ arbitrary for our main results (see [56]and Appendix G for other choices), we will use the tensor exponential exp ⊗ : R d → H , deﬁned byexp ⊗ ( x ) = (cid:18) x ⊗ m m !

Original:  as the primary example throughout this paper and in the experiments in Section 5.
Revised: (cid:19) m ≥, as the primary example throughout this paper and in the experiments in Section 5.

Original: With this choice,the induced sequence feature map is the discretized version of a classical object in analysis, calledthe path signature, see Appendix C.
Revised: With this choice,the induced sequence feature map is the discretized version of a classical object in analysis, calledthe path signature, see Appendix C.

Original:  Hypo-Elliptic Diffusions
Revised:  Hypo-Elliptic Diffusions

Original: Throughout this section, we ﬁx a labelled graph G =
Revised: Throughout this section, we ﬁx a labelled graph G =

Original: ( V , E , f ) , that is V is a set of n nodes
Revised: ( V , E , f ) , that is V is a set of n nodes

Original: V = { 1 , . . . , n } , E denotes edges and f : V → R d is the set of continuous node attributes 3 whichmap each node to an element in the vector space R d .
Revised: V = { 1 , . . . , n } , E denotes edges and f : V → R d is the set of continuous node attributes 3 whichmap each node to an element in the vector space R d .

Original: Two nodes i, j ∈ V are adjacent if ( i,
Revised: Two nodes i, j ∈ V are adjacent if ( i,

Original: j ) ∈
Revised: j ) ∈

Original: E isan edge, and we denote this by i ∼ j .
Revised: E isan edge, and we denote this by i ∼ j .

Original: The adjacency matrix A of a graph is deﬁned byA i,j = 1 ,whenever i ∼ j , and 0 otherwise.
Revised: The adjacency matrix A of a graph is deﬁned by

Original:  We denote by deg( i )  the number nodes that are adjacent to node i .
Revised: [Equation] We use deg( i ) to denote the number nodes that are adjacent to node i .

Original: Random Walks on Graphs.
Revised: Random Walks on Graphs.

Original: Let ( B k ) k ≥ 0 be the simple random walk on the nodes V of G , wherethe initial node is chosen uniformly at random.
Revised: Let ( B k ) k ≥ 0 be the simple random walk on the nodes V of G , wherethe initial node is chosen uniformly at random.

Original: The transition matrix of this time-homogeneous
Revised: The transition matrix of this time-homogeneous

Original: Markov chain is
Revised: Markov chain is

Original: [Equation]: otherwise .
Revised: [Equation]

Original: Denote by ( L k ) k ≥ 

Revised: Denote by ( L k ) k ≥

Original: the random walk lifted to the node attributes in R d , that is
Revised: the random walk lifted to the node attributes in R d , that is

Original: L k := f ( B k ) .
Revised: [Equation]

Original: Recall that the normalized graph Laplacian for random walks is deﬁned as L
Revised: Recall that the normalized graph Laplacian for random walks is deﬁned as L

Original: =
Revised: =

Original: I − D − 1 A , where
Revised: I − D − 1 A , where

Original: D is diagonal degree matrix; in particular, the entry-wise deﬁnition is
Revised: D is diagonal degree matrix; in particular, the entry-wise deﬁnition is

Original: [Equation]
Revised: [Equation]

Original: The discrete graph diffusion equation for U k ∈ R n × d is given by
Revised: The discrete graph diffusion equation for U k ∈ R n × d is given by

Original: [Equation]
Revised: [Equation]

Original:  where the initial condition U 0 ∈ R n × d is speciﬁed by the node attributes.
Revised: (6) where the initial condition U 0 ∈ R n × d is speciﬁed by the node attributes.

Original: 4 The probabilisticinterpretation of the solution to this diffusion equation is classical and given as
Revised: 4 The probabilisticinterpretation of the solution to this diffusion equation is classical and given as

Original: [Equation]
Revised: [Equation]

Original: This allows us to compute the solution u k using the transition matrix P =
Revised: This allows us to compute the solution u k using the transition matrix P =

Original: I − L .
Revised: I − L .

Original: Random Walks on Algebras.
Revised: Random Walks on Algebras.

Original: We now incorporate the history of a random walker by consideringthe quantity
Revised: We now incorporate the history of a random walker by consideringthe quantity

Original: [Equation]
Revised: [Equation]

Original: where L k =
Revised: where L k =

Original: ( L 0 , . . .
Revised: ( L 0 , . . .

Original: , L k ) .
Revised: , L k ) .

Original: Since (cid:101) ϕ captures the whole history of the random walk L k over nodeattributes, we expect this expectation to provide a much richer summary of the neighborhood of nodei than E
Revised: Since (cid:101) ϕ captures the whole history of the random walk L k over nodeattributes, we expect this expectation to provide a much richer summary of the neighborhood of nodei than E

Original: [ L k | B 

Revised: [ L k | B

Original: = i ] .
Revised: = i ] .

Original: The price is however, the computational complexity, since (9) is H -valued.
Revised: The price is however, the computational complexity, since (8) is H -valued.

Original: Weﬁrst show, that analogous to (7), the quantity (9) satisﬁes a diffusion equation that can be computedwith linear algebra.
Revised: Weﬁrst show, that analogous to (6), the quantity (8) satisﬁes a diffusion equation that can be computedwith linear algebra.

Original: To do so, we develop a graph analogue of the hypo-elliptic Laplacian and replacethe scalar entries of the matrices with entries from the algebra H .
Revised: To do so, we develop a graph analogue of the hypo-elliptic Laplacian and replacethe scalar entries of the matrices with entries from the algebra H .

Original: Matrix Rings over Algebras.
Revised: Matrix Rings over Algebras.

Original: We ﬁrst revisit the adjacency matrix A ∈ R n × n and replace it bythe tensor adjacency matrix (cid:101)
Revised: We ﬁrst revisit the adjacency matrix A ∈ R n × n and replace it bythe tensor adjacency matrix (cid:101)

Original: A =
Revised: A =

Original: ( (cid:101) A )
Revised: ( (cid:101) A )

Original: i,j ∈ H n × n , that is (cid:101)
Revised: i,j ∈ H n × n , that is (cid:101)

Original: A is a matrix but instead of scalar entries itsentries are elements in the algebra H .
Revised: A is a matrix but instead of scalar entries itsentries are elements in the algebra H .

Original: The matrix A has an entry at i, j if nodes i and j are connected;(cid:101)
Revised: The matrix A has an entry at i, j if nodes i and j are connected;(cid:101)

Original: A replaces the i, j entry with an element of H that tells us how the node attributes of i and j differ,
Revised: A replaces the i, j entry with an element of H that tells us how the node attributes of i and j differ,(cid:101) A i,j := (cid:26) ϕ ( f ( [Equation]

Original: [Equation]
Revised: [Equation]

Original:  : otherwise .[Equation]
Revised: ∼ j : otherwise .

Original: Matrix multiplication works for elements of H n × n by replacing scalar multiplication by multiplicationin H , that is ( (cid:101) B · (cid:101) C ) i,j = (cid:80) nk
Revised: Matrix multiplication works for elements of H n × n by replacing scalar multiplication by multiplicationin H , that is ( (cid:101) B · (cid:101) C ) i,j = (cid:80) nk

Original: =1 (cid:101) B
Revised: =1 (cid:101) B

Original: i,k · (cid:101) C k,j for (cid:101) B, (cid:101) C ∈ H n × n and (cid:101) B i,k · (cid:101) C k,j denotes multiplicationin H as in Equation (2).
Revised: i,k · (cid:101) C k,j for (cid:101) B, (cid:101) C ∈ H n × n and (cid:101) B i,k · (cid:101) C k,j denotes multiplicationin H as in Equation (2).

Original: For the classical adjacency matrix A , the k -th power counts the number oflength k walks in the graph, so that ( A k ) i,j is the number of walks of length k from node i to node j .
Revised: For the classical adjacency matrix A , the k -th power counts the number oflength k walks in the graph, so that ( A k ) i,j is the number of walks of length k from node i to node j .

Original: We can take powers of (cid:101)
Revised: We can take powers of (cid:101)

Original: A in the same way as in the classical case, where
Revised: A in the same way as in the classical case, where

Original: [Equation]
Revised: [Equation]

Original: where the sum is taken over all length k walks x = ( f ( i ) , . . .
Revised: where the sum is taken over all length k walks x = ( f ( i ) , . . .

Original: f ( j )) from node i to node j (full detailsare provided in Appendix D).
Revised: f ( j )) from node i to node j (full detailsare provided in Appendix D).

Original: Since (cid:101) ϕ
Revised: Since (cid:101) ϕ

Original: ( x ) characterizes each walk x , the entry (cid:101)
Revised: ( x ) characterizes each walk x , the entry (cid:101)

Original: A ki,j can be interpretedas a summary of all walks which connect nodes i and j .
Revised: A ki,j can be interpretedas a summary of all walks which connect nodes i and j .

Original: Hypo-elliptic Graph Diffusion.
Revised: Hypo-elliptic Graph Diffusion.

Original: Similar to the tensor adjacency matrix, we deﬁne the hypo-ellipticgraph Laplacian as the n × n matrix
Revised: Similar to the tensor adjacency matrix, we deﬁne the hypo-ellipticgraph Laplacian as the n × n matrix

Original: [Equation]
Revised: [Equation]

Original: where D is the degree matrix embedded into H n × n at tensor degree 0 .
Revised: where D is the degree matrix embedded into H n × n at tensor degree 0 .

Original: The entry-wise deﬁnition is
Revised: The entry-wise deﬁnition is(cid:101) L i

Original: [Equation][Equation]
Revised: [Equation]

Original: : otherwise .
Revised: : otherwise .

Original: We can now formulate the hypo-elliptic graph diffusion equation for v k ∈ H n as
Revised: We can now formulate the hypo-elliptic graph diffusion equation for v k ∈ H n as

Original: [Equation][Equation]
Revised: [Equation]

Original: Analogous to the classic graph diffusion (8), the hypo-elliptic graph diffusion (12) has a probabilisticinterpretation in terms of L as shown in Theorem 1 (the proof is given in Appendix D).
Revised: Analogous to the classic graph diffusion (7), the hypo-elliptic graph diffusion (11) has a probabilisticinterpretation in terms of L as shown in Theorem 1 (the proof is given in Appendix D).

Original: Theorem 1.
Revised: Theorem 1.

Original: Let k ∈ N , L k =
Revised: Let k ∈ N , L k =

Original: ( L 0 , . . .
Revised: ( L 0 , . . .

Original: , L k ) be the lifted random walk from (6) , and (cid:101) P
Revised: , L k ) be the lifted random walk from (5) , and (cid:101) P

Original: = I − (cid:101)
Revised: = I − (cid:101)

Original: L bethe tensor adjacency matrix .
Revised: L bethe tensor transition matrix .

Original: The solution to the hypo-elliptic graph diffusion equation (12) is[Equation] Furthermore, if F ∈ H n × n is the diagonal matrix with F i,i = ϕ ( f ( i )) , then
Revised: The solution to the hypo-elliptic graph diffusion equation (11) is

Original: [Equation]
Revised: [Equation]

Original: In the classical diffusion equation, U k captures the concentration of the random walkers after k timesteps over the nodes.
Revised: In the classical diffusion equation, U k captures the concentration of the random walkers after k timesteps over the nodes.

Original: In the hypo-elliptic diffusion equation, v k captures summaries of randomwalk histories after k time steps over the nodes since (cid:101) ϕ
Revised: In the hypo-elliptic diffusion equation, v k captures summaries of randomwalk histories after k time steps over the nodes since (cid:101) ϕ

Original: ( L k ) summarizes the whole trajectory
Revised: ( L k ) summarizes the whole trajectory

Original: L k =
Revised: L k =

Original: ( L 0 , . . .
Revised: ( L 0 , . . .

Original: , L k ) and not only the endpoint L k .
Revised: , L k ) and not only the endpoint L k .

Original: Node Features and Graph Features.
Revised: Node Features and Graph Features.

Original: Theorem 1 can then be used to compute features Φ( i
Revised: Theorem 1 can then be used to compute features Φ( i

Original: ) ∈
Revised: ) ∈

Original: Hfor individual nodes as well as a feature Ψ( G ) for the entire graph.
Revised: Hfor individual nodes as well as a feature Ψ( G ) for the entire graph.

Original: The former is given by i -thcomponent v ( i ) k of the solution v k
Revised: The former is given by i -thcomponent v ( i ) k of the solution v k

Original: = ( v ( i ) k ) i =1 ,...,n ∈ H n of Equation (12),
Revised: = ( v ( i ) k ) i =1 ,...,n ∈ H n of Equation (11),

Original: [Equation]
Revised: [Equation]

Original: since the random walk B chooses the starting node B 

Revised: since the random walk B chooses the starting node B

Original: = i uniformly at random.
Revised: = i uniformly at random.

Original: The latter can becomputed by mean pooling the node features, which also has a probabilistic interpretation as
Revised: The latter can becomputed by mean pooling the node features, which also has a probabilistic interpretation as

Original: [Equation]
Revised: [Equation]

Original: ) where 1 TH := (1 H , . .
Revised: where 1 TH := (1 H , . .

Original: 1 H ) ∈ H n is the all-ones vector in H and 1 H denotes the unit in H .
Revised: 1 H ) ∈ H n is the all-ones vector in H and 1 H denotes the unit in H .

Original: Characterizing Graphs with Random Walks.
Revised: Characterizing Graphs with Random Walks.

Original: The graph and node features obtained through thehypo-elliptic diffusion equation are highly descriptive: they characterize the entire history of therandom walk process if one also includes the time parametrization, as described in Appendix C.
Revised: The graph and node features obtained through thehypo-elliptic diffusion equation are highly descriptive: they characterize the entire history of therandom walk process if one also includes the time parametrization, as described in Appendix C.

Original: Theorem 2.
Revised: Theorem 2.

Original: Suppose Ψ is the graph feature map from Equation (13) induced by the tensor exponential algebra lifting including time parametrization.
Revised: Suppose Ψ is the graph feature map from Equation (12) induced by the tensor exponential algebra lifting including time parametrization.

Original: Let G and G (cid:48) be two labelled graphs,and
Revised: Let G and G (cid:48) be two labelled graphs,and

Original: L k =
Revised: L k =

Original: ( L 0 , . . .
Revised: ( L 0 , . . .

Original: , L k ) and L (cid:48) k
Revised: , L k ) and L (cid:48) k

Original: = ( L (cid:48) 0 , . . .
Revised: = ( L (cid:48) 0 , . . .

Original: , L (cid:48) k ) be the k -step lifted random walk as deﬁned in
Revised: , L (cid:48) k ) be the k -step lifted random walk as deﬁned in

Original: Equation (6) .
Revised: Equation (5) .

Original: Then, Ψ( G ) =
Revised: Then, Ψ( G ) =

Original: Ψ( G (cid:48) )
Revised: Ψ( G (cid:48) )

Original: if and only if the distributions of L k and L (cid:48) k are equal.
Revised: if and only if the distributions of L k and L (cid:48) k are equal.

Original: It is instructive to contrast this result with the classical diffusion case; the latter only uses themarginal distribution of L k to capture the graph structure, which at least intuitively has much lessexpressive power. Indeed, in Appendix E, we show that for elementary graphs, this already leads tobig differences in expressive power. Further, an analogous result holds for the node features, and weprove both results in Appendix E. While we use the tensor exponential in this article, many otherchoices of (cid:101) ϕ are possible and result in graph and node features with such properties: under mildconditions, if the algebra lifting ϕ : R d → H characterizes measures on R d , the resulting node featuremap Φ characterizes the random walk, see [62], which in turn implies the above results.
Revised: An analogous result holds for the node features, and we prove both results in Appendix E. While weuse the tensor exponential in this article, many other choices of (cid:101) ϕ are possible and result in graphand node features with such properties: under mild conditions, if the algebra lifting ϕ : R d → Hcharacterizes measures on R d , the resulting node feature map Φ characterizes the random walk,see [56], which in turn implies the above results.

Original: Possiblevariations are discussed in Appendix G.
Revised: Possible variations are discussed in Appendix G.

Original: General (Hypo-elliptic) Diffusions and Attention.
Revised: General (Hypo-elliptic) Diffusions and Attention.

Original: One can consider more general diffusionoperators, such as the normalized Laplacian K of a weighted graph.
Revised: One can consider more general diffusionoperators, such as the normalized Laplacian K of a weighted graph.

Original: We deﬁne its lifted operator(cid:101) K ∈ H n × n analogous to Equation (11), resulting in a generalization of Theorem 1 with (cid:101) K replacing(cid:101)
Revised: We deﬁne its lifted operator(cid:101) K ∈ H n × n analogous to Equation (10), resulting in a generalization of Theorem 1 with (cid:101) K replacing(cid:101)

Original: L .
Revised: L .

Original: In the ﬂavour of convolutional GNNs [8], we consider a weighted adjacency matrix A ∈ R n × n
Revised: In the ﬂavour of convolutional GNNs [8], we consider a weighted adjacency matrix A ∈ R n × n

Original: A i,j = (cid:26) c i,j : i ∼ j : otherwise , for c
Revised: A i,j = (cid:26) c i,j : i ∼ j : otherwise , for c

Original: i,j > 0 .
Revised: i,j > 0 .

Original: The corresponding normalized Laplacian K is given by K
Revised: The corresponding normalized Laplacian K is given by K

Original: = I − D − 1 A , where D is adiagonal matrix with D i,i = (cid:80) j ∈N ( i ) c
Revised: = I − D − 1 A , where D is adiagonal matrix with D i,i = (cid:80) j ∈N ( i ) c

Original: i,j .
Revised: i,j .

Original: A common way to learn the coefﬁcients is by introducingparameter sharing across graphs by modelling them as c i,j = exp( a ( f ( i ) , f ( j ))) using a localattention mechanism, a : R d × R d → R [65].
Revised: A common way to learn the coefﬁcients is by introducingparameter sharing across graphs by modelling them as c i,j = exp( a ( f ( i ) , f ( j ))) using a localattention mechanism, a : R d × R d → R [59].

Original: In our implementation, we use additive attention [4]given by a ( f ( i ) , f ( j )) =
Revised: In our implementation, we use additive attention [4]given by a ( f ( i ) , f ( j )) =

Original: LeakyRelu 0 .
Revised: LeakyRelu 0 .

Original: 2 ( W s f ( i ) + W t f ( j )) , where W s , W t ∈ R 1 × d are lineartransformations for the source and target nodes, but different attention mechanisms can also be used;e.g.
Revised: 2 ( W s f ( i ) + W t f ( j )) , where W s , W t ∈ R 1 × d are lineartransformations for the source and target nodes, but different attention mechanisms can also be used;e.g.

Original: scaled dot-product attention [64].
Revised: scaled dot-product attention [58].

Original: Then, the corresponding transition matrix P
Revised: Then, the corresponding transition matrix P

Original: = D − 1 A isdeﬁned as P ij = softmax k ∈N ( i ) ( a ( f ( i ) , f ( k )))
Revised: = D − 1 A isdeﬁned as P ij = softmax k ∈N ( i ) ( a ( f ( i ) , f ( k )))

Original: j .
Revised: j .

Original: The lifted transition matrix is deﬁned as
Revised: The lifted transition matrix is deﬁned as(cid:101) P = (cid:26) P i,j ϕ ( f ( j ) − f ( i )) :

Original: [Equation] otherwise .
Revised: i ∼ j : otherwise .

Original: The statements of Theorem 1 immediately generalize to this variation by replacing the expectationwith respect to a non-uniform random walk.
Revised: The statements of Theorem 1 immediately generalize to this variation by replacing the expectationwith respect to a non-uniform random walk.

Original: Hence, in this case the use of attention can be interpretedas learning the transition probabilities of a random walk on the graph.
Revised: Hence, in this case the use of attention can be interpretedas learning the transition probabilities of a random walk on the graph.

Original:  Efﬁcient Algorithms for Deep Learning
Revised:  Efﬁcient Algorithms for Deep Learning

Original: The previous sections show that the node feature Φ( i ) provides a structured description of theneighborhood of node i and it is instructive to think of a linear functional (cid:104) (cid:96)  , Φ( i ) (cid:105) as answering aspeciﬁc question about the node neighbourhood, see Appendix E for examples.
Revised: We now focus on the computation of linear maps of Φ( i ) deﬁned as Q : H → R d (cid:48) , Φ( i ) (cid:55)→( (cid:10) (cid:96) j , Φ( i ) (cid:11) ) j =1 ,...,d (cid:48) for (cid:96) j

Original: The naive computationof (cid:104) (cid:96)  , Φ( i ) (cid:105) by ﬁrst computing Φ( i ) and taking the inner product is too expensive, especially when(cid:96) =
Revised: = ( (cid:96) j 0 , (cid:96) j 1 , . . .

Original: ( (cid:96) 0 , . . . , (cid:96) M , 0 , . . . )
Revised: , (cid:96) jM , 0 , 0 , . . . )

Original: ∈ Hfor large M .
Revised: ∈ H .

Original: To address this we revisit two observations from [62]:ﬁrst, for a rank-1 functional (cid:96) ∈ H , the computation of (cid:104) (cid:96) , Φ( i ) (cid:105) is computationally cheap. Second,restriction to small M limits the expressive power but can be counteracted bycomposition: any choiceof d different functionals (cid:96) 1 , . . .
Revised: These maps capture the distributionof the random walk L k conditioned to start at node i and pooling them captures the graph as seenby the random walk, see Section 3. These linear maps are parametrized by (cid:96) j which we view as learnable parameters of our model and hence, requires efﬁcient computation.

Original: , (cid:96) d ∈ H gives a label update f ( i ) (cid:55)→ ( (cid:104) (cid:96)j , Φ( i ) (cid:105)  ) j =1 ,...,d ∈ R d forthe graph. Repeating such a label update a few times with low-degree M and rank-1 functionals turnsout to be as powerful as computing one update for general functionals with arbitrary high M .
Revised: For any (cid:96) , we couldcompute (cid:104) (cid:96) , Φ( i ) (cid:105) by ﬁrst evaluating Φ( i ) using linear algebra by Theorem 1 and subsequently takingtheir inner product with (cid:96) .

Original: Theﬁrst observation should not be too surprising given the popularity of low rank approximations; thesecond observation is reminiscent to constructing a high-degree polynomial by composing low-degreepolynomials 5 or the width-vs-depth phenomenon in neural nets and we give more details below.
Revised: However, this involves computations with high-degree tensors, whichmakes this approach not scalable. In this section, we leverage ideas about tensor decompositionsto design algorithms to compute (cid:104) (cid:96) , Φ( i ) (cid:105) efﬁciently [56]; in particular, we do not need to computeΦ( i ) ∈ H directly or store large tensors.

Original: Computing  Rank- 1 Functionals.
Revised: Computing a Rank- 1 Functional.

Original: First, we focus on a rank- 1 linear functional (cid:96) ∈ H given as(cid:96) =
Revised: First, we focus on a rank- 1 linear functional (cid:96) ∈ H given as(cid:96) =

Original: ( (cid:96) m )
Revised: ( (cid:96) m )

Original: m ≥ 0 with (cid:96) m =
Revised: m ≥ 0 with (cid:96) m =

Original: u M − m +1 ⊗ · · · ⊗ u M and (cid:96) m = 0 for m [Equation] M, (14) where u m ∈ R d for m = 1 , . . .
Revised: u M − m +1 ⊗ · · · ⊗ u M and (cid:96) m = 0 for m > M, (13) where u m ∈ R d for m = 1 , . . .

Original: , M for a ﬁxed M ≥ 1 .
Revised: , M for a ﬁxed M ≥ 1 .

Original: Theorem 3 shows that for such (cid:96) , the computation of (cid:104) (cid:96) , ˆΦ( i ) (cid:105), where ˆΦ( i ) is the node feature without the basepoint, can be done (a) efﬁcientlyby factoring this low-rank structure into the recursive computation, and (b) simultaneously for allnodes i ∈ V in parallel.
Revised: Theorem 3 shows that for such (cid:96) , thecomputation of (cid:104) (cid:96) , Φ( i ) (cid:105) can be done (a) efﬁciently by factoring this low-rank structure into therecursive computation, and (b) simultaneously for all nodes i ∈ V in parallel.

Original: This can then be used to compute rank- R functionals for R > 1 ,and for(cid:104) (cid:96) , Φ( i ) (cid:105) ; see Appendix F, where we also provide a pseudocode implementation.
Revised: This can then be usedto compute rank- R functionals for R > 1 , see Appendix F.

Original: Theorem 3.
Revised: Theorem 3.

Original: Let (cid:96) be as in (14) and deﬁne f k,m ∈ R n for m = 1 , . . .
Revised: Let (cid:96) be as in (13) and deﬁne f k,m ∈ R n for m = 1 , . . .

Original: , M as
Revised: , M as

Original: [Equation]
Revised: [Equation]

Original: where 1 T :=
Revised: where 1 T :=

Original: . , 1) ∈ R n is the all-ones vector; and for 2 ≤ k and 1 ≤ m ≤ M recursively asf k,m:= P · f k − 1 ,m +
Revised: . , 1) ∈ R n is the all-ones vector; and for 2 ≤ k and 1 ≤ m ≤ M recursively asf k,m

Original: [Equation]
Revised: [Equation]

Original: (15) where the matrix C u 
Revised: = P · [Equation] [Equation] where the matrix C u =

Original: = ( C ui,j ) ∈ R n × n is deﬁned as[Equation]
Revised: ( C ui,j ) ∈ R n × n is deﬁned as

Original:  f[Equation]
Revised: C ui,j := (cid:26) (cid:104) u, f ( j ) − f

Original: − f ( i [Equation] (cid:105) :
Revised: ( i ) (cid:105) :

Original: i ∼ j, : otherwise .
Revised: i ∼ j, : otherwise .

Original: Here (cid:12) denotes element-wise 6 multiplication, while · denotes matrix multiplication.
Revised: and (cid:12) denotes element-wise 5 multiplication, while · denotes matrix multiplication.

Original: Then, it holdsfor i ∈ V , random walk length k ∈ Z
Revised: Then, it holds fori ∈ V , random walk length k ∈ Z

Original: + , and tensor degree m = 1 , . . ., M , thatf k,m ( i ) =
Revised: + , and tensor degree m = 1 , . . .

Original: (cid:104) (cid:96) m , ˆ Φ k ( i ) (cid:105) , where ˆ Φ k ( i ) =
Revised: , M , that

Original: E [ ϕ ( δ 1 L k ) · · · ϕ ( δ k L k ) | B 0 = i ] .
Revised: [Equation] Remark 1.

Original: Overall, Eq.
Revised: Overall, Eq.

Original: (15) computes f k,m ( i ) for all i ∈ V , k = 1 , . .
Revised: (14) computes f k,m ( i ) for all i ∈ V , k = 1 , . .

Original: , K , m = 1 , . . .
Revised: , K , m = 1 , . . .

Original: , M in O ( K · M 2 · N E + M · N E · d ) operations, where N E ∈ N denotes the number of edges; see App.F. In particular, onedoes not need to compute Φ( i
Revised: , M in O ( K · M 2 · N E + M · N E · d ) operations, where N E ∈ N denotes the number of edges; see App.

Original: ) ∈ H directly or store large tensors.
Revised: F.

Original: Graph Labelling Layers.
Revised: Building Neural Networks.

Original: Fixing d rank 1-functionals (cid:96) 1 , . . , (cid:96) d ∈ H induces a label updatef ( i ) (cid:55)→ ( (cid:104) (cid:96) i , Φ( i ) (cid:105) )
Revised: Given an algorithm to compute rank-1 functionals of Φ( i ) , we proceedin the following steps to build neural network models: (1) given a random walk length k ∈ N , maxtensor degree M ∈ N and max tensor rank R ∈ N , we introduce R rank-1 degree- M functionals ˜ (cid:96) j M ,and compute f jk,m ( i ) for all j = 1 , . . .

Original: i =1 ,...,d ∈ R d .
Revised: , R and m = 1 , . .

Original:  Theorem 3 allows us to compute this update in parallel for allnodes in V . Such a label update is similar to hidden layer in a NN and we can stack such updates,see Figure 3 in App. H.1.
Revised: , M using Theorem 3 that gives a linearmapping between H → R RM .

Original: As in NN, the d functionals in each "graph labelling layer" are optimizedby gradient descent. Finally, note that a rank R functional is the sum of R rank-1 functionals so wecan immediately carry out the same construction with rank- R functionals by adding a mixing layer.
Revised: (2) Apply a linear mixing layer R RM → R R to the rank-1 functionalsto recover at most rank- R functionals.

Original: To sum up, a graph labelling layer is determined by the random walk length k , the maximal tensordegree M , maximal tensor rank R and the functionals are then found by optimization.
Revised: Update the node labels, which now incorporate neighbourhoodinformation up to hop- k . (3) Iterate this construction by repeating steps (1) and (2). The drawback of using at most rank- R functionals is a loss of expressiveness: not all linear functionalsof tensors can be represented as rank- R functionals if R is not large enough.

Original: Using a single layer of low-rank functionalslimits the expressiveness but stacking layers allowsinpractice to approximate general, high-degree M functionals. Some theoretical results can be found in[62]; however, here we simply appeal to the analogy with NN where stacking simple transformationsprovides a ﬂexible functional class with good inductive bias.
Revised: However, we hypothesizethat iterations of low-rank functionals allows to approximate any general linear functional as in [56]. Further, this has a probabilistic interpretation, where instead of considering one long random walkwe use a random walk over random walks, increasing the effective random walk length additively.

Original:  Experiments
Revised:  Experiments

Original: We implemented the above approach and call the resulting model G raph 2T ens N etworks since itrepresents the neighbourhood of a node as a sequence of tensors, which is further pushed througha low-tensor-rank constrained linear mapping, similarly to how neural networks linearly transformtheir inputs pre-activation.
Revised: We implemented the above approach and call the resulting model G raph 2T ens N etworks since itrepresents the neighbourhood of a node as a sequence of tensors, which is further pushed througha low-tensor-rank constrained linear mapping, similarly to how neural networks linearly transformtheir inputs pre-activation.

Original: A conceptual difference is that in our case the non-linearity is appliedﬁrst and the projection secondly, albeit the computation is coupled between these steps.We providefurther experiments and ablation studies of our models in Appendix H.
Revised: A conceptual difference is that in our case the non-linearity is applied ﬁrstand the projection secondly, albeit the computation is coupled between these steps.

Original: Experimental Setup.
Revised: Experimental Setup.

Original: The aim of our main experiment is to test the following key properties ofour model: (1) ability to capture long-range interactions between nodes in a graph, (2) robustness topooling operations, hence making it less susceptible to the “over-squashing” phenomenon [2].
Revised: The aim of our main experiment is to test the following key properties ofour model: (1) ability to capture long-range interactions between nodes in a graph, (2) robustness topooling operations, hence making it less susceptible to the “over-squashing” phenomenon [2].

Original: Wedo this by following the experiments in [70].
Revised: Wedo this by following the experiments in [64].

Original: In particular, we show that our model is competitivewith previous approaches for retaining long-range context in graph-level learning tasks but withoutcomputing all pairwise interactions between nodes, thus keeping the inﬂuence distribution localized[73].
Revised: In particular, we show that our model is competitivewith previous approaches for retaining long-range context in graph-level learning tasks but withoutcomputing all pairwise interactions between nodes, thus keeping the inﬂuence distribution localized[67].

Original: We further give a detailed ablation study to show the robustness of our model to variousarchitectural choices in Appendix H.2.
Revised: We further give a detailed ablation study to show the robustness of our model to variousarchitectural choices, and compare different variations on the algorithm from App. G.

Original: As a second experiment, we follow the previous applicationsof diffusion approaches to graphs that have mostly considered inductive learning tasks, e.g. on thecitation datasets [13, 60, 11].
Revised: As a secondexperiment, we follow the previous applications of diffusion approaches to graphs that have mostlyconsidered inductive learning tasks, e.g. on the citation datasets [12, 54, 10].

Original: Our experimentation on these datasets are available in Appendix H.3,where the model performs on par with short-range GNN models, but does not seem to beneﬁt fromadded long-range information a-priori. However, when labels are dropped in a k -hop sanitized wayas in [50], the performance decrease is less pronounced.
Revised: Our experimentationon these datasets are available in Appendix H.3, where the model performs on par with short-range GNN models, but does not seem to beneﬁt from added long-range information.

Original: Datasets.
Revised: Datasets.

Original: We use two biological graph classiﬁcation datasets (NCI1 and NCI109), that containaround ∼ 4000 biochemical compounds represented as graphs with ∼ 30 nodes on average [67, 1].
Revised: We use two biological graph classiﬁcation datasets (NCI1 and NCI109), that containaround ∼ 4000 biochemical compounds represented as graphs with ∼ 30 nodes on average [61, 1].

Original: The task is to predict whether a compound contains anti-lung-cancer activity.
Revised: The task is to predict whether a compound contains anti-lung-cancer activity.

Original: The dataset is split ina ratio of 80% − 10% − 10% for training, validation and testing.
Revised: The dataset is split ina ratio of 80% − 10% − 10% for training, validation and testing.

Original: Previous work [2] has found that
Revised: Previous work [2] has found that

Original: GNNs that only summarize local structural information can be greatly outperformed by models thatare able to account for global contextual relationships through the use of fully-adjacent layers.
Revised: GNNs that only summarize local structural information can be greatly outperformed by models thatare able to account for global contextual relationships through the use of fully-adjacent layers.

Original: Thiswas further improved on by [70], where a local neighbourhood encoder consisting of a GNN stackwas upgraded with a Transformer submodule [64] for learning global interactions.
Revised: Thiswas further improved on by [64], where a local neighbourhood encoder consisting of a GNN stackwas upgraded with a Transformer submodule [58] for learning global interactions.

Original: Model Details.
Revised: Model Details.

Original: We build a GNN architecture primarily motivated by the GraphTrans (small) modelfrom [70], and only ﬁne-tune the pre- and postprocessing layers(s), random walk length, functionaldegree and optimization settings.
Revised: We build a GNN architecture primarily motivated by the GraphTrans (small) modelfrom [64], and only ﬁne-tune the pre- and postprocessing layers(s), random walk length, functionaldegree and optimization settings.

Original: In detail, a preprocessing MLP layer with 128 hidden units isfollowed by a stack of 4 G2TN layers each with RW length- 5 , max rank- 128 , max tensor degree- 2 ,all equipped with JK-connections [73] and a max aggregator.
Revised: In detail, a preprocessing MLP layer with 128 hidden units isfollowed by a stack of 4 G2TN layers each with RW length- 5 , max rank- 128 , max tensor degree- 2 ,all equipped with JK-connections [67] and a max aggregator.

Original: Afterwards, the node features arecombined into a graph-level representation using gated attention pooling [41].
Revised: Afterwards, the node features arecombined into a graph-level representation using gated attention pooling [39].

Original: The pooled features aretransformed using a ﬁnal MLP layer with 256 hidden units, and then fed into a softmax classiﬁcationlayer.
Revised: The pooled features aretransformed using a ﬁnal MLP layer with 256 hidden units, and then fed into a softmax classiﬁcationlayer.

Original: The pre- and postprocessing MLP layers employ skip-connections [30].
Revised: The pre- and postprocessing MLP layers employ skip-connections [28].

Original: Both MLP and G2TNlayers are followed by layer normalization [3], where GTN layers normalize their rank- 1 functionalsindependently across different tensor degrees, which corresponds to a particular realization of groupnormalization [69].
Revised: Both MLP and G2TNlayers are followed by layer normalization [3], where GTN layers normalize their rank- 1 functionalsindependently across different tensor degrees, which corresponds to a particular realization of groupnormalization [63].

Original: We randomly drop 10% of the features for all hidden layers during training [58].
Revised: We randomly drop 10% of the features for all hidden layers during training [52].

Original: The attentional variant, G2T(A)N also randomly drops 10% of its edges and uses 8 attention heads[65].
Revised: The attentional variant, G2T(A)N also randomly drops 10% of its edges and uses 8 attention heads[59].

Original: Training is performed by minimizing the categorical cross-entropy loss with an (cid:96) 2 regularizationpenalty of 10 − 4 .
Revised: Training is performed by minimizing the categorical cross-entropy loss with an (cid:96) 2 regularizationpenalty of 10 − 4 .

Original: For optimization, Adam [32] is used with a batch size of 128 and an inital learningrate of 10 − 3 that is decayed via a cosine annealing schedule [42] over 200 epochs.
Revised: For optimization, Adam [30] is used with a batch size of 128 and an inital learningrate of 10 − 3 that is decayed via a cosine annealing schedule [40] over 200 epochs.

Original: Further intuitionabout the model and architectural choices are available in Appendix H.1.
Revised: Further intuitionabout the model and architectural choices are available in Appendix H.1.

Original: Baselines.
Revised: Baselines.

Original: We compare against (1) the baseline models reported in [70], (2) variations of GraphTrans, (3) other recently proposed hierarchical approaches for long-range graph tasks [50]. Groups [Table]
Revised: We compare against (1) the baseline models reported in [64], (2) variations of GraphTrans, (3) other recently proposed hierarchical approaches for long-range graph tasks [47].

Original: of models in Table [-
-] are separated by dashed lines if they were reported in separate papers [Table] and theﬁrst citation after the name is where the result ﬁrst appeared.
Revised: Groupsof models in Table [-
-]1 are separated by dashed lines if they were reported in separate papers, and theﬁrst citation after the name is where the result ﬁrst appeared.

Original: The number of GNN layers in HGNetare not discussed by [50], and we report it as implied by their code.
Revised: The number of GNN layers in HGNetare not discussed by [47], and we report it as implied by their code.

Original: We organize the models intothree groups divided by solid lines: (a) baselines that only apply neighbourhood aggregations, andhierarchical or global pooling schemes, (b) baselines that ﬁrst employ a local neighbourhood encoder,and afterwards fully densify the graph in one way or another so that all nodes interact with each otherdirectly , (c) our models that we emphasize thematically belong to (a).
Revised: We organize the models intothree groups divided by solid lines: (a) baselines that only apply neighbourhood aggregations, andhierarchical or global pooling schemes, (b) baselines that ﬁrst employ a local neighbourhood encoder,and afterwards fully densify the graph in one way or another so that all nodes interact with each otherdirectly , (c) our models that we emphasize thematically belong to (a).

Original: Results.
Revised: Results.

Original: In Table 1, we report the mean and standard deviation of classiﬁcation accuracy computedover 10 different seeds.
Revised: In Table 1, we report the mean and standard deviation of classiﬁcation accuracy computedover 10 different seeds.

Original: Overall, both our models improve over all baselines in group (a) on bothdatasets, maximally by 1 . 9% on NCI1 and by 4 . 8% on NCI109.In group (b), G2T(A)N is solelyoutperformed by GraphTrans (large) on NCI1 by only 0 . Interestingly, the attention-free variation, G2TN , performs better on NCI109, where it performs very slightly worse than GraphTrans (small).
Revised: Overall, both our models improve over all baselines in group (a) on bothdatasets, maximally by 1 . 9% on NCI1 and by 4 . 8% on NCI109.

Original: Discussion.

Original: The previous experiments demonstrate that our approach performs very favourablyon long-range reasoning tasks compared to GNN-based alternatives without global pairwise nodeinteractions.

Original: Several of the works we compare against have focused on extending GNNs to largerneighbourhoods by speciﬁcally designed graph coarsening and pooling operations, and we emphasizetwo important points: (1) our approach can efﬁciently capture large neighbourhoods without any needfor coarsening, (2) it already performs well with simple mean-pooling as justiﬁed by Theorem 2 andexperimentally supported by the ablation studies in Appendix H.2.

Original: Although the Transformer-based

Original: GraphTrans slightly outperforms our model potentially due to its ability to learn global interactions,it is not entirely clear how much of the global graph structure it is able to infer from interactions ofshort-range neighbourhood summaries.

Original: Finally, Transformer models can be bottlenecked by theirquadratic complexity in nodes, while our approach only scales with edges, and hence, it can be morefavourable for large sparse graphs in terms of computations.

Original: Conclusion

Original: Inspired by classical results from analysis [23], we introduce the hypo-elliptic graph Laplacian.

Original: Thisyields a diffusion equation and also generalizes its classical probabilistic interpretation via randomwalks but now taking history into account.

Original: In addition to several attractive theoretical guarantees, weprovide scalable algorithms.

Original: Our experiments show that this can lead to largely improved baselinesfor long-range reasoning tasks.

Original: A promising future research theme is to develop improvements forthe classical Laplacian in this hypo-elliptic context; including lazy random walks [71]; nonlineardiffusions [12]; and source/sink terms [60].

Original: Another theme could be to extend the geometricstudy [61] of over-squashing to this hypo-elliptic point of view which is naturally tied to subRiemannian geometry [59].

Original: A limitation in our theoretical results is that for the iterations of low-rankapproximations only partial results exist and expanding this is an interesting (algebra-heavy) topic.

Original: Broader Impacts

Original: This article introduce a method which can improve the performance on graph neural networks.

Original: Wedo not believe there are any foreseeable negative societal impacts beyond generic points that apply toany graph learning method.

Original: Appendix Outline

Original: Section A summarizes the notation and objects used in this paper.

Original: Section B gives general backgroundon tensor and the algebra H .

Original: Section C discusses the feature map (cid:101) ϕ for sequences and possiblechoices for the lift ϕ from labels to the algebra H .

Original: Section D contains the proofs of our maintheorems and some variations of hypoelliptic diffusion.

Original: Section E provides the formal statementof Theorem 2 and the extension to pooled features.

Original: Section F gives background and details ofthe low-rank algorithm.

Original: Section G discusses variations of the sequence feature map which lead todifferent node and graph features.

Original: Section H includes further experiments and discussion on theempirical results.

Original: A

Original: Notation

Original: Symbol

Original: Meaning

Original: Fixed Parameters and Indices d dimension of node attributes k length of random walk n number of nodes in graph

Original: Sequence Features

Original: R d ﬁnite dimensional vector space for node attributes

Original: Seq ( R d ) sequences x = ( x 0 , . . .

Original: , x k ) of arbitrary length k in R d δ k x increments of a sequence where δ 0 x := x 0 and δ k x := x

Original: k − x k − 1 for k >

Original: H tensor algebra of R d (see Appendix B) ϕ algebra lifting ϕ : R d → H exp ⊗ tensor exponential exp ⊗ : R d → H (main example of algebra lifting) (cid:101) ϕ sequence feature map (cid:101) ϕ :

Original: Seq ( R d ) → H , where (cid:101) ϕ

Original: ( x ) =

Original: ϕ ( δ 0 x ) · · · ϕ ( δ k x )

Original: Graphs,

Original: Adjacency and Laplacian Matrices

Original: G G = ( V , E , f ) graph with vertex set, edge set, and node attributes f : V → R d ( B k ) k ≥simple random walk on graph (valued in V ) ( L k ) k ≥lifted random walk over R d , L k :=
Revised: In group (b), G2T(A)N is solelyoutperformed by GraphTrans (large) on NCI1 by only 0 .

Original: [Table]

Original: lifted length k random walk

Original: [Table]

Original: [Table]

Original: [Table]

Original: Φ( i ) =

Original: E [ (cid:101) ϕ

Original: ( L 1 , . . .

Original: , L n ) | L 0 =

Original: i ] ∈ H feature map for node i ∈ V

Original: [Table]

Original: Ψ( G ) =

Original: E [ (cid:101) ϕ

Original: ( L 1 , . . .

Original: , L n )]

Original: ∈ H feature map for graphs

Original: A standard adjacency matrix (cid:101)

Original: A tensor adjacency matrix

Original: P standard transition matrix (cid:101) P tensor transition matrix

Original: L normalized graph Laplacian (cid:101)

Original: [Table]

Original: normalized

Original: [Table]

Original: Laplacian

Original: Notation Conventions

Original: • Bold symbols are used for tensors and sequences; vectors are unbolded, such as x ∈ R d• Coordinates for vectors are denoted using superscripts:

Original: x = ( x (1) , . . .

Original: , x ( d ) ) ∈ R d .•

Original: Tensors are denoted by v = ( v 0 , v 1 , v 2 , . . . )

Original: ∈ H , where v m ∈ ( R d ) ⊗ m .•

Original: Sequences are denoted by x = ( x 0 , . . .

Original: , x k ) ∈

Original: Seq ( R d ) .

Original: B Tensors and the Algebra H

Original: In this section, we provide a brief overview of tensors on a ﬁnite-dimensional vector space R d , alongwith the resulting algebra H .

Original: Tensors on R d .

Original: While tensor products between vector spaces are deﬁned more generally, our mainfocus is on deﬁning the tensor powers of R d , denoted by ( R d ) ⊗ m for some m ∈ N .

Original: The tensor power( R d ) ⊗ m is also a vector space.

Original: Given a basis e 1 , . . .

Original: , e d of R d , we can deﬁne a basis of ( R d ) ⊗ m asthe collection of alle I :=

Original: e i 1 ⊗ . . .

Original: ⊗ e

Original: i m over all multi-indices ( i 1 , . .

Original: , i m ) , where each i j ∈ [ d ] .

Original: Thus, any element v m ∈ ( R d ) ⊗ m can berepresented asv m =

Original: d (cid:88) i 1 ,...,i m = v ( i 1 ,...,i m )

Original: m

Original: e ( i 1 ,...,i m ) , where the v ( i 1 ,...,i m ) m ∈ R speciﬁes the coordinates of v m .

Original: In particular, ( R d ) ⊗ 1 is simply R d itself;( R d ) ⊗ 2 can be viewed as the space of d × d matrices; ( R d ) ⊗ 3 is the space of d × d × d arrays, etc.

Original: Given two vectors x, y ∈ R d which we represent coordinate-wise as

Original: [Equation]

Original: , x ( d ) ) , y =

Original: ( y (1) , . . .

Original: , y ( d ) ) , the tensor product x ⊗ y ∈ ( R d ) ⊗ 2 is given by

Original: [Equation]

Original: More generally, if u m ∈ ( R d ) ⊗ m and v n ∈ ( R d ) ⊗ n , the tensor product u m ⊗ v n ∈ ( R d ) ⊗ ( m + n ) isdeﬁned coordinate-wise by

Original: [Equation]

Original: Furthermore, we note that ( R d ) ⊗ m inherits an inner product from R d through a choice of basis, as isshown in Equation (3).

Original: Free Algebras.

Original: Our goal is to ﬁnd a vector space H that contains the vector space R d but is largeenough to support a richer algebraic structure; in particular, a multiplication.

Original: Formally, this meanswe look for an injective map R d (cid:44) → H into an algebra H .

Original: A classic mathematical construction thatturns a vector space into an algebra is the free associative algebra , deﬁned as∞ (cid:77)

Original: [Equation]

Original: where addition and multiplication of two elements u = ( u 0 , u 1 , . . . ) and v = ( v 0 , v 1 , . . . ) is givenby deﬁning the ( R d ) ⊗ m coordinate to be

Original: [Equation]

Original: We emphasize that the multiplication is not commutative.

Original: A Universal Property.

Original: [-Indeed, this construction is the most general way to turn R d into an algebraas the following classical result shows: if we denote with ι : R d (cid:44) → (cid:76) ∞ m =
-]
Revised: [-Indeed, this construction is the most general way to turn R d into an algebraas the following classical result shows: if we denote with ι : R d (cid:44) → (cid:76) ∞ m =
-]

Original: ( R d ) ⊗ m the embeddingι

Original: ( x ) = (0 , x, 0 , . . . )

Original: then any linear map from R d into any associative algebra

Original: A factors throughι .

Original: In other words, given an associative algebra A and any linear map h : R d → A , there exists ahomomorphism of algebras ˜

Original: h : (cid:76) ∞ m

Original: [-=
-]
Revised: [-=
-]

Original: ( R d ) ⊗ m → H such the following diagram commutes

Original: [Equation]

Original: [-In a sense, this shows that (cid:76) ∞ m =
-]
Revised: [-In a sense, this shows that (cid:76) ∞ m =
-]

Original: ( R d ) ⊗ m is the “most general algebra” that R d embeds into.

Original: An Inner Product.

Original: While the free associative algebra satisﬁes the above universal property, it isconvenient to work in the slightly larger space

Original: [Equation]

Original: and deﬁne an inner product

Original: [Equation]

Original: [-for elements u , v ∈ (cid:81) m ≥ 0 ( R d ) ⊗ m for which the sum (16) converges (it does not converge forgeneral elements of (cid:81) m ≥ 
-]
Revised: [-for elements u , v ∈ (cid:81) m ≥ 0 ( R d ) ⊗ m for which the sum (16) converges (it does not converge forgeneral elements of (cid:81) m ≥ 
-]

Original: ( R d ) ⊗ m ).

Original: The largest space for which this inner product exists, is

Original: H :

Original: = { v ∈ (cid:89) m ≥( R d ) ⊗ m : (cid:107) v (cid:107) < ∞} and (cid:107) v (cid:107) :=

Original: (cid:112) (cid:104) v , v (cid:105) .

Original: The space H contains all the properties we require; it contains R d as a subspace, ι ( R d ) ⊂ H ; thenon-commutative multiplication structure, u · v ∈ H for u , v ∈ H ; and a Hilbert space structure,(cid:104) u , v

Original: (cid:105) ∈

Original: R for u , v ∈ H .

Original: Linear Functionals.

Original: Any (cid:96) ∈

Original: H can be treated as a linear functional by taking (cid:104) (cid:96) , ·(cid:105) :

Original: H → R , wewill primarily consider ﬁnite linear functionals

Original: [Equation]

Original: which have only ﬁnitely many nonzero coordinates.

Original: Such ﬁnite linear functionals applied to ourfeature map (cid:101) ϕ

Original: ( x ) ∈ H for sequences x ∈ Seq ( R d ) , are rich enough so that

Original: [Equation]

Original: can approximate any functions f ( x ) of sequences x ∈ Seq ( R d ) , and their collection characterizesthe distribution of random sequences (i.e. random walks),

Original: [Equation]

Original: (cid:96) , E x ∼

Original: [Equation]

Original: x

Original: [Equation]

Original: (cid:96)

Original: [Equation]

Original: , (cid:96) M , 0 , . . . )

Original: ∈ H, M ≥ 1 } is injective; see Theorem 4.

Original: C The Sequence Feature Map

Original: In Section 2 we introduce the sequence feature map

Original: [Equation]

Original: for sequences in R d where

Original: [Equation]

Original: is an injective map from R d into the algebra H .

Original: Here, δ 0 x := x 0 and δ i x := x i − x i − 1 for i ≥ 1denote the increments of a sequence x = ( x 0 , . . .

Original: , x k ) .

Original: Our main example is the tensor exponential

Original: [Equation]

Original: The algebra H is the free algebra discussed in detail in Appendix B.

Original: In this case, a direct calculationshows that

Original: [Equation]

Original: where the sum is taken over i 1 ≤ · · · ≤ i m with i 1 , . .

Original: , i m ∈ { 0 , . . .

Original: , k } and the coefﬁcientsc ( i 1 , . .

Original: , i m ) ∈ R can be computed in explicit form, see [62].

Original: Universality and Characteristicness Universality and characteristicness follow for our “discretetime/sequence” signatures (cid:101) ϕ ( x ) ∈ H from elementary arguments that we discuss below.

Original: In oursetting it is convenient to include the sequence coordinate in the lifting, that is we consider sequences

Original: [Equation]

Original: [Equation] , where ¯ [Equation] Theorem 4.
Revised: Interestingly, the attention-free variation,

Original: The time-parametrized sequence feature map

Original: [Equation] (18) has the following properties: Universality : for any continuous 7 f : Seq ( R d ) → R , any (cid:15) > 0 , and any compact set of sequences K ⊂ Seq ( R d ) , there exists a (cid:96) =
Revised: G2TN , performs better on NCI109, where it performs very slightly worse than GraphTrans (small).

Original: ( (cid:96) 0 , (cid:96) 1 , . . .

Original: , (cid:96) m , 0 , 0 , . . . )

Original: such that

Original: [Equation]

Original: Characteristicness : let Prob ( K ) be the set of probability measures that are supported on a

Original: compact subset K ⊂ Seq ( R d ) .

Original: Then the map

Original: [Equation] is injective. Proof.
Revised: Ablation Study.

Original: This is a folk theorem in control and probability theory: to see universality, it is sufﬁcient toverify that {(cid:104) (cid:96) , (cid:101) ϕ

Original: (¯ x ) (cid:105) : (cid:96) =

Original: ( (cid:96) 0 , . . .

Original: , (cid:96) m , 0 , . . . )

Original: , m ≥ 0 } is a point-separating algebra for C ( K, R )since then the result follows from the Stone-Weierstrass theorem.

Original: Point separation follows from [22,

Original: Corollary 4.9], and a direct calculation shows the product of (cid:104) (cid:96) , (cid:101) ϕ

Original: (¯ x ) (cid:105) and (cid:104) (cid:96) (cid:48) , (cid:101) ϕ

Original: (¯ x ) (cid:105) is again a linearfunctional of (cid:101) ϕ

Original: ( ¯ x ) .

Original: This shows that the collection of functionals forms an algebra.

Original: The characteristicness follows since Prob ( K ) is contained in the dual space of the space of continuousfunctions of sequences C ( K, R ) and by universality, linear functionals of (cid:101) ϕ

Original: (¯ x ) are dense in thisspace, so the result follows.

Original: This result shows that:

Original: • non-linear functions of sequences can be approximated using linear functionals in H ; and• the distribution of random

Original: sequences is characterized as the mean of our feature map (cid:101) ϕ is a universal and characteristic feature map for the set Seq ( V ) of sequences For more background and extensions to non-compact sets of sequences or paths [34, 16] and [
Revised: Further to using attention, we give a list of ideas for variations on our models in

Original: Section a more geometric picture

Original: Path Signatures.

Original: We now explain the remark made at the end of Section 2, that for the choice ofϕ as the tensor exponential (5), the resulting sequence feature map (cid:101) ϕ can be identiﬁed as the timediscretization of a classical object in analysis, called the path signature.

Original: First, we lift a sequencex =

Original: ( x 0 , . . .

Original: , x k ) from discrete time to continuous time by identifying it as the piecewise linear path

Original: X =

Original: ( X ( t ))

Original: t ≥ 0 ,

Original: X ( t ) :

Original: = x i + ( t − i )( x i +

Original: [Equation]

Original: x i ) for t ∈ [ i, i + 1) .

Original: Then a direct calculation shows that

Original: [Equation]

Original: where (cid:90)

Original: i

Original: [Equation]

Original: and i ∈ [0 , k ] ; note that ˙

Original: X

Original: ( t ) is well-deﬁned for almost every t ∈ [0 , k ] since X is piecewise linear,which is sufﬁcient to make sense of this integral as a Riemann-Lebesgue integral (and stochasticintegrals allow to treat rougher paths).

Original: Such sequences of iterated integrals are classical in analysis,probability theory, and control theory, and are known under various names (Path-ordered Exponential,

Original: Volterra series, Chen–Fliess Series, Chronological Exponential, etc.)

Original: ; we refer to them as pathsignatures as they are known in probability theory.

Original: Diffusions and their Generator.

Original: The path signature can even can be deﬁned for highly irregularpaths such as Brownian motion by using stochastic (Ito-Stratonovich) integrals.

Original: This is the connectionto our main motivation: if X =

Original: ( X t ) t ≥ 0 is a Brownian motion in R d , then its generator is the classical

Original: Laplacian and the diffusion of Brownian particles is captured with the classical diffusion PDE, theheat equation.

Original: Gaveau [23] showed that if we lift Brownian trajectories t (cid:55)→ X ( t ) evolving in thestate space R d into paths evolving in a richer state space H via the above signature construction, [Equation] then the stochastic process (19) is again a Markov process 8 and its generator satisﬁes a property calledhypo-ellipticity (see the paragraph below). Our approach can be seen as the analogous constructionon a graph and in discrete time: in the same way that (19) lifts Brownian motion from R d to a processevolving in the state space H , the map (cid:101) ϕ lifts a simple random walk ( B k ) k ≥ 0 on the graph to arandom walk L
Revised: Appendix G, which can be summarized brieﬂy as: (i) using increments of node attributes ( Diff ),(ii) preprending a zero point to sequences ( ZeroStart ), (iii) optimizing over the algebra embedding( AlgOpt ), all of which are built into our main models.

Original: =

Original: ( L k ) k ≥ 0 in H .

Original: Hypo-elliptic Operators, Sub-elliptic Operators, and their Geometry.

Original: A full discussion is beyond the scope of this article and several books [31] have been written about this topic.

Original: Forthe interested reader, we give a very short informal picture on the space R d which motivatesour nomenclature

Original: but otherwise this paragraph can be safely skipped.

Original: A differential operator

Original: L = (cid:80) σ i,i ∂ 2 ∂x i ∂x j

Original: + (cid:80) i µ i

Original: ∂∂x j is called elliptic if the matrix ( σ i,j ( x ))

Original: i,j is invertible for all x ∈ R d .

Original: Every (time-homogenous) Markov process X =

Original: ( X t ) t ≥ 0 gives rise to a differential operator

Original: [Equation]

Original: t but generically L is not elliptic.

Original: An important example class of Markov processes are stochasticdifferential equations,

Original: [Equation]

Original: (20) where B t =

Original: ( B 1 t , . . .

Original: , B et ) denotes a Brownian motion in R e and V 1 , . .

Original: , V e are vector ﬁelds on R d .

Original: By identifying the vectors ﬁelds V 1 , . .

Original: , V e as differential operators, the generator of (20) can bewritten as

Original: [Equation]

Original: which is in general not elliptic.

Original: Sub- and hypo-ellipticity are properties of L that are weaker thanellipticty: ellipticity implies sub-ellipticity and, by a classic theorem of Hörmander, sub-ellipticityimplies hypo-ellipticity.

Original: Hypo-ellipticy is in turn general enough to cover many important examplesfor which (sub-)ellipticity fails.

Original: For example, another celebrated result of Hörmander is that the SDEgenerator (21) is hypo-elliptic, whenever the Lie algebra generated by the vector ﬁelds and theirbrackets, spans at every point the full space R d .

Original: This not only allows us to study the properties ofa large class of diffusions but also provides a natural link with geometry: the set of vector ﬁelds

Original: V 1 , . .

Original: . , V e determines a subset of R d that the SDE evolves on.

Original: For general hypo-elliptic operators,this set is not all of R d and it is not even a Riemannian manifold; rather, it is a sub-Riemannianmanifold.

Original: Intuitively, sub-Riemannian geometry is "rougher" than Riemannian geometry (e.g. nocanonical connections exist) but still regular enough so that one can use geometric tools to study theunderlying stochastic process and vice versa; see for example [59, 23].

Original: This includes SDEs such asthose evolving in free objects: for a natural choice of vector ﬁelds V 1 , . .

Original: , V e the solution of the SDE(20) is (19), see [23].

Original: D Further Details on Hypo-elliptic Graph Diffusion

Original: In Section 3, we introduced tensor analogues of classical matrix operators, which were then used todeﬁne hypo-elliptic graph diffusion.

Original: These tensor-valued matrices allow us to efﬁciently representrandom walk histories on a graph, which can be manipulated using matrix operations.

Original: In this section,we discuss further details on the tensor adjacency matrix, provide a proof of Theorem 1, and introducea variation of hypo-elliptic graph diffusion.

Original: As in Section 3, we ﬁx a labelled graph G =

Original: ( V , E , f ) , where the nodes are denoted by integers,

Original: V = { 1 , . . . , n } , and f : V → R d denotes the continuous node attributes.

Original: Powers of the Tensor Adjacency Matrix.

Original: Recall that the powers of the classical adjacency matrix

Original: A counts the number of walks between two nodes on the graph.

Original: In particular, given k ∈ N , and twonodes i, j ∈ V , the result follows as a consequence of the sparsity pattern of A ,

Original: [Equation]

Original: i

Original: [Equation]

Original: Note that the product

Original: A

Original: i,i 1 · · · A i k − 1 ,j = 0 unless each pair of consecutive indices are adjacent in thegraph, namely i q − 1 ∼ i q for all q = 1 , . . .

Original: , k .

Original: Applying the same procedure to the tensor adjacencymatrix from Equation (10), we obtain a summary of all walks between two nodes, rather than just thenumber of walks.

Original: In particular,

Original: [Equation]

Original: where x = ( f ( i 0 ) , . . .

Original: , f ( i k )) denotes the lifted sequence in the vector space R d .

Original: Note that thiscorresponds to the sequence feature map without the initial point δ 0 x .

Original: Powers of the Tensor Transition Matrix.

Original: We now consider powers of the classical transitionmatrix P =

Original: I − L , where L = I − D − 1 A is the normalized graph Laplacian.

Original: The entries of P kprovide length k random walk probabilities; in particular, we have

Original: [Equation]

Original: The powers of the tensor transition matrix (cid:101) P

Original: = I − (cid:101) L , where (cid:101)

Original: L

Original: = I − D − 1 (cid:101)

Original: A is the hypo-elliptic

Original: Laplacian, will be the conditional expectation of the sequence feature map of the random walk process.

Original: In particular,

Original: [Equation]

Original: Proof of Hypo-elliptic Diffusion Theorem.

Original: Recall from Section 3 that the hypo-elliptic graphdiffusion equation is deﬁned by

Original: [Equation]

Original: where 1 T H =

Original: (1 H , . .

Original: 1 H ) ∈ H n is the all-ones vector in H .

Original: Using the above computations forpowers of the tensor transition matrix, we can prove Theorem 1, which is restated here.

Original: Theorem 1.

Original: Let k ∈ N , L k =

Original: ( L 0 , . . .

Original: , L k ) be the lifted random walk from (6) , and (cid:101) P

Original: = I − (cid:101)

Original: L bethe tensor adjacency matrix .

Original: The solution to the hypo-elliptic graph diffusion equation (12) is

Original: [Equation]

Original: Furthermore, if F ∈ H n × n is the diagonal matrix with F i,i = ϕ

Original: ( f ( i )) , then

Original: [Equation]

Original: Proof of Theorem 1.

Original: We begin by proving the ﬁrst equation.

Original: From the deﬁnition of hypo-elliptic diffusion, it is straightforward to see that

Original: [Equation]

Original: We prove coordinate-wise that v k =

Original: ( E [ ϕ ( δ 1 L k ) · · · ϕ ( δ k L k ) | B 0 = i ]) ni =1 .

Original: Indeed, using the aboveand Equation (22), the i -th coordinate of v k is

Original: [Equation]

Original: where x = ( f ( i 0 ) , . . .

Original: , f ( i k )) is the lifted sequence corresponding to a walk i 0 ∼ . . .

Original: ∼ i k on thegraph.

Original: Next, we will also prove the second equation coordinate-wise. Using the above result, we have [Equation] where we use the fact that δ 0 L k
Revised: Further, the previous architectural choices

Original: = L 0 = f

Original: [-( i ) when we condition B 
-]
Revised: [-( i ) when we condition B 
-]

Original: = i .

Original: Forward Hypo-elliptic Diffusion.

Original: In the classical setting, we can consider both the forward andbackward Kolmogorov equations.

Original: Throughout the main text and in the appendix so far, we havebeen considering the backward variants.

Original: In this section, we formulate the forward analogue ofhypo-elliptic diffusion.

Original: In the classical graph setting, this corresponds to the following forwardequation for U k ∈ R n × d given by

Original: [Equation]

Original: where the initial condition U 0 ∈ R n × d is speciﬁed by the node attributes.

Original: Note that because

Original: P = D − 1 A is right stochastic 9 , this variation of the graph diffusion equation conserves mass in eachcoordinate of the node attributes at every time step.

Original: Similarly we can formulate the forward hypo-elliptic graph diffusion equation for v k ∈ H n as

Original: [Equation]

Original: The solution of the Equation 23 is given below.

Original: Theorem 5.

Original: Let k ∈ N , L k =

Original: ( L 0 , . . .

Original: , L k ) be the lifted random walk from (6) , and (cid:101) P

Original: = I − (cid:101)

Original: L bethe tensor adjacency matrix .

Original: The solution to the forward hypo-elliptic graph diffusion equation (23) is

Original: [Equation]

Original: Furthermore, if F ∈ H n × n is the diagonal matrix with F i,i = ϕ

Original: ( f ( i )) , then

Original: [Equation] Proof.
Revised: [Table]

Original: The proof proceeds in the same way as the backward equation.

Original: By deﬁnition of the forwardhypo-elliptic diffusion, we have

Original: [Equation]

Original: Now, recall that the initial point of the random walk process is chosen uniformly over all nodes;in other words, P [ B 0 = i ] = 1 n .

Original: Then, we show v T k

Original: =

Original: ( E [ ϕ ( δ 1 L k ) · · · ϕ ( δ k L k ) | B k

Original: = i ]) n i =

Original: 1coordinate-wise as

Original: [Equation]

Original: where x = ( f ( i 0 ) , . . .

Original: , f ( i k )) is the lifted sequence corresponding to a walk i 0 ∼ . . .

Original: ∼ i k on thegraph.

Original: We will now prove the second equation.

Original: Note that we have

Original: [Equation]

Original: Then, following the same reasoning as the ﬁrst equation, we obtain the desired result.

Original: Weighted Graphs.

Original: In the prior discussion, we considered simple random walks in which the walkchooses one of the nodes adjacent to the current node uniformly at random.

Original: We can instead considermore general random walks on graphs, which can be described using a weighted adjacency matrix,

Original: [Equation]

Original: In this case, we deﬁne the diagonal weighted degree matrix to be D i,i = (cid:80) i ∼ j A

Original: i,j .

Original: [-We now deﬁnea weighted random walk ( B k ) k ≥ 
-]
Revised: [-We now deﬁnea weighted random walk ( B k ) k ≥ 
-]

Original: on the vertices V where the transition matrix is given by

Original: [Equation]

Original: With these weighted graphs, the powers of the adjacency and transition matrix can be interpreted ina similar manner as the standard case.

Original: Powers of the adjacency matrix provide total weights overwalks, while the powers of the transition matrix provide the probability of a weighted walk goingbetween two nodes after a speciﬁed number of steps.

Original: In particular,

Original: [Equation]

Original: The tensor adjacency and tensor transition matrices are deﬁned in the same manner as(cid:101)

Original: A

Original: [Equation]

Original: ∼ j : otherwise (cid:101) P i,j :=

Original: D − 1 (cid:101)

Original: A = (cid:26) P i,j ϕ ( f ( j ) − f ( i )) : i ∼ j : otherwise

Original: Note that powers of this weighted tensor transition matrix are exactly the same as the unweighted casefrom Equation (22), and thus the weighted version of both Theorem 2 and Theorem 5 immediatelyfollow.

Original: E Characterizing Random Walks

Original: In this appendix, we will provide further details on how the features obtained via hypo-ellipticdiffusion are able to characterize the underlying random walk processes.

Original: These results rely on thecharacteristic property of the time-parametrized sequence feature map from Theorem 4.

Original: Computation with Time Parametrization.

Original: Recall that the time-parametrized sequence featuremap from Equation (18) applies the algebra lifting to the time-parametrized sequence ¯ x :=(¯ x 0 , . .

Original: , ¯ x k ) , where ¯ x i =

Original: ( i, x i ) .

Original: Note that we have δ i ¯ x = (1 , x i − x i − 1 ) .

Original: Thus, the hypo-ellipticdiffusion equations and the low-rank algorithm given in Theorem 3 easily extends to this setting.

Original: Characterizing Random Walks.

Original: Recall that hypo-elliptic diffusion yields a feature map for labelled graphs by mean-pooling the individual node features as

Original: [Equation]

Original: where L k =

Original: ( L 0 , . . .

Original: , L k ) is the lifted random walk process in R d .

Original: We will now prove Theorem 2,which we restate here with more details.

Original: Theorem 2.

Original: Let

Original: [Equation]

Original: where ¯ x appends the time parametrization to the sequence x as in Equation (17) .

Original: Furthermore,suppose we have the resulting graph feature map

Original: [Equation]

Original: Let G and G (cid:48) be two labelled graphs, and

Original: L k =

Original: ( L 0 , . . .

Original: , L k ) and L (cid:48) k

Original: = ( L (cid:48) 0 , . . .

Original: , L (cid:48) k ) be the k -steplifted random walk as deﬁned in Equation (6) , given the random walk processes B and B (cid:48) on G and

Original: G (cid:48) respectively.

Original: Then, Ψ( G ) =

Original: Ψ( G (cid:48) )

Original: if and only if the distributions of L k and L (cid:48) k are equal.

Original: Proof.

Original: First, if the distributions of the two random walks L k and L (cid:48) k are equal, then it is clear that

Original: E [ (cid:101) ϕ

Original: ( L k )]

Original: =

Original: E [ (cid:101) ϕ

Original: ( L (cid:48) k )] .

Original: Next, suppose E [ (cid:101) ϕ

Original: ( L k )]

Original: =

Original: E [ (cid:101) ϕ

Original: ( L (cid:48) k )] .

Original: Then, note that the random walk distributions are ﬁnitelysupported (hence compactly supported) distributions, and thus by Theorem 4, they must be equal.

Original: This result shows that the hypo-elliptic diffusion completely characterizes random walk histories;thus providing a highly descriptive summary of labelled graphs.

Original: There is an analogous result for theindividual node features in terms of conditional expectations.

Original: Theorem 6.

Original: Let [Equation]
Revised: [Table]

Original: where ¯ x appends the time parametrization to the sequence x as in Equation (17) .

Original: Furthermore,suppose we have the resulting node feature map

Original: [Equation]

Original: Let G and G (cid:48) be two labelled graphs, and

Original: L k =

Original: ( L 0 , . . .

Original: , L k ) and L (cid:48) k

Original: = ( L (cid:48) 0 , . . .

Original: , L (cid:48) k ) be the k -steplifted random walk as deﬁned in Equation (6) , given the random walk processes B and B (cid:48) on G and

Original: G (cid:48) respectively.

Original: Then for two nodes i ∈ V and i (cid:48)

Original: ∈ V (cid:48) on the two respective graphs, Φ( i ) =

Original: Φ( i (cid:48) ) ifand only if the conditional distributions of P [ L k | B 0 = i ] and P [ L (cid:48)

Original: [-k | B (cid:48) 
-]
Revised: [-k | B (cid:48) 
-]

Original: = i (cid:48) ] are equal.

Original: Proof.

Original: The proof is analogous to the proof of Theorem 2 given above.

Original: Examples.

Original: As mentioned after Theorem 2 in the main text, the beneﬁts of capturing pathdependence can be signiﬁcant.

Original: In addition to the experimental results one can gain intuitionfor this by looking at simple toy examples.

Original: For example, consider as labels RGB colors, thatis f : V → [0 , 1] 3 ⊂ R 3 and random walks of length k

Original: = 2 on the two graphs G , G (cid:48) with vertex sets

Original: V = { v 1 , v 2 , v 3 } , resp.

Original: V (cid:48)

Original: = { v (cid:48) 1 , v (cid:48) 2 , v (cid:48) 3 } .

Original: We will now see two examples where features computed using classical diffusion cannot distinguisheither the nodes or the graphs, but features computed using hypo-elliptic diffusion can.

Original: Consider colors a, b, c ∈ R 3 as labels on the nodes of a graph with three nodes and two edges asare represented in Figures 1 and 2, with colors a and c on the end nodes and b on the middle node.

Original: After a random walk B 2 of length two, simple calculations show that the labels on the nodes will be12 ( a + c ) on the two end nodes, and b on the middle node.

Original: The ﬁrst example in Figure 1 considers node features.

Original: If one only has access to the marginaldistribution L 2 = f ( B 2 ) to construct node features, then the left-most and right-most nodes in thegraphs G and G (cid:48) from Figure 1 are indistinguishable, that is

Original: [Equation]

Original: In stark contrast, the laws of ( L 0 , L 1 , L 2 ) and ( L (cid:48) 0 , L (cid:48) 1 , L (cid:48) 2 ) clearly differ since L 1 , resp.

Original: L (cid:48) 1 , are withprobability one equal to green, resp.

Original: blue.

Original: Thus Theorem 2 guarantees that

Original: [Equation]

Original: In fact, for this example, a direct calculation shows that

Original: [Equation]

Original: The second example in Figure 2 considers graph features.

Original: The global graph features for 2-stepclassical diffusion is computed as the average of the node features, i.e. the global graph featureis 13 ( a + b + c ) .

Original: Hence, the graphs of Figure 2 are not distinguishable through features computedfrom standard diffusion.

Original: However, using hypo-elliptic diffusion allows us to take into account theorder in which the colors appear in the random walk.

Original: Indeed the colors red and green never appearconsecutively in the graph G (cid:48) , hence the e r ⊗ e g component of Ψ( G (cid:48) ) is zero; in other words,(cid:104)

Original: e r ⊗

Original: [Equation]

Original: [-g , Ψ( G ) (cid:105) = 1 (cid:54) = 
-]
Revised: [-g , Ψ( G ) (cid:105) = 1 (cid:54) = 
-]

Original: = (cid:104) e r ⊗ e g , Ψ

Original: ( G (cid:48) )

Original: (cid:105) , showing that Ψ( G ) (cid:54) =

Original: Ψ( G (cid:48) ) .

Original: Simple examples as the above demonstrate how using only the marginal distribution after k stepslacks expressive power for both node and graph features.

Original: Therefore, while classical diffusion smoothsout neighbourhood features, hypo-elliptic diffusion retains ﬁner descriptions of local neighbourhoods.

Original: F Details on the Low Rank Algorithm.

Original: In this section, we will provide further details and proofs on the low-rank approximation methoddiscussed in Section 4.

Original: We can use low-rank tensors to deﬁne corresponding low-rank functionalsof the features obtained via hypo-elliptic diffusion.

Original: The following is the deﬁnition of CP-rank of atensor from [10].

Original: Deﬁnition 1.

Original: The rank of a level m tensor v m ∈ ( R d ) ⊗ m is the smallest number r ≥ 0 such that wecan express v m as

Original: [Equation]

Original: We say that v = ( v 0 , v 1 , . . . )

Original: ∈ H is rank 1 if all v m are rank 1 tensors.

Original: We will now prove Theorem 3, which is stated using the node feature map without ZeroStart(see Appendix G).

Original: In particular, the node feature map is given by

Original: [Equation]

Original: where we explicitly specify the walk length k in the subscript.

Original: Furthermore, if we also need to specifythe tensor degree m , we will use two subscripts, where

Original: [Equation]

Original: is the level m component of the hypo-elliptic diffusion with a walk length of k .

Original: Throughout the proof,we omit the H subscript for the all-ones vector, such that 1 T :=

Original: (1 H , . .

Original: [-1 H ) ∈ H n , and we denote
-]
Revised: [-1 H ) ∈ H n , and we denote
-]

Original: Ti = (0 , . .

Original: , 1 H , . . . , 0) ∈ H n to be the unit vector in the i th coordinate.

Original: Proof of Theorem 3.

Original: First, we will show that f 1 ,m ( i ) = (cid:104) (cid:96) m , ˆΦ 1 ( i ) (cid:105) for all m = 1 , . . . , M . By thedeﬁnition of hypo-elliptic diffusion, we know that
Revised: aimed at incorporating several commonly used tricks for training GNNs.

Original: [Equation]

Original: By explicitly expressing the level m component, and by factoring out the inner product, we get

Original: [Equation]

Original: Then by linearity of the inner product, we get f 1 ,m ( i ) =

Original: (cid:104) (cid:96) m , ˆΦ 1 ( i ) (cid:105) .

Original: Next, we continue by induction and suppose that f k − 1 ,m ( i ) =

Original: (cid:104) (cid:96) m , ˆΦ k − 1 ( i ) (cid:105) holds for all m =1 , . .

Original: , M .

Original: Starting once again from the deﬁnition of hypo-elliptic diffusion, we know that

Original: [Equation]

Original: Fix a degree m .

Original: We explicitly write out the level m component of this equation by expanding (cid:101) P i,jand the tensor product as

Original: [Equation]

Original: Note that the ﬁrst sum is equivalent to(cid:88)

Original: [Equation]

Original: Applying the linear functional (cid:96) m and the induction hypothesis to this, we have

Original: [Equation]

Original: For the second sum in Equation (25), we can factor the inner product and apply the inductionhypothesis to get

Original: [Equation]

Original: Putting this all together, we get

Original: [Equation]

Original: Computing the ZeroStart Variation.

Original: We can adapt the recursive algorithm provided by Theorem 3above in order to compute low rank approximations to the ZeroStart variation of the node features,which is used throughout the main text (see also Appendix G).

Original: In particular, we consider

Original: [Equation]

Original: where ˆΦ k is the variation without ZeroStart deﬁned in Equation (24).

Original: [-Note that we can factor theϕ ( δ 0 L k ) term out of the expectation due to the conditioning B 
-]
Revised: [-Note that we can factor theϕ ( δ 0 L k ) term out of the expectation due to the conditioning B 
-]

Original: =

Original: i , and thus ϕ ( δ 0 L k ) =

Original: ϕ ( f ( i )) isﬁxed.

Original: Thus, we can compute low rank functionals of Φ k ( i ) using one additional step.

Original: Theorem 7.

Original: Using the same hypotheses as Theorem 3; let

Original: [Equation]

Original: where u m ∈ R d for m = 1 , . .

Original: , M and let

Original: [Equation]

Original: i )

Original: [Equation]

Original: Then,

Original: [Equation]

Original: where f k,m ( i ) = (cid:104) (cid:96) m , ˆΦ k ( i ) (cid:105) from Theorem 3.

Original: Proof.

Original: Using the deﬁnition of Φ k ( i ) from Equation (26), and expanding out the deﬁnition of ϕ ( δ 0 L k )at level M , we have

Original: [Equation]

Original: Then, taking the linear functional and distributing, we obtain the result

Original: [Equation]

Original: 1: Input: Graph G =

Original: ([ n ] , E , f ) with adjacency matrix A and transition matrix P 2: a rank- 1 tensor (cid:96) =

Original: u 1 ⊗ · · · ⊗ u M ∈ T ( R d ) , truncation level M ∈ N , walk length k ∈ N 3: Compute C [ m, i, j ] ← A

Original: [ i, j ] · (cid:104) u m , f ( j ) − f ( i ) (cid:105) for m ∈

Original: [ M ] , r ∈

Original: [ R ] , and i, j ∈ [ n ] 4: for i = 1 to k do 5: parfor m = 1 to M do 6: if i =

Original: = 1 then 7:

Original: Assign Q [1 , m, :] ← 1 m !

Original: ( P (cid:12) C

Original: [ M − m + 1 , : , :] (cid:12) . . .

Original: (cid:12) C

Original: [ M, : , :]) ·8: else 9:

Original: Assign Q [ i, m, :] ← P · Q

Original: [ i − 1 , m, :]

Original: + (cid:80) mj

Original: =1 1 j !

Original: ( P (cid:12) C

Original: [ M − m + 1 , : , :] (cid:12) . . .

Original: (cid:12)

Original: C [ M − m + j ]) ·

Original: Q [ i − 1 , m − j, :] 10: end if 11: end parfor 12: end for 13: Output: Vector Q [ k, m, :] of length n representing node features ( (cid:104) (cid:96) , ˆΦ k ( i ) (cid:105) )

Original: i =1 ,...,n

Original: Computational Complexity.

Original: We will now consider the computational complexity of our algorithms.

Original: We begin by noting that the naive approach of computing Φ k ( i ) =

Original: ( F (cid:101) P k 1 H ) ( i ) has thecomputational complexity of matrix multiplication; though this counts tensor operations, which itselfrequires O ( d m ) scalar multiplications at tensor degree

Original: m .

Original: This is computationally too expensive forpractical applications.

Original: Next, we consider the complexity of the recursive low-rank algorithm from Theorem 3, where theprimary computational advantage is the fact that we only perform scalar operations rather than tensoroperations. We consider the recursive step from Equation (15), reproduced here for m = M ,f k,M : [Equation]
Revised: We investigate the effect ofthe previous variations and ablate the architectural “tricks” by measuring the performance changeresulting from ceteris paribus removing it.

Original: P ·

Original: [Equation]

Original: k

Original: [Equation]

Original: 1 ,M

Original: [Equation]

Original: Given a graph G =

Original: ( V , E , f ) with n = |V| nodes and E = |E| edges, both P and C u are sparse n × n(scalar) matrices with O ( E ) nonzero entries, and f k,m is an n -dimensional column vector.

Original: Recallthat (cid:12) denotes element-wise multiplication, and thus both the sparse matrix-matrix multiplication andthe sparse matrix-vector multiplication have complexity O ( E ) .

Original: Furthermore, the entry-wise products

Original: C u 1 (cid:12) · · · (cid:12) C

Original: u r differ by only one factor between r = m and r = m + 1 , and thus, computingf k,M assuming all lower f k (cid:48) ,m (cid:48) have been computed has complexity O ( ME ) .

Original: Taking into accountthe two recursive parameters results in a complexity of O ( kM 2 E ) .

Original: Note that this is the complexityto compute features for all nodes .

Original: Once the f k,m are computed, the complexity of adding the start point from Theorem 7 is O ( M ) .

Original: Pseudocode.

Original: We provide pseudocode in Algorithm 1 to demonstrate the implementation of Theorem 3.

Original: The primary idea is to recursively compute the f k,m vector from Theorem 3 (renamed as anarray Q [: , : , :] to avoid confusion with the node features f ).

Original: In particular, note that the inner loop overm can be parallelized.

Original: Thus, while the theoretical complexity in terms of max tensor level is O ( M 2 )as discussed above, when we run the algorithm in parallel on a GPU, the computation time scalesroughly linearly, as shown empirically in Table 7 in Appendix H.4.

Original: G Variations and Hyperparameters of Hypo-Elliptic Diffusion

Original: In this appendix, we summarize possible variations of the sequence feature map, leading to differenthypo-elliptic diffusion features.

Original: The choice of variation is learned during training, and we alsosummarize the hyperparameters used for our features.

Original: While the theoretical results on characterizingrandom walks, such as Theorem 2, depend on speciﬁc choices of the sequence feature map, thereexist analogous results for these variations, which can characterize random walks up to certainequivalences.

Original: Furthermore, the computation of these variations can be performed in the same way:through tensorized linear algebra for exact solutions, and through an analogous low-rank method (asin Theorem 3) for approximate solutions.

Original: We ﬁx an algebra lifting ϕ : R d → H and let x = ( x 0 , . . .

Original: , x k ) ∈

Original: Seq ( R d ) .

Original: The simplest sequencefeature map to deﬁne simply multiplies the terms in the sequence together as

Original: [Equation]

Original: Note that this is not the sequence feature map used in the main text.

Original: We will now discuss severalvariations of this map, where (cid:101) ϕ inc , zs is the one primarily used in the main text and (cid:101) ϕ inc , zs , tp is used tocharacterize random walks in Theorem 2 and

Original: Appendix E.

Original: Increments ( Diff ).

Original: Rather than directly multiplying terms in the sequence together, we can insteadmultiply the increments as

Original: [Equation]

Original: where δ i x := x i − x i − 1 for i ≥ 1 .

Original: In both cases, the sequence feature map is the pathsignature of a continuous piecewise-linear path when we set ϕ = exp ⊗ , as discussedin Appendix C, and it is instructive to use this perspective to understand the effect ofincrements.

Original: If we use increments, the path corresponding to the sequence is

Original: X inc (

Original: [Equation]

Original: [Equation]

Original: x i + ( t − i )( x i +1 − x i ) for t ∈

Original: [ i, i + 1) , while if we do not use increments, the path corresponding to the sequence is

Original: X ( t ) : = i − 1 (cid:88) j = x j + ( t − i ) x i for t ∈ [ i, i + 1) .
Revised: Table 2 shows the result for G2T(A)N .

Original: Thus, when we use increments the sequence x corresponds to the vertices of the path X inc ,while if we do not, it corresponds to the vectors between vertices of the path X .

Original: In practice,this variation corresponds to taking ﬁrst-differences of the sequence x before using eq.

Original: Zero starting point ( ZeroStart ).

Original: The sequence feature map with increments, (cid:101) ϕ inc , as deﬁnedabove is translation-invariant , meaning (cid:101) ϕ inc ( x + a )

Original: = (cid:101) ϕ inc ( x ) , where x + a = ( x 0 + a, x 1 +a, . . .

Original: , x k + a ) for some a ∈ R d .

Original: In order to remove translation invariance, we can starteach sequence at the origin 0 ∈ R d by pre-appending a 0 to each sequence.

Original: A concise wayto deﬁne the resulting zero started sequence feature map is(cid:101)

Original: [Equation]

Original: where we deﬁne δ 0 x := x 0 .

Original: This is the sequence feature map deﬁned in Equation (1).

Original: Notethat this variation does not change the sequence feature map if we do not use increments.

Original: Time parametrization.

Original: When we relate sequences to piecewise linear paths as described in Appendix C, we can use the fact that the path signature is invariant under reparametrization,or more generally, tree-like equivalence [28].

Original: In terms of discrete sequences, this includesinvariance with respect to 0 elements in the sequence (without increments), and repeatedelements in the sequence (with increments).

Original: In order to remove this invariance, we caninclude time parametrization by setting

Original: [Equation]

Original: where ¯ x : = (¯ x 0 , . .

Original: , ¯ x k ) ∈

Original: Seq ( R d +1 ) , with ¯ x i : =

Original: ( i, x i ) ∈

Original: R d +1 .

Original: This is a simple form ofpositional encoding, but other encodings are also possible, e.g. sinusoidal waves as in [64].

Original: Algebra lifting ( AlgOpt ).

Original: Throughout this article, we have used the tensor exponential as thealgebra lifting.

Original: However, we can also scale each level of the lifting independently, and keepthese as hyperparameters to optimize.

Original: In particular, for a sequence c

Original: = ( c 0 , c 1 , . . . )

Original: ∈ R N ,we deﬁne ϕ c : R d → H to be

Original: [Equation]

Original: where 1 /m ! is used as initialization for c m and learned along with the other parameters.

Original: The choice of which variant of the sequence feature map to use depends on which invariance propertiesare important for the speciﬁc problem.

Original: In practice, the choice can be learned during the training,which is done in our experiments in Section 5.

Original: Furthermore, the features obtained through thelow-rank hypo-elliptic diffusion depend on three hyperparameters:

Original: [Figure]

Original: • the length of random walks;

Original: [Figure]

Original: • the number of low-rank functionals;

Original: • the maximal tensor degree;

Original: [Figure]

Original: • the number of iterations (layers).

Original: Note that the ﬁrst three hyperparameters can also potentially vary across different iterations.

Original: H Experiments

Original: We have implemented the low-rank algorithm for our layers given in Theorem 3 using Tensorﬂow,

Original: Keras and Spektral [25].

Original: Code is available at GITHUBAUTHOR .

Original: [-All experiments were run on one of 3computing clusters that were overall equipped with 5 NVIDIA Geforce 2080Ti, 2 Quadro GP100, 
-]
Revised: [-All experiments were run on one of 3computing clusters that were overall equipped with 5 NVIDIA Geforce 2080Ti, 2 Quadro GP100, 
-]

Original: A100 GPUs.

Original: The largest GPU memory allocation any one of the experiments required was around∼ 5 GB.

Original: For the experiments ran using different random seeds, the seed was used to control therandomness in the (1) data splitting process, (2) parameter initialization, (3) optimization procedure.

Original: For an experiment with overall n runs number of runs, the used seeds were { 0 , . . . , n runs − 1 } .

Original: H.1 Model details.

Original: The architecture described in Section 5 is visualized conceptually on Figure 3, where for G2T(A)Nthe G2TN stack is replaced by the attentional variation without changing any hyperparameters.

Original: Furtherto the discussion given in the main text, here we provide more details on the model implementation.

Original: Initialization.

Original: In our G2TN and G2T(A)N layers, we linearly project tensor-valued features usinglinear functionals that use the same low-rank parametrization as the recursive variation in [62,

Original: App. D.2].

Original: There, the authors also propose in Appendix D.5 an initialization heuristic for setting thevariances of these component vectors, that we have employed with a centered uniform distribution.

Original: Regularization.

Original: We also apply (cid:96) 2 regularization for both experiments in Sections 5 and H.3 forwhich we discuss the implementation here.

Original: Given a max.

Original: tensor degree M ≥ 1 and a max.

Original: tensorrank R ≥ 1 representing the layer width, there are overall RM

Original: rank-

Original: 1 linear functionals of the form(cid:96)

Original: [Equation]

Original: for m = 1 , . . .

Original: M and r = 1 , . .

Original: R.

Original: Since in practice these are represented using only the component vectors u ri ∈ R d , a naive applicationof (cid:96) 2 regularization would lead to computing the 2 -norm of each component vector u ri to penalize theloss function.

Original: However, we found this approach to underperform compared to the following idea.

Original: Although our algorithm represents the functionals using a rank- 1 decomposition and computes theprojection of each tensor-valued node feature without explicitly building high-dimensional tensors,conceptually we still have tensors ( (cid:96) rm ) acting on tensors ( Φ( i ) for i ∈ V ), and hence the tensor norm,(cid:107) (cid:96) rm (cid:107) 2 , should be used for regularization.

Original: Fortunately, this can also be computed efﬁciently:

Original: [Equation]

Original: Further, as is common, we replace the (cid:96) 2 norm with its squared value, sum over all functionals in(28), and multiply by the regularization parameter λ > 0 so that the ﬁnal penalty is given by

Original: [Equation]

Original: H

Original: 2 Experiment Details.

Original: Here we provide further details on the main experiment described in Section 5.

Original: Hyperparameter Selection.

Original: The model architecture was primarily motivated by GraphTrans(small) from [70]. Speciﬁcally, the number and width of GNN layers, and the dropout rate wasadopted as is. The (cid:96) 2 regularization strength was chosen equal to  the weight decay rate. The otherhyperparameters were tuned on a single split of NCI1.
Revised: To summarizethe main observations, the model is robust to all the architectural changes, removing the layer normeven improves on NCI109.

Original: For the random walk length, we experimentedwith values 2 , 5 , 10 .

Original: For the given GNN depth ( 4 ), 2 RW steps per layer was not enough to learnlong-range interactions, while 10 signiﬁcantly slowed down the convergence rate during training.

Original: Forthe maximal degree of tensors, we experimented with values from 2 , 3 , 4 , and using values beyond2 did not provide improvements.

Original: Intuitively, the tensor degree represents the order of nonlinearinteractions that are learnable by the layer, e.g. a degree of 2 encodes pairwise interactions betweennode features in the neighbourhood, while a higher degree of M allows to encode interactionsbetween certain M -tuples of nodes.

Original: We suggest that a degree of 2 is a good baseline setting, andthat increasing the GNN depth instead allows to efﬁciently capture higher order interactions, whileincreasing the effective inﬂuence radius at the same time.

Original: For the pre- and postprocessing layers, weexperimented with various depths and choosing more than 1 layer for each was counterproductive.

Original: The number of units was simply set to the GNN width ( 128 ) in the preprocessing layer, while for thepostprocessing layer slightly increasing it was found to provide improvements ( 256 ), potentially tocompensate for the large amount of information that is compressed in the pooling step.

Original: [Table]

Original: Ablation Studies.

Original: Further to using attention, we give a list of ideas for variations on our models in

Original: Appendix G, which can be summarized brieﬂy as: (i) using increments of node attributes ( Diff ),(ii) preprending a zero point to sequences ( ZeroStart ), (iii) optimizing over the algebra embedding( AlgOpt ), all of which are built into our main models.

Original: Further, the previous architectural choicesaimed at incorporating several commonly used tricks for training GNNs.

Original: We investigate the effect ofthe previous variations and ablate the architectural “tricks” by measuring the performance changeresulting from ceteris paribus removing it.

Original: Table 2 shows the result for G2T(A)N . To summarizethe main observations, the model is robust to all the architectural changes, removing the layer normeven improves on NCI109. Importantly, replacing the attention pooling with mean pooling does notsigniﬁcantly affect the performance, but actually slightly improves on NCI1.
Revised: Importantly, replacing the attention pooling with mean pooling does notsigniﬁcantly affect the performance, but actually slightly improves on NCI1.

Original: Regarding variations,
Revised: Regarding variations,

Original: AlgOpt slightly improves on both datasets, while removing Diff and ZeroStart signiﬁcantlydegrades the accuracy on NCI1.
Revised: AlgOpt slightly improves on both datasets, while removing Diff and ZeroStart signiﬁcantlydegrades the accuracy on NCI1.

Original: The latter means that translations of node attributes are important,not just their distances.
Revised: The latter means that translations of node attributes are important,not just their distances.

Original: We give further discussions and ablations in Appendix H.2 including G2TN .Here we extend the investigation of our model architectures that was partially given in Section 5 in Table 2.
Revised: We give further discussions and ablations in Appendix H.2 including G2TN .

Original: We give the analogous ablation study for the G2TN model in Table 3, and compare the derivedconclusions between the attentional and attention-free versions. First, we discuss the layer variations. Similarly to G2T(A)N , NoDiff slightly decreases the accuracy. However the conclusions regarding
Revised: Discussion.

Original: NoZeroStart and NoAlgOpt are different. In this case, removing ZeroStart actually improvesthe performance, while on G2T(A)N the opposite was true. An interpretation of this phenomenon isthat only the attention mapping that is used to learn the random walks required information abouttranslations of the layer’s features, and not the tensor-features themselves. Another difference is that
Revised: The previous experiments demonstrate that our approach performs very favourablyon long-range reasoning tasks compared to GNN-based alternatives without global pairwise nodeinteractions.

Original: NoAlgOpt degrades the accuracy more signiﬁcantly for G2TN . A possible explanation is that since G2T(A)N layers are more ﬂexible thanks to their use of attention, they rely less on being able to learnthe algebraic lift, while as G2TN layers are more rigid in their random walk deﬁnition, and beneﬁtmore from the added ﬂexibility of AlgOpt . Additionally, it seems that G2TN is more sensitive to thevarious architectural options, and removing any of them, i.e. NoJK , NoSkip , NoNorm or AvgPool ,degrades the accuracy by 1% or more on at least one dataset.
Revised: Several of the works we compare against have focused on extending GNNs to largerneighbourhoods by speciﬁcally designed graph coarsening and pooling operations, and we emphasizetwo important points: (1) our approach can efﬁciently capture large neighbourhoods without anyneed for coarsening, (2) it already performs well with simple mean-pooling as justiﬁed by Theorem 2and experimentally supported by the ablations.

Original: Intuitively, it seems that overall the G2T(A)N model is more robust to the various architectural “tricks”, and more adaptable due to itsability to learn  the the random walk. [Table] [Table]
Revised: Although the Transformer-based GraphTrans slightlyoutperforms our model potentially due to its ability to learn global interactions, it is not entirelyclear how much of the global graph structure it is able to infer from interactions of short-rangeneighbourhood summaries.

Original: Further Experiments

Original: [Table]

Original: Citation datasets.

Original: Additionally to transductive learning on the biological datasets, we have carriedout inductive learning tasks on some of the common citation datasets, i [Table] e. Cora, Citeseer [56] and
Revised: Intuitively, better and larger summaries should also help the Transformersubmodule to make inference about questions regarding the global structure.

Original: Pubmed [47].

Original: We follow [57] in carrying out the experiment

Original: [Table]

Original: and use the largest connected componentfor each dataset with 20 training examples per class, 30 validation examples per class, and the restof the data used for testing. The hyperparameters of our models and optimization procedure werebased on the settings of the GAT model in [57, Table 4], which we have slightly ﬁne-tuned on Coraand used for the other datasets. In particular, a single layer of G2TN or G2T(A)N is used with 64functionals, max. tensor degree 2 and random walk length 5 .
Revised: Finally, Transformermodels can be bottlenecked by their quadratic complexity in nodes, while our approach only scaleswith edges, and hence, it can be more favourable for large sparse graphs in terms of computations.

Original:  The dropout rate was tuned to 0 . 9 ,while the attentional dropout was set to 0 . 3 in G2T(A)N . Optimization is carried out with Adam[32], a ﬁxed learning rate of 0 .
Revised:  Conclusion

Original: 01 and (cid:96) 2 regularization strength 0 .

Original: Training is stopped once thevalidation loss does not improve for 50 epochs, and restored to the best checkpoint.

Original: For both of ourmodels, NoDiff is used that we found to improve on the results as opposed to using increments ofnode features.

Original: The dropout rate had to be tuned as high as 0 . 9 , which suggests very strong overﬁtting,hence the additional complexity of AlgOpt was also contrabeneﬁcial, and NoAlgOpt was used.

Original: Assuch, each model employs a single hidden layer, which is followed by layer normalization that wasfound to perform slightly better than other normalizations, e.g. graph-level normalization. The resultsof our models over 100 runs are reported in Table 4 compared with the 4 consistentlybest performing baselines on these datasets from [57], i.e. GCN [33], GAT [65], MoNet [45], and GraphSage [29] with a mean aggregator. Firstly on Cora, both G2TN and G2T(A)N outperform thebaselines with a more signiﬁcant improvement for G2TN .
Revised: Inspired by classical results from analysis [22], we introduce the hypo-elliptic graph Laplacian.

Original: For CiteSeer, our models are left somewhatbehind compared to the baselines in terms of accuracy.

Original: Finally, they are again competitive on Pubmed,where G2TN takes the top score with a very slight lead.

Original: Two consistent observations are: (1) G2TNand G2T(A)N have a lower variance than all baselines, (2) G2TN consistently outperforms G2T(A)N .

Original: The latter may be attributed to the observation that due to the severe overﬁtting on these datasets, theadditional complexity of the attention mechanism in G2T(A)N is unhelpful for generalization.

Original: K -hop Sanitized Splits.

Original: Recent work [50] has demonstrated that it is possible to make the previously considered citation datasets more suitable for testing the ability of a model to learn long-rangeinformation by dropping node labels in a structured way. Concretely, they use a label resamplingstrategy to guarantee that if a node is selected for a data split, none of its k -th degree neighbours areincluded in any splits, i.e. training, validation nor testing, allowing to reduce the effect of short-range“label imprinting”. In practice, we select a maximal independent set from the graph with respect tothe k -th power of the adjacency matrix with self-loops, andrepeat the previous experiment with thesame data splitting method, model choice and training settings as before. In this case, the experimentseed is also used to control the random maximal independent set that is selected.
Revised: Thisyields a diffusion equation and also generalizes its classical probabilistic interpretation via randomwalks but now takes their history into account.

Original: The results of our models trained on the citation datasets sanitized this way are available in Table 5computed over 100 seeds for k = 1 , 2 .

Original: As baseline results, we compare against the 5 best performingmodels from [50], where various GNN depths were also considered for each model, and we use thebest reported result for each of the baselines. Overall, we can observe that all baseline models exhibita very sharp drop in performance as k is increased, while for G2TN and G2T(A)N , the performance [Table] [Table]
Revised: In addition to several attractive theoretical guarantees,we provide scalable algorithms.

Original: decrease is not nearly as pronounced.

Original: For both k = 1 and k = 2 , our models perform better thanthe baselines on all datasets. This is explained by the fact that due to using random walks length of5 , the models canefﬁciently pick up on information outside of the sanitized neighbourhoods. Aspreviously, G2TN performs better than G2T(A)N as the additional ﬂexibility of the attention layer doesnot lead to  improved generalization performance when severe overﬁtting is present. This experimentdemonstrates that the proposed models efﬁciently pick up on long-range information within largerneighbourhoods, and it suggests that on inductive learning tasks they should be more robust to sparselabeling rates compared to common short-range GNN models.
Revised: Our experiments show that this can lead to largely improved baselinesfor long-range reasoning tasks.

Original: Table 6 shows the computation time of one forward pass of a G2TN layer with various graphs. Thisempirically demonstrates that our layer does not depend on the number of nodes, and only dependslinearly on the the number of edges as expected from the theoretical complexity. Table 7 shows the computation time of one forward pass of a G2TN layer with a ﬁxed graph andvarious model parameters.
Revised: A promising future research theme is to develop improvements forthe classical Laplacian in this hypo-elliptic context; this includes lazy random walks [65]; nonlineardiffusions [11]; and source/sink terms [54].

Original: Empirically, our layer scales roughly linearly in both the walk length kand the maximum tensor level M .
Revised: Another theme could be to extend the geometric study[55] of over-squashing to this hypo-elliptic point of view which is naturally tied to sub-Riemanniangeometry [53]. Checklist For all authors...

Original: Linear complexity inwalk length is expected, but the linearity inthe maximum tensor level M is due to parallelization of the algorithm presented in Theorem 3.
Revised: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope? [Yes] (b) Did you describe the limitations of your work?

Revised: [Yes] While our approach is less

Revised: susceptible to the “over-squashing” phenomenon and empirically performs well on

Original: Thisis discussed inthe pseudocode section of Appendix F.
Revised: learning long-range interactions, we ﬁnd that our methods are less beneﬁcial for inductive learning tasks. This is stated explicitly in Section 5, with experiments shown in Appendix H.3.

Original: H.5 Parameter Count
Revised: (c) Did you discuss any potential negative societal impacts of your work? [No] This article introduce a method which can improve the performance on graph neural networks. We

Original: In both the NCI datasets, the G2TN and G2T(A)N have 505k and 519k trainable parameters respectively.
Revised: don’t think there are any foreseeable negative societal impacts beyond generic points that apply to any graph learning method. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: Thus, our models are comparable in size to the GraphTrans, which is reported to have 500ktrainable parameters [70].
Revised: [Yes] If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results? [Yes] All theorem

Original: H.6 Summary of Dataset Parameters
Revised: statements contain the full set of assumptions. (b) Did you include complete proofs of all theoretical results? [Yes] Proofs to all of our theoretical results are contained in the appendix, with speciﬁc references to sectionswithin the main text.

Original: Here, we provide a summary of the dataset statistics and  the model parameters used in the experiments.
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] , see Appendix

Original: Here, M denotes the maximum tensor degree, k denotes the walk length, and R denotes the maximumrank of the low rank functions.
Revised: H (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] This is detailed in Section 5, with further discussion in Appendix H.2.

Revised: (c) Did you report error bars (e.g., with respect to the random seed after running exper-

Revised: iments multiple times)?

Original: Concretely, R is  the number of node features used within the network,and is analogous to the width of a neural network.
Revised: [Yes] All experiments are run using multiple seeds, and the standard deviation over these seeds is provided. (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: For the computation of the diameter of adisconnected graph, we take the largest diameter of any connected component.
Revised: [Yes] See Appendix H. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators? [Yes]

Revised: (b) Did you mention the license of the assets?

Original: [Table]
Revised: [Yes] We include it in our code. (c) Did you include any new assets either in the supplemental material or as a URL? [Yes] We include code as part of supplementary material.

Original: [Table]
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating? [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable

Original: 64 Pubmed 19k
Revised: information or offensive content? [N/A] If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if

Original: [Table]
Revised: applicable? [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [Table]
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]