Original: D
Revised: D

Original: IRECTIONAL G RAPH N ETWORKS
Revised: IRECTIONAL G RAPH N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In order to overcome the expressive limitations of graph neural networks (GNNs), we propose the ﬁrst method that exploits vector ﬂows over graphs to develop globally consistent directional and asymmetric aggregation functions.
Revised: In order to overcome the expressive limitations of graph neural networks (GNNs), we propose the ﬁrst method that exploits vector ﬂows over graphs to develop globally consistent directional and asymmetric aggregation functions.

Original: We show that our directional graph networks (DGNs) generalize convolutional neural networks (CNNs) when applied on a grid.
Revised: We show that our directional graph networks (DGNs) generalize convolutional neural networks (CNNs) when applied on a grid.

Original: Whereas recent theoretical works focus on understanding local neighbourhoods, local structures and local isomorphism with no global information ﬂow, our novel theoretical framework allows directional convolutional kernels in any graph.
Revised: Whereas recent theoretical works focus on understanding local neighbourhoods, local structures and local isomorphism with no global information ﬂow, our novel theoretical framework allows directional convolutional kernels in any graph.

Original: First, by deﬁning a vector ﬁeld in the graph, we develop a method of applying directional derivatives and smoothing by projecting node-speciﬁc messages into the ﬁeld.
Revised: First, by deﬁning a vector ﬁeld in the graph, we develop a method of applying directional derivatives and smoothing by projecting node-speciﬁc messages into the ﬁeld.

Original: Then we propose the use of the Laplacian eigenvectors as such vector ﬁeld, and we show that the method generalizes CNNs on an n -dimensional grid.
Revised: Then we propose the use of the Laplacian eigenvectors as such vector ﬁeld, and we show that the method generalizes CNNs on an n -dimensional grid, and is provably more discriminative than standard GNNs regarding the Weisfeiler-Lehman 1-WL test.

Original: Finally, we bring the power of CNN data augmentation to graphs by providing a means of doing reﬂection, rotation and distortion on the underlying directional ﬁeld.
Revised: Finally, we bring the power of CNN data augmentation to graphs by providing a means of doing reﬂection, rotation and distortion on the underlying directional ﬁeld.

Original: We evaluate our method on different standard benchmarks and see a relative error reduction of 8% on the CIFAR10 graph dataset and 11% to 32% on the molecular ZINC dataset.
Revised: We evaluate our method on different standard benchmarks and see a relative error reduction of 8% on the CIFAR10 graph dataset and 11% to 32% on the molecular ZINC dataset.

Original: An important outcome of this work is that it enables to translate any physical or biological problems with intrinsic directional axes into a graph network formalism with an embedded directional ﬁeld.
Revised: An important outcome of this work is that it enables to translate any physical or biological problems with intrinsic directional axes into a graph network formalism with an embedded directional ﬁeld.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: One of the most important distinctions between convolutional neural networks (CNNs) and graph neural networks (GNNs) is that CNNs allow for any convolutional kernel, while most GNN methods are limited to symmetric kernels (also called isotropic kernels in the literature)
Revised: One of the most important distinctions between convolutional neural networks (CNNs) and graph neural networks (GNNs) is that CNNs allow for any convolutional kernel, while most GNN methods are limited to symmetric kernels (also called isotropic kernels in the literature)

Original: (Kipf & Welling, 2016; Xu et al., 2018a; Gilmer et al., 2017).
Revised: (Kipf & Welling, 2016; Xu et al., 2018a; Gilmer et al., 2017).

Original: There are some implementation of asymmetric kernels using gated mechanisms (Bresson & Laurent, 2017;
Revised: There are some implementation of asymmetric kernels using gated mechanisms (Bresson & Laurent, 2017;

Original: Veli ˇ ckovi ´ c et al., 2017), motif attention (Peng et al., 2019), edge features (Gilmer et al., 2017) or by using the 3D structure of molecules for message passing (Klicpera et al., 2019).
Revised: Veli ˇ ckovi ´ c et al., 2017), motif attention (Peng et al., 2019), edge features (Gilmer et al., 2017) or by using the 3D structure of molecules for message passing (Klicpera et al., 2019).

Original: However, to the best of our knowledge, there are currently no methods that allow asymmetric graph kernels that are dependent on the full graph structure oron directional ﬂows.
Revised: However, to the best of our knowledge, there are currently no methods that allow asymmetric graph kernels that are dependent on the full graph structure or directional ﬂows.

Original: They either depend on local structures or local features.
Revised: They either depend on local structures or local features.

Original: This is in opposition to images which exhibit canonical directions: the horizontal and vertical axes.
Revised: This is in opposition to images which exhibit canonical directions: the horizontal and vertical axes.

Original: The absence of an analogous concept in graphs makes it difﬁcult to deﬁne directional message passing and to produce an analogue of the directional frequency ﬁlters (or Gabor ﬁlters) widely present in image processing (Olah et al., 2020).
Revised: The absence of an analogous concept in graphs makes it difﬁcult to deﬁne directional message passing and to produce an analogue of the directional frequency ﬁlters (or Gabor ﬁlters) widely present in image processing (Olah et al., 2020).

Original: We propose a novel idea for GNNs: use vector ﬁelds in the graph to deﬁne directions for the propagation of information.
Revised: We propose a novel idea for GNNs: use vector ﬁelds in the graph to deﬁne directions for the propagation of information, with an overview of the paper presented in 1.

Original: Hence, the aggregation or message passing will be projected onto these directions so that the contribution of each neighbouring node n v will be weighted by its alignment with the vector ﬁelds at the receiving node n u .
Revised: Hence, the aggregation or message passing will be projected onto these directions so that the contribution of each neighbouring node n v will be weighted by its alignment with the vector ﬁelds at the receiving node n u .

Original: This enables our method to propagate information via directional derivatives or smoothing of the features.
Revised: This enables our method to propagate information via directional derivatives or smoothing of the features.

Original: We also explore using the gradients of the low-frequency eigenvectors of the Laplacian of the graph φ k , since they exhibit interesting properties (Bronstein et al., 2017; Chung et al., 1997).
Revised: We also explore using the gradients of the low-frequency eigenvectors of the Laplacian of the graph φ k , since they exhibit interesting properties (Bronstein et al., 2017; Chung et al., 1997).

Original: In particular, they can be used to deﬁne optimal partitions of the nodes in a graph, to give a natural ordering (Levy, 2006), and to ﬁnd the dominant directions of the graph diffusion process (Chung & Yau, 2000).
Revised: In particular, they can be used to deﬁne optimal partitions of the nodes in a graph, to give a natural ordering (Levy, 2006), and to ﬁnd the dominant directions of the graph diffusion process (Chung & Yau, 2000).

Original: Further, we show that they generalize the horizontal and vertical directional ﬂows in a grid (see ﬁgure 1), allowing them to guide the aggregation and mimic the asymmetric and directional kernels present in computer vision.
Revised: Further, we show that they generalize the horizontal and vertical directional ﬂows in a grid (see [Figure] [Table] [Figure]

Revised: [Table]

Original: In fact, we demonstrate mathematically that our work generalizes CNNs by reproducing all convolutional kernels of radius R in an n -dimensional grid, while also bringing the powerful data augmentation capabilities of reﬂection, rotation or distortion of the directions.
Revised: [Figure] [Table] [Figure] [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Original:  We further show that our directional graph network (DGN) model theoretically and empirically allows for efﬁcent message passing across distant communities, which reduces the well known problem of over-smoothing, and aligns well with the need of independent aggregation rules (Corso et al., 2020).
Revised: [Figure] ﬁgure 2), allowing them to guide the aggregation and mimic the asymmetric and directional kernels present in computer vision. In fact, we demonstrate mathematically that our work generalizes CNNs by reproducing all convolutional kernels of radius R in an n -dimensional grid, while also bringing the powerful data augmentation capabilities of reﬂection, rotation or distortion of the directions. We further show that our directional graph network (DGN) model theoretically and empirically allows for efﬁcient message passing across distant communities, which reduces the well-known problem of over-smoothing, and aligns well with the need of independent aggregation rules (Corso et al., 2020).

Original: Alternative methods reduce the impact of over-smoothing by using skip connections (Luan et al., 2019) or global pooling (Alon & Yahav, 2020),  but without solving the underlying problem.
Revised: Alternative methods reduce the impact of over-smoothing by using skip connections (Luan et al., 2019), global pooling (Alon & Yahav, 2020), or randomly dropping edges during training time (Rong et al., 2020), but without solving the underlying problem.

Revised: In fact, we also prove that DGN is more discriminative than standard GNNs in regards to the Weisfeiler-Lehman 1-WL test, showing that the reduction of over-smoothing is accompanied by an increase of expressiveness.

Original: Our method distinguishes itself from other spectral GNNs since the literature usually uses the low frequencies to estimate local Fourier transforms in the graph (Levie et al., 2018; Xu et al., 2019).
Revised: Our method distinguishes itself from other spectral GNNs since the literature usually uses the low frequencies to estimate local Fourier transforms in the graph (Levie et al., 2018; Xu et al., 2019).

Original: Instead, we do not try to approximate the Fourier transform, but only to deﬁne a directional ﬂow at each node and guide the aggregation.
Revised: Instead, we do not try to approximate the Fourier transform, but only to deﬁne a directional ﬂow at each node and guide the aggregation.

Original:  T HEORETICAL DEVELOPMENT 2. I NTUITIVE OVERVIEW
Revised:  T HEORETICAL DEVELOPMENT 2. I NTUITIVE OVERVIEW

Original: One of the biggest limitations of current GNN methods compared to CNNs is the inability to do message passing in a speciﬁc direction such as the horizontal one in a grid graph.
Revised: One of the biggest limitations of current GNN methods compared to CNNs is the inability to do message passing in a speciﬁc direction such as the horizontal one in a grid graph.

Original: In fact, it is difﬁcult to deﬁne directions or coordinates based solely on the shape of the graph.
Revised: In fact, it is difﬁcult to deﬁne directions or coordinates based solely on the shape of the graph.

Original: The lack of directions strongly limits the discriminative abilities of GNNs to understand local structures and simple feature transformations.
Revised: The lack of directions strongly limits the discriminative abilities of GNNs to understand local structures and simple feature transformations.

Original: Most GNNs are invariant to the permutation of the neighbours’ features, so the nodes’ received signal is not inﬂuenced by swapping the features of 2 neighbours.
Revised: Most GNNs are invariant to the permutation of the neighbours’ features, so the nodes’ received signal is not inﬂuenced by swapping the features of 2 neighbours.

Original: Therefore, several layers in a deep network will be employed to understand these simple changes instead of being used for higher level features, thus over-squashing the message sent between 2 distant nodes (Alon & Yahav, 2020).
Revised: Therefore, several layers in a deep network will be employed to understand these simple changes instead of being used for higher level features, thus over-squashing the message sent between 2 distant nodes (Alon & Yahav, 2020).

Original: In this work, one of the main contribution is the realisation that low frequency eigenvectors of the
Revised: In this work, one of the main contributions is the realisation that low-frequency eigenvectors of the

Original: Laplacian can overcome this limitation by providing a variety of intuitive directional ﬂows.
Revised: Laplacian can overcome this limitation by providing a variety of intuitive directional ﬂows.

Original: As a ﬁrst example, taking a grid-shaped graph of size N × M with N 2 < M < N , we ﬁnd that the eigenvector associated to the smallest non-zero eigenvalue increases in the direction of the width N and the second one increases in the direction of the height M .
Revised: As a ﬁrst example, taking a grid-shaped graph of size N × M with N 2 < M < N , we ﬁnd that the eigenvector associated to the smallest non-zero eigenvalue increases in the direction of the width N and the second one increases in the direction of the height M .

Original: This property generalizes to n-dimensional grids and motivated the use of gradients of eigenvectors as preferred directions for general graphs.
Revised: This property generalizes to n-dimensional grids and motivated the use of gradients of eigenvectors as preferred directions for general graphs.

Original: We validated this intuition by looking at the ﬂow of the gradient of the eigenvectors for a variety of graphs, as shown in ﬁgure 1.
Revised: We validated this intuition by looking at the ﬂow of the gradient of the eigenvectors for a variety of graphs, as shown in ﬁgure 2.

Original: For example, in the Minnesota map, the ﬁrst 3 non-constant eigenvectors produce logical directions, namely South/North, suburb/city, and West/East.
Revised: For example, in the Minnesota map, the ﬁrst 3 non-constant eigenvectors produce logical directions, namely South/North, suburb/city, and West/East.

Original: Another important contribution also noted in ﬁgure 1 is the ability to deﬁne any kind of direction based ona prior knowledge of the problem.
Revised: Another important contribution also noted in ﬁgure 2 is the ability to deﬁne any kind of direction based on prior knowledge of the problem.

Original: Hence, instead of relying on eigenvectors to ﬁnd directions in a map, we can simply use the cardinal directions or the rush-hour trafﬁc ﬂow.
Revised: Hence, instead of relying on eigenvectors to ﬁnd directions in a map, we can simply use the cardinal directions or the rush-hour trafﬁc ﬂow.

Original: 2. V ECTOR FIELDS IN A GRAPH
Revised: 2. V ECTOR FIELDS IN A GRAPH

Original: Based on a recent review from Bronstein et al.
Revised: Based on a recent review from Bronstein et al.

Original: (2017), this section presents the ideas of differential geometry applied to graphs, with the goal of ﬁnding proper deﬁnitions of scalar products, gradients and directional derivatives.
Revised: (2017), this section presents the ideas of differential geometry applied to graphs, with the goal of ﬁnding proper deﬁnitions of scalar products, gradients and directional derivatives.

Original: Let G =
Revised: Let G =

Original: ( V, E ) be a graph with V the set of vertices and E
Revised: ( V, E ) be a graph with V the set of vertices and E

Original: ⊂ V × V the set of edges.
Revised: ⊂ V × V the set of edges.

Original: The graph is undirected meaning
Revised: The graph is undirected meaning

Original: that ( i, j ) ∈
Revised: that ( i, j ) ∈

Original: E iff ( j, i ) ∈ E .
Revised: E iff ( j, i ) ∈ E .

Original: Deﬁne the vector spaces L 2 ( V ) and L 2 ( E ) as the set of maps V → R and E
Revised: Deﬁne the vector spaces L 2 ( V ) and L 2 ( E ) as the set of maps V → R and E

Original: → R with x , y ∈ L 2 ( V ) and F ,
Revised: → R with x , y ∈ L 2 ( V ) and F ,

Original: H ∈
Revised: H ∈

Original: L 2 ( E ) and scalar products(cid:104) x , y (cid:105) L 2 ( V ) : = (cid:88) i ∈ V x i
Revised: L 2 ( E ) and scalar products

Original: y i , (cid:104) F , H (cid:105)

Original: L 2 ( E ) :

Original: = (cid:88) (

Original: [Equation]
Revised: [Equation]

Original: Think of E as the “tangent space” to V and of L 2 ( E ) as the set of “vector ﬁelds” on the space V with each row F i, : representing a vector at the i -th node.
Revised: Think of E as the “tangent space” to V and of L 2 ( E ) as the set of “vector ﬁelds” on the space V with each row F i, : representing a vector at the i -th node.

Original: Deﬁne the pointwise scalar product as the map L 2 ( E ) ×
Revised: Deﬁne the pointwise scalar product as the map L 2 ( E ) ×

Original: L 2 ( E ) → L 2 ( V ) taking 2 vector ﬁelds and returning their inner product at each point of V , at the node i is deﬁned by equation 2.
Revised: L 2 ( E ) → L 2 ( V ) taking 2 vector ﬁelds and returning their inner product at each point of V , at the node i is deﬁned by equation 2.

Original: [Equation]
Revised: [Equation]

Original: In equation 3, we deﬁne the gradient ∇ as a mapping L 2 ( V ) →
Revised: In equation 3, we deﬁne the gradient ∇ as a mapping L 2 ( V ) →

Original: L 2 ( E ) and the divergence div as a mapping L 2 ( E ) → L 2 ( V ) , thus leading to an analogue of the directional derivative in equation 4.
Revised: L 2 ( E ) and the divergence div as a mapping L 2 ( E ) → L 2 ( V ) , thus leading to an analogue of the directional derivative in equation 4.

Original: [Equation]
Revised: [Equation]

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: The directional derivative of the function x on the graph G in the direction of the vector ﬁeld ˆ F where each vector is of unit-norm is
Revised: The directional derivative of the function x on the graph G in the direction of the vector ﬁeld ˆ F where each vector is of unit-norm is

Original: [Equation]
Revised: [Equation]

Original: | F | will denote the absolute value of F and || F i, :
Revised: | F | will denote the absolute value of F and || F i, :

Original: || L p the L p -norm of the i -th row of F .
Revised: || L p the L p -norm of the i -th row of F .

Original: We also deﬁne the forward/backward directions as the positive/negative parts of the ﬁeld F ± .
Revised: We also deﬁne the forward/backward directions as the positive/negative parts of the ﬁeld F ± .

Original: 2. D IRECTIONAL SMOOTHING AND DERIVATIVES
Revised: 2. D IRECTIONAL SMOOTHING AND DERIVATIVES

Original: Next, we show how the vector ﬁeld F is used to guide the graph aggregation by projecting the incoming messages.
Revised: Next, we show how the vector ﬁeld F is used to guide the graph aggregation by projecting the incoming messages.

Original: Speciﬁcally, we deﬁne the weighted aggregation matrices B av and B dx that allow to compute the directional smoothing and directional derivative of the node features.
Revised: Speciﬁcally, we deﬁne the weighted aggregation matrices B av and B dx that allow to compute the directional smoothing and directional derivative of the node features.

Original: The directional average matrix B av is the weighted aggregation matrix such that all weights are positives and all rows have an L 1 -norm equal to 1, as shown in equation 5 and theorem 2.1, with a proof in the appendix C.1.
Revised: The directional average matrix B av is the weighted aggregation matrix such that all weights are positives and all rows have an L 1 -norm equal to 1, as shown in equation 5 and theorem 2.1, with a proof in the appendix C.1.

Original: [Equation]
Revised: [Equation]

Original: The variable (cid:15) is an arbitrarily small positive number used to avoid ﬂoating point errors.
Revised: The variable (cid:15) is an arbitrarily small positive number used to avoid ﬂoating-point errors.

Original: The L 1norm denominator is a local row-wise normalization.
Revised: The L 1norm denominator is a local row-wise normalization.

Original: The aggregator works by assigning a large weight to the elements in the forward or backward direction of the ﬁeld, while assigning a small weight to the other elements, with a total weight of 1.
Revised: The aggregator works by assigning a large weight to the elements in the forward or backward direction of the ﬁeld, while assigning a small weight to the other elements, with a total weight of 1.

Original: Theorem 2.1 (Directional smoothing) .
Revised: Theorem 2.1 (Directional smoothing) .

Original: The operation y
Revised: The operation y

Original: =
Revised: =

Original: B av x is the directional average of x , in the sense that y u is the mean of x v , weighted by the direction and amplitude of F .
Revised: B av x is the directional average of x , in the sense that y u is the mean of x v , weighted by the direction and amplitude of F .

Original: The directional derivative matrix B dx is deﬁned in (6) and theorem 2.2, with the proof in appendix C.2.
Revised: The directional derivative matrix B dx is deﬁned in (6) and theorem 2.2, with the proof in appendix C.2.

Original: Again, the denominator is a local row-wise normalization but can be replaced by a global normalization.
Revised: Again, the denominator is a local row-wise normalization but can be replaced by a global normalization.

Original: diag ( a ) is a square, diagonal matrix with diagonal entries given by a .
Revised: diag ( a ) is a square, diagonal matrix with diagonal entries given by a .

Original: The aggregator works by subtracting the projected forward message by the backward message (similar to a center derivative), with an additional diagonal term to balance both directions.
Revised: The aggregator works by subtracting the projected forward message by the backward message (similar to a center derivative), with an additional diagonal term to balance both directions.

Original: [Equation]
Revised: [Equation]

Original: Theorem 2.2 (Directional derivative) .
Revised: Theorem 2.2 (Directional derivative) .

Original: Suppose ˆ F have rows of unit L 1 norm.
Revised: Suppose ˆ F have rows of unit L 1 norm.

Original: The operation y =
Revised: The operation y =

Original: B dx ( ˆ F ) x is the centered directional derivative of x in the direction of F , in the sense of equation 4, i.e.
Revised: B dx ( ˆ F ) x is the centered directional derivative of x in the direction of F , in the sense of equation 4, i.e.

Original: [Equation]
Revised: [Equation]

Original: These aggregators are directional, interpretable and complementary, making them ideal choices for
Revised: These aggregators are directional, interpretable and complementary, making them ideal choices for

Original: GNNs.
Revised: GNNs.

Original: We discuss the choice of aggregators in more details in appendix A, while also providing alternative aggregation matrices such as the center-balanced smoothing, the forward-copy, the phantom zero-padding, and the hardening of the aggregators using softmax/argmax on the ﬁeld.
Revised: We discuss the choice of aggregators in more details in appendix A, while also providing alternative aggregation matrices such as the center-balanced smoothing, the forward-copy, the phantom zero-padding, and the hardening of the aggregators using softmax/argmax on the ﬁeld.

Original: We further provide a visual interpretation of the B av and B dx aggregators in ﬁgure 2.
Revised: We further provide a visual interpretation of the B av and B dx aggregators in ﬁgure 3.

Original: Interestingly, we also note in appendix A.1 that B av and B dx yield respectively the mean and Laplacian aggregations when F is a vector ﬁeld such that all entries are constant F ij = ± C . 
Revised: Interestingly, we also note in appendix A.1 that B av and B dx yield respectively the mean and Laplacian aggregations when F is a vector ﬁeld such that all entries are constant F ij = ± C . [Table] [Equation] [Table]

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: 2. G RADIENT OF THE EIGENVECTORS AS INTERPRETABLE VECTOR FIELDS
Revised: 2. G RADIENT OF THE EIGENVECTORS AS INTERPRETABLE VECTOR FIELDS

Original: In this section we give theoretical support for the choice of gradients of the eigenfunctions of the
Revised: In this section we give theoretical support for the choice of gradients of the eigenfunctions of the

Original: Laplacian as sensible vectors along which to do directional message passing since they are interpretable and allow to reduce the over-smoothing.
Revised: Laplacian as sensible vectors along which to do directional message passing since they are interpretable and allow to reduce the over-smoothing.

Original: As usual the combinatorial, degree-normalized and symmetric normalized Laplacian are deﬁned as
Revised: As usual the combinatorial, degree-normalized and symmetric normalized Laplacian are deﬁned as

Original: [Equation]
Revised: [Equation]

Original: The problems of over-smoothing and over-squashing are critical issues in GNNs (Alon & Yahav, 2020; Hamilton, 2020).
Revised: The problems of over-smoothing and over-squashing are critical issues in GNNs (Alon & Yahav, 2020; Hamilton, 2020).

Original: In most GNN models, node representations become over-smoothed after several rounds of message passing (i.e., convolutions), as the representations tend to reach a meanﬁeld equilibrium equivalent to the stationary distribution of a random walk (Hamilton, 2020).
Revised: In most GNN models, node representations become over-smoothed after several rounds of message passing (i.e., convolutions), as the representations tend to reach a meanﬁeld equilibrium equivalent to the stationary distribution of a random walk (Hamilton, 2020).

Original: Oversmoothing is also related to the problem of over-squashing, which reﬂects the inability for GNNs to propagate informative signals between distant nodes (Alon & Yahav, 2020) and is a major bottleneck to training deep GNN models (Xu et al., 2019).
Revised: Oversmoothing is also related to the problem of over-squashing, which reﬂects the inability for GNNs to propagate informative signals between distant nodes (Alon & Yahav, 2020) and is a major bottleneck to training deep GNN models (Xu et al., 2019).

Original: Both problems are related to the fact that the inﬂuence of one node’s input on the ﬁnal representation of another node in a GNN is determined by the likelihood of the two nodes co-occurring on a truncated random walk (Xu et al., 2018b).
Revised: Both problems are related to the fact that the inﬂuence of one node’s input on the ﬁnal representation of another node in a GNN is determined by the likelihood of the two nodes co-occurring on a truncated random walk (Xu et al., 2018b).

Original: We show in theorem 2.3 (proved in appendix C.3) that by passing information in the direction of φ 1 , the eigenvector associated to the lowest non-trivial frequency of L norm , DGNs can efﬁciently share information between the farthest nodes of the graph, when using the K-walk distance to measure the difﬁculty of passing information.
Revised: We show in theorem 2.3 (proved in appendix C.3) that by passing information in the direction of φ 1 , the eigenvector associated to the lowest non-trivial frequency of L norm , DGNs can efﬁciently share information between the farthest nodes of the graph, when using the K-walk distance to measure the difﬁculty of passing information.

Original: Thus, DGNs provide a natural way to address both the oversmoothing and over-squashing problems: they can efﬁciently propagate messages between distant nodes and in a direction that counteracts over-smoothing.
Revised: Thus, DGNs provide a natural way to address both the oversmoothing and over-squashing problems: they can efﬁciently propagate messages between distant nodes and in a direction that counteracts over-smoothing.

Original: Deﬁnition 2 (K-walk distance) .
Revised: Deﬁnition 2 (K-walk distance) .

Original: The K-walk distance d K ( v i , v j ) on a graph is the average number of times v i is hit in a K step random walk starting from v j .
Revised: The K-walk distance d K ( v i , v j ) on a graph is the average number of times v i is hit in a K step random walk starting from v j .

Original: Theorem 2.3 (K-Gradient of the low frequency eigenvectors) .
Revised: Theorem 2.3 (K-Gradient of the low-frequency eigenvectors) .

Original: Let λ
Revised: Let λ

Original: i and φ i be the eigenvalues and eigenvectors of the normalized Laplacian of a connected graph L norm and let a, b  be the nodes that have a farthest K-walk distance.
Revised: i and φ i be the eigenvalues and eigenvectors of the normalized Laplacian of a connected graph L norm and let a, b = arg max 1 ≤ i,j ≤ n { d K ( v i , v j ) } be the nodes that have highest K-walk distance.

Original: Suppose that λ 2 (cid:29) λ 1 , then the optima of φ 1 approximates a, b  .
Revised: Let m = arg min 1 ≤ i ≤ n ( φ 1 ) i and M = arg max 1 ≤ i ≤ n ( φ 1 ) i , then d K ( v m , v M ) − d K ( v a , v b ) has order O (1 − λ 2 ) .

Original: As another point of view into the problem of oversmoothing, consider the hitting time Q ( x, y ) deﬁned as the expected number of steps in a random walk starting from node x ending in node y with the probability transition P ( x, y )
Revised: As another point of view into the problem of oversmoothing, consider the hitting time Q ( x, y ) deﬁned as the expected number of steps in a random walk starting from node x ending in node y with the probability transition P ( x, y )

Original: =
Revised: =

Original: 1 d x .
Revised: 1 d x .

Original: In appendix C.4 we give an informal argument supporting the following conjecture.
Revised: In appendix C.4 we give an informal argument supporting the following conjecture.

Revised: Deﬁnition 3 (Gradient step) .

Revised: Suppose the two neighboring nodes x and z are such that φ ( z ) − φ ( x ) is maximal among the neighbors of x , then we will say z is obtained from x by taking a step in the direction of the gradient ∇ φ .

Original: Conjecture 2.4 (Gradient steps reduce expected hitting time) .
Revised: Conjecture 2.4 (Gradient steps reduce expected hitting time) .

Original: Suppose that x, y are uniformly distributed random nodes such that φ i ( x ) <
Revised: Suppose that x, y are uniformly distributed random nodes such that φ i ( x ) <

Original: φ i ( y ) .
Revised: φ i ( y ) .

Original: Let z be the node obtained from x by taking one step in the direction of ∇ φ i , then the expected hitting time is decreased proportionally to λ − 1 i and
Revised: Let z be the node obtained from x by taking one step in the direction of ∇ φ i , then the expected hitting time is decreased proportionally to λ − 1 i and

Original: [Equation]
Revised: [Equation]

Original: These results from theorem 2.3 and conjecture 2.4 have the following immediate corollaries.
Revised: The next two corollaries follow from theorem 2.3 (and also conjecture 2.4 if it is true).

Original: Corollary 2.5 (Reduces over-squashing) .
Revised: Corollary 2.5 (Reduces over-squashing) .

Original: Following the direction of ∇ φ 1 is an efﬁcient way of passing information between the farthest nodes of the graph (in terms of the K-walk distance).
Revised: Following the direction of ∇ φ 1 is an efﬁcient way of passing information between the farthest nodes of the graph (in terms of the K-walk distance).

Original: Corollary 2.6 (Reduces over-smoothing) .
Revised: Corollary 2.6 (Reduces over-smoothing) .

Original: Following the direction of ∇ φ 1 allows the inﬂuence distribution between node representations to be decorrelated from random-walk hitting times (assuming the deﬁnition of inﬂuence introduced in Xu et al.
Revised: Following the direction of ∇ φ 1 allows the inﬂuence distribution between node representations to be decorrelated from random-walk hitting times (assuming the deﬁnition of inﬂuence introduced in Xu et al.

Original: (2018b)).
Revised: (2018b)).

Original: Our method also aligns perfectly with a recent proof that multiple independent aggregators are needed to distinguish neighbourhoods of nodes with continuous features (Corso et al., 2020).
Revised: Our method also aligns perfectly with a recent proof that multiple independent aggregators are needed to distinguish neighbourhoods of nodes with continuous features (Corso et al., 2020).

Original: When using eigenvectors of the Laplacian φ i to deﬁne directions in a graph, we need to keep in mind that there is never a single eigenvector associated to an eigenvalue, but a whole eigenspace.
Revised: When using eigenvectors of the Laplacian φ i to deﬁne directions in a graph, we need to keep in mind that there is never a single eigenvector associated to an eigenvalue, but a whole eigenspace.

Original: For instance, a pair of eigenvalues can have a multiplicity of 2 meaning that they can be generated by different pairs of orthogonal eigenvectors.
Revised: For instance, a pair of eigenvalues can have a multiplicity of 2 meaning that they can be generated by different pairs of orthogonal eigenvectors.

Original: For an eigenvalue of multiplicity 1, there are alwaysnorm 1 eigenvectors of opposite sign, which poses a problem during the directional aggregation.
Revised: For an eigenvalue of multiplicity 1, there are always two unit norm eigenvectors of opposite sign, which poses a problem during the directional aggregation.

Original: We can make a choice of sign and later take the absolute value (i.e. B av in equation 5).
Revised: We can make a choice of sign and later take the absolute value (i.e. B av in equation 5).

Original: An alternative is to take a sample of orthonormal basis of the eigenspace and use each choice to augment the training (see section 2.7).
Revised: An alternative is to take a sample of orthonormal basis of the eigenspace and use each choice to augment the training (see section 2.8).

Original: Note that although eigenvalues with multiplicity higher than one do happen (e.g. square grids have a multiplicity 2 for λ 1 ) this is not common for “real-world graphs”; we found no λ 1 multiplicity greater than 1 on the ZINC and PATTERN datasets.
Revised: Although multiplicities higher than one do happen for low-frequencies (square grids have a multiplicity 2 for λ 1 ) this is not common in “real-world graphs”; we found no λ 1 multiplicity greater than 1 on the ZINC and PATTERN datasets (see appendix B.4).

Original: Further, although all φ are orthogonal, their gradients, used to deﬁne directions, are not always locally orthogonal (e.g. there are many horizontal ﬂows in the grid).
Revised: Further, although all φ are orthogonal, their gradients, used to deﬁne directions, are not always locally orthogonal (e.g. there are many horizontal ﬂows in the grid).

Original: This last concern is left to be addressed in future work.
Revised: This last concern is left to be addressed in future work.

Original: 2. G ENERALIZATION OF THE CONVOLUTION ON A GRID
Revised: 2. G ENERALIZATION OF THE CONVOLUTION ON A GRID

Original: In this section we show that our method generalizes CNNs by allowing to deﬁne any radius- R convolutional kernels in grid-shaped graphs.
Revised: In this section we show that our method generalizes CNNs by allowing to deﬁne any radius- R convolutional kernels in grid-shaped graphs.

Original: The radius- R kernel at node u is a convolutional kernel that takes the weighted sum of all nodes v at a distance d ( u, v ) ≤ R .
Revised: The radius- R kernel at node u is a convolutional kernel that takes the weighted sum of all nodes v at a distance d ( u, v ) ≤ R .

Original: Consider the lattice graph Γ of size N 1 × N 2 × ...
Revised: Consider the lattice graph Γ of size N 1 × N 2 × ...

Original: × N n where each vertices are connected to their direct non-diagonal neighbour.
Revised: × N n where each vertices are connected to their direct non-diagonal neighbour.

Original: We know from Lemma C.1 that, for each dimension, there is an eigenvector that is only a function of this speciﬁc dimension.
Revised: We know from Lemma C.1 that, for each dimension, there is an eigenvector that is only a function of this speciﬁc dimension.

Original: For example, the lowest frequency eigenvector φ 1 always ﬂows in the direction of the longest length.
Revised: For example, the lowest frequency eigenvector φ 1 always ﬂows in the direction of the longest length.

Original: Hence, the Laplacian eigenvectors of the grid can play a role analogous to the axes in Euclidean space, as shown in ﬁgure 1.
Revised: Hence, the Laplacian eigenvectors of the grid can play a role analogous to the axes in Euclidean space, as shown in ﬁgure 2.

Original: With this knowledge, we show in theorem 2.7 (proven in C.7), that we can generalize all convolutional kernels in an n-dimensional grid.
Revised: With this knowledge, we show in theorem 2.7 (proven in C.7), that we can generalize all convolutional kernels in an n-dimensional grid.

Original: This is a strong result since it demonstrates that our DGN framework generalizes CNNs when applied on a grid, thus closing the gap between GNNs and the highly successful CNNs on image tasks.
Revised: This is a strong result since it demonstrates that our DGN framework generalizes CNNs when applied on a grid, thus closing the gap between GNNs and the highly successful CNNs on image tasks.

Original: Theorem 2.7 (Generalization radius-
Revised: Theorem 2.7 (Generalization radius-

Original: R convolutional kernel in a lattice) .
Revised: R convolutional kernel in a lattice) .

Original: For an n -dimensional lattice, any convolutional kernel of radius R can be realized by a linear combination of directional aggregation matrices and their compositions.
Revised: For an n -dimensional lattice, any convolutional kernel of radius R can be realized by a linear combination of directional aggregation matrices and their compositions.

Original: As an example, ﬁgure 2.5 shows how a linear combination of the ﬁrst and m -th aggregators
Revised: As an example, ﬁgure 4 shows how a linear combination of the ﬁrst and m -th aggregators

Original: B ( ∇ φ 1 ,m ) realize a kernel on an N × M grid, where m = (cid:100) N/M (cid:101) and N > M .[Figure] Graph
Revised: B ( ∇ φ 1 ,m ) realize a kernel on an N × M grid, where m = (cid:100) N/M (cid:101) and N > M .

Original: [Figure]
Revised: [Figure]

Original: 1𝒚 = 2 𝑩 𝑎𝑣 1 𝒙 1𝒚 = 2 𝑩 𝑑𝑥 1 𝒙  𝒚 = 2 𝑩 𝑎𝑣𝑚 𝒙𝒚 = 2 𝑩 𝑑𝑥𝑚 𝒙 𝑤 1 𝑤 2 + 𝑤𝑤 4 + 𝑤𝑤 4 −𝑤𝑤 2 −𝑤

Original: 2. E XTENDING THE RADIUS OF THE AGGREGATION KERNEL
Revised: 2. E XTENDING THE RADIUS OF THE AGGREGATION KERNEL

Original: Having aggregation kernels for neighbours of distance 2 or 3 is important to improve the expressiveness of GNNs, their ability to understand patterns, and to reduce the number of layers required.
Revised: Having aggregation kernels for neighbours of distance 2 or 3 is important to improve the expressiveness of GNNs, their ability to understand patterns, and to reduce the number of layers required.

Original: However, the lack of directions in GNNs strongly limits the radius of the kernels since, given a graph of regular degree d , a mean/sum aggregation at a radius- R will result in a heavy over-squashing of d R messages.
Revised: However, the lack of directions in GNNs strongly limits the radius of the kernels since, given a graph of regular degree d , a mean/sum aggregation at a radius- R will result in a heavy over-squashing of d R messages.

Original: Using the directional ﬁelds, we can enumerate different paths, thus assigning a different weight for different R -distant neighbours.
Revised: Using the directional ﬁelds, we can enumerate different paths, thus assigning a different weight for different R -distant neighbours.

Original: This method, proposed in appendix A.7, avoids the over-squashing, but empirical results are left for future work.
Revised: This method, proposed in appendix A.7, avoids the over-squashing, but empirical results are left for future work.

Revised: 2. C OMPARISON WITH W EISFEILER -L EHMAN (WL) TEST

Revised: We also compare the expressiveness of the Directional Graph Networks with the classical WL graph isomorphism test which is often used to classify the expressivity of graph neural networks (Xu et al., 2018a).

Revised: In theorem 2.8 (proven in appendix C.8) we prove that DGNs are capable of distinguishing pairs of graphs that the 1-WL test (and so ordinary GNNs) cannot differentiate.

Revised: Theorem 2.8 (Comparison with 1-WL test) .

Revised: DGNs using the mean aggregator, any directional aggregator of the ﬁrst eigenvector and injective degree-scalers are strictly more powerful than the 1-WL test.

Original: 2. D ATA AUGMENTATION
Revised: 2. D ATA AUGMENTATION

Original: Another important result is that the directions in the graph allow to replicate some of the most common data augmentation techniques used in computer vision, namely reﬂection, rotation and distortion.
Revised: Another important result is that the directions in the graph allow to replicate some of the most common data augmentation techniques used in computer vision, namely reﬂection, rotation and distortion.

Original: The main difference is that, instead of modifying the image (such as a 5 ◦ rotation), the proposed transformation is applied on the vector ﬁeld deﬁning the aggregation kernel (thus rotating the kernel by − 5 ◦ without changing the image).
Revised: The main difference is that, instead of modifying the image (such as a 5 ◦ rotation), the proposed transformation is applied on the vector ﬁeld deﬁning the aggregation kernel (thus rotating the kernel by − 5 ◦ without changing the image).

Original: This offers the advantage of avoiding to pre-process the data since the augmentation is done directly on the kernel at each iteration of the training.
Revised: This offers the advantage of avoiding to pre-process the data since the augmentation is done directly on the kernel at each iteration of the training.

Original: The simplest augmentation is the vector ﬁeld ﬂipping, which is done changing the sign of the ﬁeld
Revised: The simplest augmentation is the vector ﬁeld ﬂipping, which is done changing the sign of the ﬁeld

Original: F , as stated in deﬁnition 3.
Revised: F , as stated in deﬁnition 4.

Original: This changes the sign of B dx , but leaves B av unchanged.
Revised: This changes the sign of B dx , but leaves B av unchanged.

Original: Deﬁnition 3 (Reﬂection of the vector ﬁeld) .
Revised: Deﬁnition 4 (Reﬂection of the vector ﬁeld) .

Original: For a vector ﬁeld F , the reﬂected ﬁeld is − F .
Revised: For a vector ﬁeld F , the reﬂected ﬁeld is − F .

Original: Let F 1 , F 2 be vector ﬁeld in a graph. Let α be the angle between the vector ﬁelds such that (cid:104) F 1 , F 2 (cid:105)
Revised: Let F 1 , F 2 be vector ﬁelds in a graph, with ˆ F 1 and ˆ F 2 being the ﬁeld normalized such that each row has a unitary L 2 -norm. Deﬁne the angle vector α by (cid:104)

Original: = (cid:107) F 1 (cid:107)(cid:107) F 2 (cid:107) cos( α  ) .
Revised: ( ˆ F 1 ) i, : , ( ˆ F 2 ) i, : (cid:105) = cos( α i ) .

Original: The vector ﬁeld  F ⊥ 2 is the component of  F 2 perpendicular to  F 1 :
Revised: The vector ﬁeld ˆ F ⊥ 2 is the component of ˆ F 2 perpendicular to ˆ F 1 . The equation below deﬁnes ˆ F ⊥ 2 . The next equation deﬁnes the angle

Original: [Equation]
Revised: [Equation]

Revised: Notice that we then have the decomposition ( ˆ F 2 ) i, : = cos( α i )( ˆ F 1 ) i, : + sin( α i )( ˆ F ⊥ 2 ) i, : .

Original: Deﬁnition 4 (Rotation of the vector ﬁelds) .
Revised: Deﬁnition 5 (Rotation of the vector ﬁelds) .

Original: For  F 1 and  F 2 non-colinear vector ﬁelds, their rotation by the angle θ in the plane formed by {  F 1 ,  F 2 } is {+ˆ F θ 
 = ˆ F 1 diag (cos θ )+ ˆ F ⊥ 2 diag (sin θ ) ,+}
Revised: For ˆ F 1 and ˆ F 2 non-colinear vector ﬁelds with each vector of unitary length, their rotation by the angle θ in the plane formed by { ˆ F 1 , ˆ F 2 } is {+ˆ F θ 
 = ˆ F 1 diag (cos θ )+ ˆ F ⊥ 2 diag (sin θ ) ,+}

Original: [Equation]
Revised: [Equation]

Original: Finally, the following augmentation has  similar effect to a wave distortion applied on images.
Revised: Finally, the following augmentation has a similar effect to a wave distortion applied on images.

Original: Deﬁnition 5 (Random distortion of the vector ﬁeld) .
Revised: Deﬁnition 6 (Random distortion of the vector ﬁeld) .

Original: For vector ﬁeld F and anti-symmetric random noise matrix R , its randomly distorted ﬁeld is F (cid:48)
Revised: For vector ﬁeld F and anti-symmetric random noise matrix R , its randomly distorted ﬁeld is F (cid:48)

Original: =
Revised: =

Original: F + R ◦ A .
Revised: F + R ◦ A .

Original:  I MPLEMENTATION
Revised:  I MPLEMENTATION

Original: We implemented the models using the DGL and PyTorch libraries and we provide the code at the address https://anonymous.4open.science/r/a752e2b1-22e3-40ce-851c-a564073e1fca/. We test our method on standard benchmarks from (Dwivedi et al., 2020), namely ZINC, CIFAR10 and PATTERN, with more details on the datasets and how we enforce a fair comparison in appendix B.1.
Revised: We implemented the models using the DGL and PyTorch libraries and we provide the code at the address https://anonymous.4open.science/r/a752e2b1-22e3-40ce-851c-a564073e1fca/. We test our method on standard benchmarks from Dwivedi et al. (2020) and Hu et al. (2020), namely ZINC, CIFAR10, PATTERN and MolHIV with more details on the datasets and how we enforce a fair comparison in appendix B.1.

Original: For the empirical experiments we inserted our proposed aggregation method in two different type of message passing architecture used in the literature: a simple one similar to the one present in GCN (equation 10a) (Kipf & Welling, 2016) and a more complex and general one typical of MPNN (10b) (Gilmer et al., 2017) with or without edge features e ji .
Revised: For the empirical experiments we inserted our proposed aggregation method in two different type of message passing architecture used in the literature: a simple one similar to the one present in GCN (equation 9a) (Kipf & Welling, 2016) and a more complex and general one typical of MPNN (9b) (Gilmer et al., 2017) with or without edge features e ji .

Original: Hence, the time complexity O
Revised: Hence, the time complexity O

Original: ( Em ) is identical to the PNA (Corso et al., 2020), where E is the number of edges and m the number of aggregators, with an additional O ( Ek ) to pre-compute the k -ﬁrst eigenvectors, as explained in the appendix B.2.
Revised: ( Em ) is identical to the PNA (Corso et al., 2020), where E is the number of edges and m the number of aggregators, with an additional O ( Ek ) to pre-compute the k -ﬁrst eigenvectors, as explained in the appendix B.2.

Original: [Equation]
Revised: [Equation]

Original: where (cid:76) is an operator which concatenates the results of multiple aggregators, X is the node features, M is a linear transformation and U a multiple layer perceptron.
Revised: where (cid:76) is an operator which concatenates the results of multiple aggregators, X is the node features, M is a linear transformation and U a multiple layer perceptron.

Original: We tested the directional aggregators across the datasets using the gradient of the ﬁrst k eigenvectors ∇ φ 1 ,...,k as the underlying vector ﬁelds.
Revised: We tested the directional aggregators across the datasets using the gradient of the ﬁrst k eigenvectors ∇ φ 1 ,...,k as the underlying vector ﬁelds.

Revised: Here, k is a hyperparameter, usually 1 or 2, but could be bigger for high-dimensional graphs.

Original: To deal with the arbitrary sign of the eigenvectors, we take the absolute value of the result of equation 6, making it invariant to a reﬂection of the ﬁeld.
Revised: To deal with the arbitrary sign of the eigenvectors, we take the absolute value of the result of equation 6, making it invariant to a reﬂection of the ﬁeld.

Original: In case of a disconnected graph, φ i is the i -th eigenvector of each connected component.
Revised: In case of a disconnected graph, φ i is the i -th eigenvector of each connected component.

Original: Despite the numerous aggregators proposed in appendix A, only B dx and B av are tested empirically.
Revised: Despite the numerous aggregators proposed in appendix A, only B dx and B av are tested empirically [Table]

Original:  R ESULTS AND DISCUSSION
Revised:  R ESULTS AND DISCUSSION

Original: Directional aggregation Using the benchmarks introduced in section 3, we present in ﬁgure 4 a fair comparison of various aggregation strategies using the same parameter budget and hyperparameters.
Revised: Directional aggregation Using the benchmarks introduced in section 3, we present in ﬁgure 5 a fair comparison of various aggregation strategies using the same parameter budget and hyperparameters.

Original: We see a consistent boost in the performance for simple , complex and complex with edges models using directional aggregators compared to the mean-aggregator baseline.
Revised: We see a consistent boost in the performance for simple , complex and complex with edges models using directional aggregators compared to the mean-aggregator baseline

Original: For brevity, we denote dx i and av i as the directional derivative B idx and smoothing B iav aggregators.

Original: [Table]
Revised: [Table]

Original: Simple

Original: Complex

Original: Simple

Original: Complex MAE

Original: MAE

Original: MAE %

Original: acc %

Original: acc %

Original: acc % acc mean 0.

Original: 62.8 mean dx0.

Original: 83.44 mean dx 1 dx0.

Original: 69.8 mean av0.

Original: 83.21 mean av 1 av0.

Original: [Figure]
Revised: [Figure]

Original: dx 1 av0. Best [Figure] In particular, we see a signiﬁcant improvement on ZINC  using the derivative aggregator.
Revised: In particular, we see a signiﬁcant improvement in ZINC and MolHIV using the directional aggregators.

Original: We believe this is due to the capacity to move efﬁciently messages across opposite parts of the molecule and to better understand the role of atom pairs.
Revised: We believe this is due to the capacity to move efﬁciently messages across opposite parts of the molecule and to better understand the role of atom pairs.

Original: Further, the thesis that DGNs can bridge the gap between CNNs and GNNs is supported by the clear improvements on CIFAR10 over the baselines.
Revised: Further, the thesis that DGNs can bridge the gap between CNNs and GNNs is supported by the clear improvements on CIFAR10 over the baselines.

Original:  With our theoretical analysis in mind, we expected to perform well on PATTERN since the ﬂow of the ﬁrst eigenvectors are meaningful directions in a stochastic block model and passing messages using those directions allows the network to efﬁciently detect the two communities.
Revised: This contrasts with the positional encoding which showed no clear improvement. With our theoretical analysis in mind, we expected to perform well on PATTERN since the ﬂow of the ﬁrst eigenvectors are meaningful directions in a stochastic block model and passing messages using those directions allows the network to efﬁciently detect the two communities.

Original: The results match our expectations, outperforming all the previous models.
Revised: The results match our expectations, outperforming all the previous models.

Original: Comparison to the literature In order to compare our model with the literature, we ﬁne tuned it on the various datasets and we report its performance in ﬁgure 5.
Revised: Comparison to the literature In order to compare our model with the literature, we ﬁne-tuned it on the various datasets and we report its performance in ﬁgure 6.

Original: We observe that DGN provides signiﬁcant improvement across all benchmarks, highlighting the importance of anisotropic kernels.
Revised: We observe that DGN provides signiﬁcant improvement across all benchmarks, highlighting the importance of anisotropic kernels.

Original: In the work by Dwivedi et al.
Revised: In the work by Dwivedi et al.

Original: , they proposed the use of positional encoding of the eigenvectors in node features, but these bring signiﬁcant improvement when many eigenvectors and high network depths are used.
Revised: , they proposed the use of positional encoding of the eigenvectors in node features, but these bring signiﬁcant improvement when many eigenvectors and high network depths are used.

Original: Our results outperform them with fewer parameters, less depth, and only 1-2 eigenvectors, further motivating their use as directional ﬂows instead of positional encoding.
Revised: Our results outperform them with fewer parameters, less depth, and only 1-2 eigenvectors, further motivating their use as directional ﬂows instead of positional encoding.

Original: Data augmentation To evaluate the effectiveness of the proposed augmentation, we trained the models on a reduced version of the CIFAR10 dataset.
Revised: Data augmentation To evaluate the effectiveness of the proposed augmentation, we trained the models on a reduced version of the CIFAR10 dataset.

Original: The results in ﬁgure 6 show clearly a higher expressive power of the dx aggregator, enabling it to ﬁt well the training data.
Revised: The results in ﬁgure 7 show clearly a higher expressive power of the dx aggregator, enabling it to ﬁt well the training data.

Original: For a small dataset, this 
Revised: For a small dataset, this comes at the cost of overﬁtting and a reduced test-set performance, but we observe that randomly rotating or distorting the kernels counteracts the overﬁtting and improves the generalization. As expected, the performance decreases when the rotation or distortion is too high since the augmented graph changes too much. In computer vision images similar to CIFAR10 are usually rotated by less than 30 ◦ (Shorten & Khoshgoftaar; O’Gara & McGuinness, 2019). Further, due to the constant number of parameters across models, less parameters are attributed to the mean aggregation in

Original: [Figure]
Revised: [Figure]

Original: , dx 2 , max } .comes at the cost of overﬁtting and reduced performance on the test set.
Revised: those with * who use 300 k to 1 .

Original: However, we observe that randomly rotating the kernels counteract the overﬁtting and allows the model to  better generalize.
Revised: 9 M . In ZINC the DGN aggregators are { mean, dx 1 , max, min } , in PATTERN { mean, dx 1 , av 1 } , in CIFAR10 { mean, dx 1 , dx 2 , max } , in MolHIV { mean, dx 1 , av 1 , max, min } . the directional models, thus it cannot ﬁt well the data when the rotation/distortion is too strong since the directions are less informative. We expect large models to perform better at high angles.

Original: [Figure]
Revised: [Figure]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: The proposed DGN method allows to solve many problems of GNNs, including the lack of anisotropy, the low expressiveness, the over-smoothing and over-squashing.
Revised: The proposed DGN method allows to solve many problems of GNNs, including the lack of anisotropy, the low expressiveness, the over-smoothing and over-squashing.

Original: For the ﬁrst time in graph networks, we generalize the directional properties of CNNs and their data augmentation capabilities.
Revised: For the ﬁrst time in graph networks, we generalize the directional properties of CNNs and their data augmentation capabilities.

Original: Based on an intuitive idea and backed by a set of strong theoretical and empirical results, we believe this work will give rise to a new family of directional GNNs.
Revised: Based on an intuitive idea and backed by a set of strong theoretical and empirical results, we believe this work will give rise to a new family of directional GNNs.

Original: Future work can focus on the implementation of radius- R kernels and improving the choice of multiple orthogonal directions.
Revised: Future work can focus on the implementation of radius- R kernels and improving the choice of multiple orthogonal directions.

Original: Broader Impact This work will extend the usability of graph networks to all problems with physically deﬁned directions, thus making GNN a new laboratory for physics, material science and biology.
Revised: Broader Impact This work will extend the usability of graph networks to all problems with physically deﬁned directions, thus making GNN a new laboratory for physics, material science and biology.

Original: In fact, the anisotropy present in a wide variety of systems could be expressed as vector ﬁeld (spinor, tensor) compatible with the DGN framework, without the need of eigenvectors.
Revised: In fact, the anisotropy present in a wide variety of systems could be expressed as vector ﬁelds (spinor, tensor) compatible with the DGN framework, without the need of eigenvectors.

Original: One example is magnetic anisotropicity in metals, alloys and also in molecules such as benzene ring, alkene, carbonyl, alkyne that are easier or harder to magnetise depending on the directions or which way the object is rotated.
Revised: One example is magnetic anisotropicity in metals, alloys and also in molecules such as benzene ring, alkene, carbonyl, alkyne that are easier or harder to magnetise depending on the directions or which way the object is rotated.

Original: Other examples are the response of material to high electromagnetic ﬁelds (e.g. to study material responses at terahertz frequency); all kind of ﬁeld propagation in crystals lattices (vibrations, heat, shear and frictional force, young modulus, light refraction, birefringence); multi-body or liquid motion; trafﬁc modeling; and design of novel materials and constrained structures.
Revised: Other examples are the response of materials to high electromagnetic ﬁelds (e.g. to study material responses at terahertz frequency); all kind of ﬁeld propagation in crystals lattices (vibrations, heat, shear and frictional force, young modulus, light refraction, birefringence); multi-body or liquid motion; trafﬁc modelling; and design of novel materials and constrained structures.

Original: This also enables GNNs to be used for virtual prototyping systems since the added directional constraints could improve the analysis of a product’s functionality, manufacturing and behavior.
Revised: This also enables GNNs to be used for virtual prototyping systems since the added directional constraints could improve the analysis of a product’s functionality, manufacturing and behavior.

Original: A A PPENDIX - C HOICES OF DIRECTIONAL AGGREGATORS
Revised: A A PPENDIX - C HOICES OF DIRECTIONAL AGGREGATORS

Original: This appendix helps understand the choice of B av and B dx in section 2.3 and presents different directional aggregators that can be used as an alternative to the ones proposed.
Revised: This appendix helps understand the choice of B av and B dx in section 2.3 and presents different directional aggregators that can be used as an alternative to the ones proposed.

Original: A simple alternative to the directional smoothing and directional derivative operator is to simply take the forward/backward values according to the underlying positive/negative parts of the ﬁeld
Revised: A simple alternative to the directional smoothing and directional derivative operator is to simply take the forward/backward values according to the underlying positive/negative parts of the ﬁeld

Original: F , since it can effectively replicate them.
Revised: F , since it can effectively replicate them.

Original: However, there are many advantage of using B av,dx .
Revised: However, there are many advantage of using B av,dx .

Original: First, one can decide to use either of them and still have an interpretable aggregation with half the parameters.
Revised: First, one can decide to use either of them and still have an interpretable aggregation with half the parameters.

Original: Then, we also notice that B av,dx regularize the parameter by forcing the network to take both forward and backward neighbours into account at each time, and avoids one of the neighbours becoming too important.
Revised: Then, we also notice that B av,dx regularize the parameter by forcing the network to take both forward and backward neighbours into account at each time and avoids one of the neighbours becoming too important.

Original: Lastly, they are robust to a change of sign of the eigenvectors since B av is sign invariant and B dx will only change the sign of the results, which is not the case for forward/backward aggregations.
Revised: Lastly, they are robust to a change of sign of the eigenvectors since B av is sign invariant and B dx will only change the sign of the results, which is not the case for forward/backward aggregations.

Original: A. R ETRIEVING THE MEAN AND L APLACIAN AGGREGATIONS
Revised: A. R ETRIEVING THE MEAN AND L APLACIAN AGGREGATIONS

Original: It is interesting to note that we can recover simple aggregators from the aggregation matrices
Revised: It is interesting to note that we can recover simple aggregators from the aggregation matrices

Original: B av ( F ) and B dx ( F ) .
Revised: B av ( F ) and B dx ( F ) .

Original: Let F be a vector ﬁeld such that all edges are equally weighted F ij = ± C for all edges ( i, j ) .
Revised: Let F be a vector ﬁeld such that all edges are equally weighted F ij = ± C for all edges ( i, j ) .

Original: Then, the aggregator B av is equivalent to a mean aggregation:
Revised: Then, the aggregator B av is equivalent to a mean aggregation:

Original: [Equation]
Revised: [Equation]

Original: Under the condition F ij = C , the differential aggregator is equivalent to a Laplacian operator L normalized using the degree D
Revised: Under the condition F ij = C , the differential aggregator is equivalent to a Laplacian operator L normalized using the degree D

Original: [Equation]
Revised: [Equation]

Original: A. G LOBAL FIELD NORMALIZATION
Revised: A. G LOBAL FIELD NORMALIZATION

Original: The proposed aggregators are deﬁned with a row-wise normalized ﬁeld
Revised: The proposed aggregators are deﬁned with a row-wise normalized ﬁeld

Original: [Equation]
Revised: [Equation]

Original: meaning that all the vectors are of unit-norm and the aggregation/message passing is done only according to the direction of the vectors, not their amplitude.
Revised: meaning that all the vectors are of unit-norm and the aggregation/message passing is done only according to the direction of the vectors, not their amplitude.

Original: However, it is also possible to do a global normalization of the ﬁeld F by taking aa matrix-norm instead of a vector-norm.
Revised: However, it is also possible to do a global normalization of the ﬁeld F by taking a matrix-norm instead of a vector-norm.

Original: Doing so will modulate the aggregation by the amplitude of the ﬁeld at each node.
Revised: Doing so will modulate the aggregation by the amplitude of the ﬁeld at each node.

Original: One need to be careful since a global normalization might be very sensible to the number of nodes in the graph.
Revised: One needs to be careful since a global normalization might be very sensitive to the number of nodes in the graph.

Original: A. C ENTER - BALANCED AGGREGATORS
Revised: A. C ENTER - BALANCED AGGREGATORS

Original: A problem arises in the aggregators B dx and B av proposed in equations 5 and 6 when there is an imbalance between the positive and negative terms of F ± .
Revised: A problem arises in the aggregators B dx and B av proposed in equations 5 and 6 when there is an imbalance between the positive and negative terms of F ± .

Original: In that case, one of the directions overtakes the other in terms of associated weights.
Revised: In that case, one of the directions overtakes the other in terms of associated weights.

Original: An alternative is also to normalize the forward and backward directions separately, to avoid having either the backward or forward direction dominating the message.
Revised: An alternative is also to normalize the forward and backward directions separately, to avoid having either the backward or forward direction dominating the message.

Original: [Equation]
Revised: [Equation]

Original: The same idea can be applied to the derivative aggregator equation 12 where the positive and negative parts of the ﬁeld F ± are normalized separately to allow to project both the forward and backward messages into a vector ﬁeld of unit-norm.
Revised: The same idea can be applied to the derivative aggregator equation 11 where the positive and negative parts of the ﬁeld F ± are normalized separately to allow to project both the forward and backward messages into a vector ﬁeld of unit-norm.

Original: F + is the out-going ﬁeld at each node and is used for the forward direction, while F − is the in-going ﬁeld used for the backward direction.
Revised: F + is the out-going ﬁeld at each node and is used for the forward direction, while F − is the in-going ﬁeld used for the backward direction.

Original: By averaging the forward and backward derivatives, the proposed matrix B dx -center represents the centered derivative matrix.
Revised: By averaging the forward and backward derivatives, the proposed matrix B dx -center represents the centered derivative matrix.

Original: [Equation]
Revised: [Equation]

Original:  (cid:88)

Original: j

Original: F (cid:48) : ,j 

Original:  i, : ,

Original: F (cid:48) i, : = 1 

Original: F + i, : || F

Original: + i, : ||

Original: L 1 + (cid:15) (cid:124) (cid:123)(cid:122) (cid:125) forward ﬁeld +

Original: F − i, : || F − i, : ||

Original: L 1 + (cid:15) (cid:124) (cid:123)(cid:122) (cid:125) backward ﬁeld   (12)

Original: A. H ARDENING THE AGGREGATORS
Revised: A. H ARDENING THE AGGREGATORS

Original: The aggregation matrices that we proposed, mainly B dx and B av depend on a smooth vector ﬁeld
Revised: The aggregation matrices that we proposed, mainly B dx and B av depend on a smooth vector ﬁeld

Original: F .
Revised: F .

Original: At any given node, the aggregation will take a weighted sum of the neighbours in relation to the direction of F .
Revised: At any given node, the aggregation will take a weighted sum of the neighbours in relation to the direction of F .

Original: Hence, if the ﬁeld F v at a node v is diagonal in the sense that it gives a non-zero weight to many neighbours, then the aggregator will compute a weighted average of the neighbours.
Revised: Hence, if the ﬁeld F v at a node v is diagonal in the sense that it gives a non-zero weight to many neighbours, then the aggregator will compute a weighted average of the neighbours.

Original: Although there are clearly good reasons to have this weighted-average behaviour, it is not necessarily desired in every problem.
Revised: Although there are clearly good reasons to have this weighted-average behaviour, it is not necessarily desired in every problem.

Original: For example, if we want to move a single node across the graph, this behaviour will smooth the node at every step.
Revised: For example, if we want to move a single node across the graph, this behaviour will smooth the node at every step.

Original: Instead, we propose below to soften and harden the aggregations by forcing the ﬁeld into making a decision on the direction it takes.
Revised: Instead, we propose below to soften and harden the aggregations by forcing the ﬁeld into making a decision on the direction it takes.

Original: Soft hardening the aggregation is possible by using a softmax with a temperature T on each row to obtain the ﬁeld F softhard . 
Revised: Soft hardening the aggregation is possible by using a softmax with a temperature T on each row to obtain the ﬁeld F softhard . [Equation]

Original: [Equation]
Revised: [Equation]

Original: Hardening the aggregation is possible by using an inﬁnite temperature, which changes the softmax functions into argmax.
Revised: Hardening the aggregation is possible by using an inﬁnite temperature, which changes the softmax functions into argmax.

Original: In this speciﬁc case, the node with the highest component of the ﬁeld will be copied, while all other nodes will be ignored.
Revised: In this speciﬁc case, the node with the highest component of the ﬁeld will be copied, while all other nodes will be ignored.

Original: [Equation]
Revised: [Equation]

Original: An alternative to the aggregators above is to take the softmin/argmin of the negative part and the softmax/argmax of the positive part.
Revised: An alternative to the aggregators above is to take the softmin/argmin of the negative part and the softmax/argmax of the positive part.

Original: A. F ORWARD AND BACKWARD COPY
Revised: A. F ORWARD AND BACKWARD COPY

Original: The aggregation matrices B av and B dx have the nice property that if the ﬁeld is ﬂipped (change of sign), the aggregation gives the same result, except for the sign of B dx .
Revised: The aggregation matrices B av and B dx have the nice property that if the ﬁeld is ﬂipped (change of sign), the aggregation gives the same result, except for the sign of B dx .

Original: However, there are cases where we want to propagate information in the forward direction of the ﬁeld, without smoothing it with the backward direction.
Revised: However, there are cases where we want to propagate information in the forward direction of the ﬁeld, without smoothing it with the backward direction.

Original: In this case, we can deﬁne the strictly forward and strictly backward ﬁelds below, and use them directly with the aggregation matrices.
Revised: In this case, we can deﬁne the strictly forward and strictly backward ﬁelds below, and use them directly with the aggregation matrices.

Original: [Equation]
Revised: [Equation]

Original: Further, we can use the hardened ﬁelds in order to deﬁne a forward copy and backward copy, which will simply copy the node in the direction of the highest ﬁeld component.
Revised: Further, we can use the hardened ﬁelds in order to deﬁne a forward copy and backward copy, which will simply copy the node in the direction of the highest ﬁeld component.

Original: [Equation]
Revised: [Equation]

Original: A. P HANTOM ZERO - PADDING
Revised: A. P HANTOM ZERO - PADDING

Original: Some recent work in computer vision have shown the importance of zero-padding to improve CNNs by allowing the network to understand it’s position relative to the border (Islam et al., 2020).
Revised: Some recent work in computer vision has shown the importance of zero-padding to improve CNNs by allowing the network to understand it’s position relative to the border (Islam et al., 2020).

Original: In contrast, using boundary conditions or reﬂection padding makes the network completely blind to positional information.
Revised: In contrast, using boundary conditions or reﬂection padding makes the network completely blind to positional information.

Original: In this section, we show that we can mimic the zero-padding in the direction of the ﬁeld F for both aggregation matrices B av and B dx .
Revised: In this section, we show that we can mimic the zero-padding in the direction of the ﬁeld F for both aggregation matrices B av and B dx .

Original: Starting with the B av matrix, in the case of a missing neighbour in the forward/backward direction, the matrix will compensate by adding more weights to the other direction, due to the denominator which performs a normalization.
Revised: Starting with the B av matrix, in the case of a missing neighbour in the forward/backward direction, the matrix will compensate by adding more weights to the other direction, due to the denominator which performs a normalization.

Original: Instead, we would need the matrix to consider both directions separately so that a missing direction would result in zero padding.
Revised: Instead, we would need the matrix to consider both directions separately so that a missing direction would result in zero padding.

Original: Hence, we deﬁne B av, 0 pad below, where either the F + or F − will be 0 on a boundary with strictly in-going/out-going ﬁeld.
Revised: Hence, we deﬁne B av, 0 pad below, where either the F + or F − will be 0 on a boundary with strictly in-going/out-going ﬁeld.

Original: [Equation]
Revised: [Equation]

Original: Following the same argument

Original: [Equation]
Revised: [Equation]

Original:  we deﬁne B dx, 0 pad below, where either the forward or backward term are ignored.
Revised: Following the same argument, we deﬁne B dx, 0 pad below, where either the forward or backward term is ignored.

Original: The diagonal term is also removed at the boundary so that the result is a center derivative equal to the subtraction of  forward term with the 0-term on the back (or vice-versa), instead of a forward derivative. 
Revised: The diagonal term is also removed at the boundary so that the result is a center derivative equal to the subtraction of the forward term with the 0-term on the back (or vice-versa), instead of a forward derivative. [Equation] [Equation] , otherwise

Original: [Equation]
Revised: [Equation]

Original: A. E XTENDING THE RADIUS OF THE AGGREGATION KERNEL
Revised: A. E XTENDING THE RADIUS OF THE AGGREGATION KERNEL

Original: We aim at providing a general radius- R kernel B R that assigns different weights to different subsets of nodes n u at a distance R from the center node n v .
Revised: We aim at providing a general radius- R kernel B R that assigns different weights to different subsets of nodes n u at a distance R from the center node n v .

Original: First, we decompose the matrix B ( F ) into positive and negative parts B ± ( F ) representing the forward and backward steps aggregation in the ﬁeld F .
Revised: First, we decompose the matrix B ( F ) into positive and negative parts B ± ( F ) representing the forward and backward steps aggregation in the ﬁeld F .

Original: [Equation]
Revised: [Equation]

Original: Thus, deﬁning B ± fb ( F ) i, : =
Revised: Thus, deﬁning B ± fb ( F ) i, : =

Original: F ± i, : || F i, : ||
Revised: F ± i, : || F i, : ||

Original: Lp , we can ﬁnd different aggregation matrices by using different combinations of walks of radius R .
Revised: Lp , we can ﬁnd different aggregation matrices by using different combinations of walks of radius R .

Original: First demonstrated for a grid in theorem 2.7, we generalize it in equation 20 for any graph G .
Revised: First demonstrated for a grid in theorem 2.7, we generalize it in equation 19 for any graph G .

Original: Deﬁnition 6 (General radius R n-directional kernel) .
Revised: Deﬁnition 7 (General radius R n-directional kernel) .

Original: Let S n be the group of permutations over n elements with a set of directional ﬁelds F i .
Revised: Let S n be the group of permutations over n elements with a set of directional ﬁelds F i .

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: V

Revised: N (cid:89)

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] 
Revised: [Equation] [Table]

Revised: following

Revised: [Table]

Revised: steps V ,

Revised: [Table]

Revised: by

Original:  In this equation, n is the number of directional ﬁelds and R is the desired radius.
Revised: [Equation] n [Equation] In this equation, n is the number of directional ﬁelds and R is the desired radius.

Original: V represents all the choices of walk { v 1 , v 2 , ..., v n } in the direction of the ﬁelds { F 1 , F 2 , ..., F n } .
Revised: V represents all the choices of walk { v 1 , v 2 , ..., v n } in the direction of the ﬁelds { F 1 , F 2 , ..., F n } .

Original: For example,
Revised: For example,

Original: V = { 3 , 1 , 0 , − 2 } has a radius R = 6 , with 3 steps forward of F 1 , 1 step forward of F 2 , and 2 steps backward of F 4 .
Revised: V = { 3 , 1 , 0 , − 2 } has a radius R = 6 , with 3 steps forward of F 1 , 1 step forward of F 2 , and 2 steps backward of F 4 .

Original: The sign of each B ± fb is dependant to the sign of v σ ( j ) , and the power | v σ ( j ) | is the number of aggregation steps in the directional ﬁeld F σ ( j ) .
Revised: The sign of each B ± fb is dependant to the sign of v σ ( j ) , and the power | v σ ( j ) | is the number of aggregation steps in the directional ﬁeld F σ ( j ) .

Original: The full equation is thus the combination of all possible choices of paths across the set of ﬁelds F i , with all possible permutations.
Revised: The full equation is thus the combination of all possible choices of paths across the set of ﬁelds F i , with all possible permutations.

Original: Note that we are restricting the sum to v i having only a possible sign; although matrices don’t commute, we avoid choosing different signs since it will likely self-intersect a lower radius walk.
Revised: Note that we are restricting the sum to v i having only a possible sign; although matrices don’t commute, we avoid choosing different signs since it will likely self-intersect a lower radius walk.

Original: The permutations σ are required since, for example, the path up → left is different (in a general graph) than the path left → up .
Revised: The permutations σ are required since, for example, the path up → left is different (in a general graph) than the path left → up .

Original: This matrix B R has a total of (cid:80)
Revised: This matrix B R has a total of (cid:80)

Original: Rr =0 (2 n ) r
Revised: Rr =0 (2 n ) r

Original: =
Revised: =

Original: (2 n ) R +1 − 1 2 n −parameters, with a high redundancy since some permutations might be very similar, e.g. for a grid graph we have that up → left is identical to left → up .
Revised: (2 n ) R +1 − 1 2 n −parameters, with a high redundancy since some permutations might be very similar, e.g. for a grid graph we have that up → left is identical to left → up .

Original: Hence, we can replace the permutation S n by a reverse ordering, meaning that (cid:81) Nj B j = B N ...
Revised: Hence, we can replace the permutation S n by a reverse ordering, meaning that (cid:81) Nj B j = B N ...

Original: B 2 B 1 .
Revised: B 2 B 1 .

Original: Doing so does not perfectly generalize the radius- R kernel for all graphs, but it generalizes it on a grid and signiﬁcantly reduces the number of parameters to (cid:80)
Revised: Doing so does not perfectly generalize the radius- R kernel for all graphs, but it generalizes it on a grid and signiﬁcantly reduces the number of parameters to (cid:80)

Original: Rr =0 (cid:80) min ( n,r ) l = 2 r (cid:0) nl (cid:1)(cid:0) r − 1 l − 1 (cid:1) .
Revised: Rr =0 (cid:80) min ( n,r ) l = [Equation] (cid:1) .

Original: B A PPENDIX - I MPLEMENTATION DETAILS B. B ENCHMARKS AND DATASETS
Revised: B A PPENDIX - I MPLEMENTATION DETAILS B. B ENCHMARKS AND DATASETS

Original: We use a variety of benchmarks proposed by Dwivedi et al.
Revised: We use a variety of benchmarks proposed by Dwivedi et al.

Revised: (2020) and Hu et al.

Original: (2020) to test the empirical performance of our proposed methods.
Revised: (2020) to test the empirical performance of our proposed methods.

Original: In particular, to have a wide variety of graphs and tasks we chose:
Revised: In particular, to have a wide variety of graphs and tasks we chose:

Original: ZINC, a graph regression dataset from molecular chemistry.
Revised: ZINC, a graph regression dataset from molecular chemistry.

Original: The task is to predict a score that is a subtraction of computed properties logP − SA , with logP being the computed octanol-water partition coefﬁcient, and SA being the synthetic accessibility score (Jin et al., 2018).
Revised: The task is to predict a score that is a subtraction of computed properties logP − SA , with logP being the computed octanol-water partition coefﬁcient, and SA being the synthetic accessibility score (Jin et al., 2018).

Original: CIFAR10, a graph classiﬁcation dataset from computer vision (Krizhevsky, 2009).
Revised: CIFAR10, a graph classiﬁcation dataset from computer vision (Krizhevsky, 2009).

Original: The task is to classify the images into 10 different classes, with a total of 5000 training image per class and 1000 test image per class.
Revised: The task is to classify the images into 10 different classes, with a total of 5000 training image per class and 1000 test image per class.

Original: Each image has 32 × 32 pixels, but the pixels have been clustered into a graph of ∼ 100 super-pixels.
Revised: Each image has 32 × 32 pixels, but the pixels have been clustered into a graph of ∼ 100 super-pixels.

Original: Each super-pixel becomes a node in an almost grid-shaped graph, with 8 edges per node.
Revised: Each super-pixel becomes a node in an almost grid-shaped graph, with 8 edges per node.

Original: The clustering uses the code from Knyazev et al.
Revised: The clustering uses the code from Knyazev et al.

Original: (2019), and results in a different number of super-pixels per graph.
Revised: (2019), and results in a different number of super-pixels per graph.

Original: PATTERN, a node classiﬁcation synthetic benchmark generated with Stochastic Block Models, which are widely used to model communities in social networks.
Revised: PATTERN, a node classiﬁcation synthetic benchmark generated with Stochastic Block Models, which are widely used to model communities in social networks.

Original: The task is to classify the nodes into 2 communities and it tests the fundamental ability of recognizing speciﬁc predetermined subgraphs.
Revised: The task is to classify the nodes into 2 communities and it tests the fundamental ability of recognizing speciﬁc predetermined subgraphs.

Revised: MolHIV, a graph classiﬁcation benchmark from molecular chemistry.

Revised: The task is to predict whether a molecule inhibits HIV virus replication or not.

Revised: The molecules in the training, validation and test sets are divided using a scaffold splitting procedure that splits the molecules based on their two-dimensional structural frameworks.

Original: Our goal is to provide a fair comparison to demonstrate the capacity of our proposed aggregators.
Revised: Our goal is to provide a fair comparison to demonstrate the capacity of our proposed aggregators.

Original: Therefore, we compare the various methods on both types of architectures using the same hyperparameters tuned in previous works (Corso et al., 2020) for similar networks.
Revised: Therefore, we compare the various methods on both types of architectures using the same hyperparameters tuned in previous works (Corso et al., 2020) for similar networks.

Original: The models vary exclusively in the aggregation method and the width of the architectures to keep a set parameter budget.
Revised: The models vary exclusively in the aggregation method and the width of the architectures to keep a set parameter budget.

Original: In CIFAR10 it is impossible to numerically compute a deterministic vector ﬁeld with eigenvectors due to the multiplicity of λ 1 being greater than 1.
Revised: In CIFAR10 it is impossible to numerically compute a deterministic vector ﬁeld with eigenvectors due to the multiplicity of λ 1 being greater than 1.

Original: This is caused by the symmetry of the square image, and is extremely rare in real-world graphs.
Revised: This is caused by the symmetry of the square image, and is extremely rare in real-world graphs.

Original: Therefore, we used as underlying vector ﬁeld the gradient of the coordinates of the image.
Revised: Therefore, we used as underlying vector ﬁeld the gradient of the coordinates of the image.

Original: Note that these directions are provided in the nodes’ features in the dataset and available to all models, that they are co-linear to the eigenvectors of the grid as per lemma C.1, and that they mimic the inductive bias in CNNs.
Revised: Note that these directions are provided in the nodes’ features in the dataset and available to all models, that they are co-linear to the eigenvectors of the grid as per lemma C.1, and that they mimic the inductive bias in CNNs.

Original: B. I MPLEMENTATION AND COMPUTATIONAL COMPLEXITY
Revised: B. I MPLEMENTATION AND COMPUTATIONAL COMPLEXITY

Original: Unlike several more expressive graph networks (Kondor et al., 2018; Maron et al., 2018), our method does not require a computational complexity superlinear with the size of the graph.
Revised: Unlike several more expressive graph networks (Kondor et al., 2018; Maron et al., 2018), our method does not require a computational complexity superlinear with the size of the graph.

Original: The calculation of the ﬁrst k eigenvectors during pretraining, done using Lanczos method (Lanczos, 1950) and the sparse module of Scipy, has a time complexity of O ( Ek ) where E is the number of edges.
Revised: The calculation of the ﬁrst k eigenvectors during pretraining, done using Lanczos method (Lanczos, 1950) and the sparse module of Scipy, has a time complexity of O ( Ek ) where E is the number of edges.

Original: During training the complexity is equivalent to a m -aggregator GNN O
Revised: During training the complexity is equivalent to a m -aggregator GNN O

Original: ( Em ) (Corso et al., 2020).
Revised: ( Em ) (Corso et al., 2020) for the aggregation and O ( Nm ) for the MLP.

Original: To all the architectures we added residual connections (He et al., 2016), batch normalization (Ioffe & Szegedy, 2015) and graph size normalization (Dwivedi et al., 2020).
Revised: To all the architectures we added residual connections (He et al., 2016), batch normalization (Ioffe & Szegedy, 2015) and graph size normalization (Dwivedi et al., 2020).

Original: For all the datasets with non-regular graphs, we combine the various aggregators with logarithmic degree-scalers as in Corso et al.
Revised: For all the datasets with non-regular graphs, we combine the various aggregators with logarithmic degree-scalers as in Corso et al.

Original: An important thing to note is that, for dynamic graphs, the eigenvectors need to be re-computed dynamically with the changing edges.
Revised: An important thing to note is that, for dynamic graphs, the eigenvectors need to be re-computed dynamically with the changing edges.

Original: Fortunately, there are random walk based algorithms that can estimate φ 1 quickly, especially for small changes to the graph (Doshi & Eun, 2000).
Revised: Fortunately, there are random walk based algorithms that can estimate φ 1 quickly, especially for small changes to the graph (Doshi & Eun, 2000).

Original: In the current empirical results, we do not work with dynamic graphs.
Revised: In the current empirical results, we do not work with dynamic graphs.

Original: C A PPENDIX - M ATHEMATICAL PROOFS C. P ROOF FOR T HEOREM 2.1 (D IRECTIONAL SMOOTHING )
Revised: B. R UNNING TIME

Revised: The precomputation of the ﬁrst four eigenvectors for all the graphs in the datasets takes 38 s for ZINC, 96 s for PATTERN and 120 s for MolHIV on CPU. Table 1 shows the average running time on GPU for all the various model from ﬁgure 5.

Revised: On average, the epoch running time is 16% slower for the DGN compared to the mean aggregation, but a faster convergence for DGN means that the total training time is on average 8% faster for DGN.

Revised: [Table]

Revised: Simple

Revised: Complex mean 3.29s/1505s 3.58s/1584s 3.56s/1654s 153.1s/10154s 117.8s/9031s mean dx3.86s/1122s 3.77s/1278s 4.22s/1371s 144.9s/8109s 127.2s/8417s mean dx 1 dx4.23s/1360s 4.55s/1560s 4.63s/1680s 153.3s/8057s

Revised: 167.9s/9326s mean av3.68s/1297s 3.84s/1398s 3.92s/1272s 128.0s/8680s 88.1s/7456s

Revised: mean av 1 av3.95s/1432s 4.03s/1596s 4.07s/1721s 134.2s/8115s

Revised: 170.4s/11114s mean dx 1 av3.89s/1079s 4.09s/1242s 4.58s/1510s 118.6s/6221s 144.2s/9112s avg increase +19%/-16%

Revised: +13%/-11% +20%/-9%

Revised: -11%/-23% +18%/+1%

Revised: CIFAR

Revised: MolHIV

Revised: Aggregators

Revised: Simple

Revised: Complex

Revised: Simple mean 83.6s/10526s 78.7s/10900s

Revised: 11.4s/2189s mean dx12.6s/2348s mean dx 1 dx98.4s/8405s 100.9s/5191s 14.1s/2345s mean av12.2s/2177s mean av 1 av117.1s/12834s 89.5s/14481s 13.9s/2150s mean dx 1 av14.0s/2070s avg increase +29%

Revised: [Table]

Revised: B. E IGENVECTOR MULTIPLICITY

Revised: The possibility to deﬁne equivariant directions using the low-frequency Laplacian eigenvectors is subject to the uniqueness of those vectors.

Revised: When the dimension of the eigenspaces associated with the lowest eigenvalues is 1 , the eigenvectors are deﬁned up to a constant factor.

Revised: In section 2.4, we propose the use of unit vector normalization and an absolute value to eliminate the scale and sign ambiguity.

Revised: When the dimension of those eigenspaces is greater than 1 , it is not possible to deﬁne equivariant directions using the eigenvectors.

Revised: Fortunately, it is very rare for the Laplacian matrix to have repeated eigenvalues in real-world datasets.

Revised: We validate this claim by looking at ZINC and PATTERN datasets where we found no graphs with repeated Fiedler vector and only one graph out of 26k with multiplicity of the second eigenvector greater than 1.

Revised: When facing a graph that presents repeated Laplacian eigenvalues, we propose to randomly shufﬂe, during training time, different eigenvectors randomly sampled in the eigenspace.

Revised: This technique will act as a data augmentation of the graph during training time allowing the network to train with multiple directions at the same time.

Revised: C A PPENDIX - M ATHEMATICAL PROOFS C. P ROOF FOR T HEOREM 2.1 (D IRECTIONAL SMOOTHING )

Original: The operation y
Revised: The operation y

Original: =
Revised: =

Original: B av x is the directional average of x , in the sense that y u is the mean of x v , weighted by the direction and amplitude of F .
Revised: B av x is the directional average of x , in the sense that y u is the mean of x v , weighted by the direction and amplitude of F .

Original: Proof.
Revised: Proof.

Original: This should be a simple proof, that if we want a weighted average of our neighbours, we simply need to multiply the weights by each neighbour, and divide by the sum of the weights.
Revised: This should be a simple proof, that if we want a weighted average of our neighbours, we simply need to multiply the weights by each neighbour, and divide by the sum of the weights.

Original: Of course, the weights should be positive.
Revised: Of course, the weights should be positive.

Original: C. P ROOF FOR T HEOREM 2.2 (D IRECTIONAL DERIVATIVE )
Revised: C. P ROOF FOR T HEOREM 2.2 (D IRECTIONAL DERIVATIVE )

Original: Suppose ˆ F have rows of unit L 1 norm.
Revised: Suppose ˆ F have rows of unit L 1 norm.

Original: The operation y =
Revised: The operation y =

Original: B dx ( ˆ F ) x is the centered directional derivative of x in the direction of F , in the sense of equation 4, i.e.
Revised: B dx ( ˆ F ) x is the centered directional derivative of x in the direction of F , in the sense of equation 4, i.e.

Original: [Equation]
Revised: [Equation]

Revised: x

Original: Proof.
Revised: Proof.

Original: Since F rows have unit L 1 norm, ˆ F = F .
Revised: Since F rows have unit L 1 norm, ˆ F = F .

Original: The i -th coordinate of the vector (cid:16)
Revised: The i -th coordinate of the vector (cid:16)

Original: F − diag (cid:16)(cid:80) j F : ,j (cid:17)(cid:17) x is  
Revised: F − diag (cid:16)(cid:80) j F : ,j (cid:17)(cid:17) x is   F x − diag  (cid:88) j F   x  

Original: [Equation]
Revised: i = (cid:88) j F i,j x ( j ) −  (cid:88) j

Original: (cid:88)
Revised: F i,j   x ( i ) = (cid:88)

Original: [Equation]
Revised: j :( i,j ) ∈ E ( x ( j ) − x ( i )) F i,j = D F x ( i )

Original: C. P ROOF FOR T HEOREM 2.3 (K-G RADIENT OF THE LOW  FREQUENCY EIGENVECTORS )
Revised: C. P ROOF FOR T HEOREM 2.3 (K-G RADIENT OF THE LOW - FREQUENCY EIGENVECTORS )

Original: Let λ i and φ i be the eigenvalues and eigenvectors of the normalized Laplacian of a connected graph
Revised: Let λ i and φ i be the eigenvalues and eigenvectors of the normalized Laplacian of a connected graph

Original: L norm and let a, b  be the nodes that have a farthest K-walk distance.
Revised: L norm and let a, b = arg max 1 ≤ i,j ≤ n { d K ( v i , v j ) } be the nodes that have highest K-walk distance.

Original: Suppose that λ 2 (cid:29) λ 1 , then the optima of φ 1 approximates a, b  .
Revised: Let m = arg min 1 ≤ i ≤ n ( φ 1 ) i and M = arg max 1 ≤ i ≤ n ( φ 1 ) i , then d K ( v m , v M ) − d K ( v a , v b ) has order O (1 − λ 2 ) .

Original: Proof.
Revised: Proof.

Revised: For this theorem, we use the indices i = 0 , ..., ( N − 1) , sorted such that λ i ≤ λ i +1 .

Original: {+Hence, λ 0 = 
+}
Revised: {+Hence, λ 0 = 
+}

Revised: and λ 1 is the ﬁrst non-trivial eigenvalue.

Original: First we need the following proposition:
Revised: First we need the following proposition:

Original: Proposition 1 (K-walk distance matrix) .
Revised: Proposition 1 (K-walk distance matrix) .

Original: The K-walk distance matrix P associated with a graph is the matrix such that ( P ) i,j = d K ( v i , v j ) can be written as (cid:80)
Revised: The K-walk distance matrix P associated with a graph is the matrix such that ( P ) i,j = d K ( v i , v j ) can be written as (cid:80)

Original: Kp =1 W p , where W = D − 1 A is the random walk matrix.
Revised: Kp =1 W p , where W = D − 1 A is the random walk matrix.

Original: Let’s deﬁne W
Revised: Let’s deﬁne W

Original: = D [Equation]{+− 
+}
Revised: = D {+− 
+}

Original: 1 A the random walk matrix of the graph.
Revised: A the random walk matrix of the graph.

Original: First, we are going to show that W is jointly diagonalizable with L norm and we are going to relate its eigenvectors φ (cid:48)
Revised: First, we are going to show that W is jointly diagonalizable with L norm and we are going to relate its eigenvectors φ (cid:48)

Original: i and its eigenvalues λ (cid:48)
Revised: i and its eigenvalues λ (cid:48)

Original: i with the ones of W .
Revised: i with the ones of W .

Original: Indeed, L sym is a symmetric real matrix which is semi-positive deﬁnite diagonalizable by the spectral theorem.
Revised: Indeed, L sym is a symmetric real matrix which is semi-positive deﬁnite diagonalizable by the spectral theorem.

Original: Since the matrix L norm is similar to D 12 L norm D − 12 =
Revised: Since the matrix L norm is similar to D 12 L norm D − 12 =

Original: D − 12 LD − 12 =
Revised: D − 12 LD − 12 =

Original: L sym and the matrix of similarity is D 12 , a positive deﬁnite matrix, L norm is diagonalizable and semi-positive deﬁnite.
Revised: L sym and the matrix of similarity is D 12 , a positive deﬁnite matrix, L norm is diagonalizable and semi-positive deﬁnite.

Original: By
Revised: By

Original: [Equation]
Revised: [Equation]

Original: the random walk matrix is jointly diagonalizable with the random walk Laplacian.
Revised: the random walk matrix is jointly diagonalizable with the random walk Laplacian.

Original: Also their eigenvalues and eigenvectors are related to each other by φ i = φ (cid:48) n +1 − i and λ (cid:48)
Revised: Also their eigenvalues and eigenvectors are related to each other by φ i = φ (cid:48) n − 1 − i and λ (cid:48)

Original: i = 1 − λ n +1 − i
Revised: i = 1 − λ n [Equation]

Original:  Moreover, the constant eigenvector associated with eigenvalue 0 of the Random walk Laplacian, is the eigenvector associated with the highest eigenvalue of the Random walk matrix and by the formula obtained, λ (cid:48) n  = 1 − λ 1 =
Revised: [Equation] i Moreover, the constant eigenvector associated with eigenvalue 0 of the Random walk Laplacian, is the eigenvector associated with the highest eigenvalue of the Random walk matrix and by the formula obtained, λ (cid:48) n − 1 = 1 − λ 0 =

Original: Now, we are going to approximate the K-walk distance matrix P using the 2 eigenvectors of the
Revised: Now, we are going to approximate the K-walk distance matrix P using the 2 eigenvectors of the

Original: Random walk matrix associated with the highest eigenvalues.
Revised: Random walk matrix associated with the highest eigenvalues.

Original: By Proposition 1 we have that P = (cid:80)
Revised: By Proposition 1 we have that P = (cid:80)

Original: Kp =1 W p , which can be written as
Revised: Kp =1 W p , which can be written as

Original: [Equation]
Revised: [Equation]

Original: (cid:88)

Original: [Equation]

Original: [Equation]

Original: (cid:88)

Original: [Equation]

Original: [Equation]

Original: by eigen-decomposition.
Revised: by eigen-decomposition.

Original: Since λ n −  i = 1 − λ (cid:48)
Revised: Since λ n − 1 − i = 1 − λ (cid:48)

Original: i and λ 2 (cid:29) λ 1 , we have that λ (cid:48)n − 1 (cid:29) λ (cid:48) n − 2 , hence we can approximate P =
Revised: i and λ 2 (cid:29) λ 1 , we have that λ (cid:48)

Original: K (cid:88) p = ( n (cid:88) i = φ [Equation]
Revised: n − 2 (cid:29) λ (cid:48) n − 3 , hence we can approximate

Original: [Equation]

Original: (cid:48) [Equation] [Equation] ) ≈
Revised: P

Original: [Equation]
Revised: [Equation]

Original: (cid:88)

Original: [Equation]

Original: = (

Original: [Equation]

Original: (cid:88)

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: (cid:48)

Original: [Equation]

Original: [Equation]

Original: = =

Original: K (cid:88) p =

Original: ( 1 (cid:88) i =

Original: φ i

Original: [Equation]

Original: [Equation]

Original: where κ = (cid:80)
Revised: where κ = (cid:80)

Original: Kp =1 (1 − λ 1 ) p is a positive constant.
Revised: Kp =1 (1 − λ 1 ) p is a positive constant.

Original: Now we are going to show that the farthest nodes with respect to the K -walk distance are the ones associated with the highest and lowest value of φ 1 .
Revised: Now we are going to show that the farthest nodes with respect to the K -walk distance are the ones associated with the highest and lowest value of φ 1 .

Original: Indeed if we want to choose i, j to be at the farthest distance we need to minimise
Revised: Indeed if we want to choose i, j to be at the farthest distance we need to minimise

Original: [Equation]
Revised: [Equation]

Original: which is minimum when φ 1 ( i ) φ 1 ( j ) is minimum.
Revised: which is minimum when φ 1 ( i ) φ 1 ( j ) is minimum.

Original: The quantity φ 1 ( i ) φ 1 ( j ) is minimised when it has negative sign and highest absolute value, hence when i, j are associated with the negative and positive values with the highest absolute value: the lowest and the highest value of φ 1 . 
Revised: The quantity φ 1 ( i ) φ 1 ( j ) is minimised when it has negative sign and highest absolute value, hence when i, j are associated with the negative and positive values with the highest absolute value: the lowest and the highest value of φ 1 . Hence, d K ( v M , v m ) − d K ( v a , v b ) = O (1 − λ 2 )

Original: C. I NFORMAL ARGUMENT IN SUPPORT OF C ONJECTURE 2.4(G RADIENT STEPS REDUCE EXPECTED HITTING TIME )
Revised: C. I NFORMAL ARGUMENT IN SUPPORT OF C ONJECTURE 2.4 (G RADIENT STEPS REDUCE EXPECTED HITTING TIME )

Original: Suppose that x, y are uniformly distributed random nodes such that φ i ( x ) <
Revised: Suppose that x, y are uniformly distributed random nodes such that φ i ( x ) <

Original: φ i ( y ) .
Revised: φ i ( y ) .

Original: Let z be the node obtained from x by taking one step in the direction of ∇ φ i , then the expected hitting time is decreased proportionally to λ − 1 iand
Revised: Let z be the node obtained from x by taking one step in the direction of ∇ φ i , then the expected hitting time is decreased proportionally to λ − 1 i

Original: [Equation]
Revised: [Equation]

Revised: As a reminder, the deﬁnition of a gradient step is given in the deﬁnition 3, copied below.

Revised: Suppose the two neighboring nodes x and z are such that φ ( z ) − φ ( x ) is maximal among the neighbors of x , then we will say z is obtained from x by taking a step in the direction of the gradient ∇ φ .

Original: In (Chung & S.T.Yau, 2000), it is shown the hitting time Q ( x, y ) is given by the equation
Revised: In (Chung & S.T.Yau, 2000), it is shown the hitting time Q ( x, y ) is given by the equation

Original: [Equation]
Revised: [Equation]

Original: With λ k and φ k being the k -th eigenvalues and eigenvectors of the symmetric normalized Laplacian
Revised: With λ k and φ k being the k -th eigenvalues and eigenvectors of the symmetric normalized Laplacian

Original: L sym , vol the sum of the degrees of all nodes, d x the degree of node x and G Green’s function for the graph
Revised: L sym , vol the sum of the degrees of all nodes, d x the degree of node x and G Green’s function for the graph

Original: [Equation]
Revised: [Equation]

Original: Since the sign of the eigenvector is not deterministic, the choice φ i ( x ) <
Revised: Since the sign of the eigenvector is not deterministic, the choice φ i ( x ) <

Original: φ i ( y ) is used to simplify the argument without having to consider the change in sign.
Revised: φ i ( y ) is used to simplify the argument without having to consider the change in sign.

Original: Supposing λ 1 (cid:28) λ 2 , the ﬁrst term of the sum of G has much more weight than the following terms.
Revised: Supposing λ 1 (cid:28) λ 2 , the ﬁrst term of the sum of G has much more weight than the following terms.

Original: With z obtained from x by taking a step in the direction of the gradient of φ 1 we have
Revised: With z obtained from x by taking a step in the direction of the gradient of φ 1 we have

Original: [Equation]
Revised: [Equation]

Original: We want to show that the following inequality holds
Revised: We want to show that the following inequality holds

Original: [Equation]
Revised: [Equation]

Original: this is equivalent to the following inequality
Revised: this is equivalent to the following inequality

Original: [Equation]
Revised: [Equation]

Original: By the hypothesis λ 1 (cid:28) λ 2 , we can approximate G ( x, y ) ∼
Revised: By the hypothesis λ 1 (cid:28) λ 2 , we can approximate G ( x, y ) ∼

Original: d 12 x d − 12 y [Equation] ( x ) φ 1 ( y ) so the last inequality is equivalent to
Revised: d 12 x d − 12 y 1 λ 1 φ 1 ( x ) φ 1 ( y ) so the last inequality is equivalent to

Original: [Equation]
Revised: [Equation]

Original: Removing all equal terms from both sides, the inequality is equivalent to
Revised: Removing all equal terms from both sides, the inequality is equivalent to

Original: [Equation]
Revised: [Equation]

Original: But showing this last inequality is not easy.
Revised: But showing this last inequality is not easy.

Original: We know that φ 1 ( z ) > φ 1 ( x ) and from the choice of z being a step in the direction of ∇ φ 1 , we know it is less likely to be on the border of the graph so we believe E ( d z ) ≥ E ( d x ) .
Revised: We know that φ 1 ( z ) > φ 1 ( x ) and from the choice of z being a step in the direction of ∇ φ 1 , we know it is less likely to be on the border of the graph so we believe E ( d z ) ≥ E ( d x ) .

Original: Thus we also believe that the conjecture should hold in general.
Revised: Thus we also believe that the conjecture should hold in general.

Original: We believe this should be true even without the assumption on λ 1 and λ 2 and for more eigenvectors than φ 1 .
Revised: We believe this should be true even without the assumption on λ 1 and λ 2 and for more eigenvectors than φ 1 .

Original: C. P ROOF FOR L EMMA C.1 (C OSINE EIGENVECTORS )
Revised: C. P ROOF FOR L EMMA C.1 (C OSINE EIGENVECTORS )

Original: Consider the lattice graph Γ of size N 1 × N 2 × ...
Revised: Consider the lattice graph Γ of size N 1 × N 2 × ...

Original: × N n , that has vertices (cid:81) i
Revised: × N n , that has vertices (cid:81) i

Original: =1 ,...,n { 1 , ..., N i } and the vertices ( x i ) i
Revised: =1 ,...,n { 1 , ..., N i } and the vertices ( x i ) i

Original: =1 ,...,n and ( y i ) i
Revised: =1 ,...,n and ( y i ) i

Original: =1 ,...,n are connected by an edge iff | x i − y
Revised: =1 ,...,n are connected by an edge iff | x i − y

Original: i | = 1 for one index i and 

Revised: i | = 1 for one index i and

Original: for all other indices.
Revised: for all other indices.

Original: Note that there are no diagonal edges in the lattice.
Revised: Note that there are no diagonal edges in the lattice.

Original: The eigenvector ofthe Laplacian of the grid L (Γ) are given by φ j .
Revised: The eigenvector of

Original: Lemma C.1 (Cosine eigenvectors) .
Revised: Lemma C.1 (Cosine eigenvectors) .

Original: The Laplacian of Γ has an eigenvalue 2 − 2 cos (cid:16) πN i (cid:17) with the associated eigenvector φ j that depends only the variable in the i -th dimension and is constant in all others, with φ j = 1 N 1 ⊗ 1 N 2 ⊗ ...
Revised: The Laplacian of Γ has an eigenvalue 2 − 2 cos (cid:16) πN i (cid:17) with the associated eigenvector φ j that depends only the variable in the i -th dimension and is constant in all others, with φ j = 1 N 1 ⊗ 1 N 2 ⊗ ...

Original: ⊗ x 1 ,N i ⊗ ... ⊗ 1 N n , and x 1 ,N i ( j ) = cos (cid:0) πjn − π 2 n (cid:1)
Revised: ⊗ x 1 ,N i ⊗ ... ⊗ 1 N n , and x 1 ,N i ( j ) = cos (cid:0) πjn − π 2 n (cid:1)

Original: Proof.
Revised: Proof.

Original: First, recall the well known result that the path graph on N vertices P N has eigenvalues
Revised: First, recall the well known result that the path graph on N vertices P N has eigenvalues

Original: [Equation]
Revised: [Equation]

Original: with associated eigenvector x k with i -th coordinate
Revised: with associated eigenvector x k with i -th coordinate

Original: [Equation]
Revised: [Equation]

Original: The Cartesian product of two graphs G =
Revised: The Cartesian product of two graphs G =

Original: ( V G , E G ) and H =
Revised: ( V G , E G ) and H =

Original: ( V H , E H ) is deﬁned as G × H =
Revised: ( V H , E H ) is deﬁned as G × H =

Original: ( V G × H , E G × H ) with V G × H
Revised: ( V G × H , E G × H ) with V G × H

Original: = V G × V H and (( u 1 , u 2 ) , (( v 1 , v 2 ))
Revised: = V G × V H and (( u 1 , u 2 ) , (( v 1 , v 2 ))

Original: ∈
Revised: ∈

Original: E G × H iff either u 1 = v 1 and ( u 2 , v 2 ) ∈
Revised: E G × H iff either u 1 = v 1 and ( u 2 , v 2 ) ∈

Original: E H or ( u 1 , v 1 )
Revised: E H or ( u 1 , v 1 )

Original: ∈ V G and u 2 = v 2 .
Revised: ∈ V G and u 2 = v 2 .

Original: It is shown in (Fiedler, 1973)
Revised: It is shown in (Fiedler, 1973)

Original: that if ( µ i ) i =1 ,...,m and ( λ j ) j =1 ,...,n are the eigenvalues of G and H respectively, then the eigenvalues of the Cartesian product graph G × H are µ i + λ j for all possible eigenvalues µ i and λ j .
Revised: that if ( µ i ) i =1 ,...,m and ( λ j ) j =1 ,...,n are the eigenvalues of G and H respectively, then the eigenvalues of the Cartesian product graph G × H are µ i + λ j for all possible eigenvalues µ i and λ j .

Original: Also, the eigenvectors associated to the eigenvalue µ
Revised: Also, the eigenvectors associated to the eigenvalue µ

Original: i
Revised: i

Original: + λ
Revised: + λ

Original: j are u
Revised: j are u

Original: i ⊗ v j with u i an eigenvector of the Laplacian of G associated to the eigenvalue µ i and v j an eigenvector of the Laplacian of H associated to the eigenvalue λ j .
Revised: i ⊗ v j with u i an eigenvector of the Laplacian of G associated to the eigenvalue µ i and v j an eigenvector of the Laplacian of H associated to the eigenvalue λ j .

Original: Finally, noticing that a lattice of shape N 1 × N 2 × ...
Revised: Finally, noticing that a lattice of shape N 1 × N 2 × ...

Original: ×
Revised: ×

Original: N n is really the Cartesian product of path graphs of length N 1 up to N n , we conclude that there are eigenvalues 2 − 2 cos (cid:16) πN i (cid:17) .
Revised: N n is really the Cartesian product of path graphs of length N 1 up to N n , we conclude that there are eigenvalues 2 − 2 cos (cid:16) πN i (cid:17) .

Original: Denoting by 1 N j the vector in R N j with only ones as coordinates, then the eigenvector associated to the eigenvalue 2 − 2 cos (cid:16) πN i (cid:17) is
Revised: Denoting by 1 N j the vector in R N j with only ones as coordinates, then the eigenvector associated to the eigenvalue 2 − 2 cos (cid:16) πN i (cid:17) is

Original: [Equation]
Revised: [Equation]

Original: where x 1 ,N i is the eigenvector of the Laplacian of P N i associated to its ﬁrst non-zero eigenvalue.
Revised: where x 1 ,N i is the eigenvector of the Laplacian of P N i associated to its ﬁrst non-zero eigenvalue.

Original: [Equation]
Revised: [Equation]

Original: C. R ADIUS 1 CONVOLUTION KERNELS IN A GRID
Revised: C. R ADIUS 1 CONVOLUTION KERNELS IN A GRID

Original: In this section we show any radius 1 convolution kernel can be obtained as a linear combination of the B dx ( ∇ φ i ) and B av ( ∇ φ i ) matrices for the right choice of Laplacian eigenvectors φ i .
Revised: In this section we show any radius 1 convolution kernel can be obtained as a linear combination of the B dx ( ∇ φ i ) and B av ( ∇ φ i ) matrices for the right choice of Laplacian eigenvectors φ i .

Original: First we show this can be done for 1-d convolution kernels.
Revised: First we show this can be done for 1-d convolution kernels.

Original: Theorem C.2.
Revised: Theorem C.2.

Original: On a path graph, any 1D convolution kernel of size 3 k is a linear combination of the aggregators B av , B dx and the identity I .
Revised: On a path graph, any 1D convolution kernel of size 3 k is a linear combination of the aggregators B av , B dx and the identity I .

Original: Proof.
Revised: Proof.

Original: Recall from the previous proof that the ﬁrst non zero eigenvalue of the path graph P N has associated eigenvector φ 1 ( i ) =
Revised: Recall from the previous proof that the ﬁrst non zero eigenvalue of the path graph P N has associated eigenvector φ 1 ( i ) =

Original: cos( πiN − π 2 N ) .
Revised: cos( πiN − π 2 N ) .

Original: Since this is a monotone decreasing function in i , the i -th row of ∇ φ 1 will be (0 , ..., 0 , s i − 1 , 0 ,
Revised: Since this is a monotone decreasing function in i , the i -th row of ∇ φ 1 will be (0 , ..., 0 , s i − 1 , 0 ,

Original: − s i +1 , 0 , ..., 0) with s
Revised: − s i +1 , 0 , ..., 0) with s

Original: i − 1 and s i +1 > 0 .
Revised: i − 1 and s i +1 > 0 .

Original: We are trying to solve( a B av + b B dx
Revised: We are trying to solve

Original: + c Id ) i, : = (0 , ..., 0 , x, y, z, 0 , ..., 0) with x, y, z, in positions i − 1 , i and i + 1 .
Revised: [Equation] ..., 0) with x, y, z, in positions i − 1 , i and i + 1 .

Original: This simpliﬁes to solving
Revised: This simpliﬁes to solving

Original: [Equation]
Revised: [Equation]

Original: with s =
Revised: with s =

Original: ( s i − 1 , 0 , − s i +1 ) , which always has a solution because s i − 1 , s i +1 > 0 .
Revised: ( s i − 1 , 0 , − s i +1 ) , which always has a solution because s i − 1 , s i +1 > 0 .

Original: Theorem C.3 (Generalization radius-1 convolutional kernel in a grid) .
Revised: Theorem C.3 (Generalization radius-1 convolutional kernel in a grid) .

Original: Let Γ be the n -dimensional lattice as above and let φ j be the eigenvectors of the Laplacian of the lattice as in theorem C.1.
Revised: Let Γ be the n -dimensional lattice as above and let φ j be the eigenvectors of the Laplacian of the lattice as in theorem C.1.

Original: Then any radius 1 kernel k on Γ is a linear combination of the aggregators B av ( φ i ) , B dx ( φ i ) and
Revised: Then any radius 1 kernel k on Γ is a linear combination of the aggregators B av ( φ i ) , B dx ( φ i ) and

Original: I .
Revised: I .

Original: Proof.
Revised: Proof.

Original: This is a direct consequence of C.2 obtained by adding n 1-dimensional kernels, with each kernel being in a different axis of the grid as per Lemma C.1.
Revised: This is a direct consequence of C.2 obtained by adding n 1-dimensional kernels, with each kernel being in a different axis of the grid as per Lemma C.1.

Original: See ﬁgure 2.5 for a visual example in 2D.
Revised: See ﬁgure 4 for a visual example in 2D.

Original: C. P ROOF FOR T HEOREM 2.7 (G ENERALIZATION RADIUS - R CONVOLUTIONAL KERNEL IN A LATTICE )
Revised: C. P ROOF FOR T HEOREM 2.7 (G ENERALIZATION RADIUS - R CONVOLUTIONAL KERNEL IN A LATTICE )

Original: For an n -dimensional lattice, any convolutional kernel of radius R can be realized by a linear combination of directional aggregation matrices and their compositions.
Revised: For an n -dimensional lattice, any convolutional kernel of radius R can be realized by a linear combination of directional aggregation matrices and their compositions.

Original: Proof.
Revised: Proof.

Original: For clarity, we ﬁrst do the 2 dimensional case for a radius 2, then extended to the general case.
Revised: For clarity, we ﬁrst do the 2 dimensional case for a radius 2, then extended to the general case.

Original: Let k be the radius 2 kernel on a grid represented by the matrix a 5 × 5 = 
Revised: Let k be the radius 2 kernel on a grid represented by the matrix [Equation] [Equation]

Original:   a − 2 ,   a − 1 , −a − 1 ,a − 1 , a 0 , −a 0 , −a 0 ,a 0 ,a 0 , 2a 1 , −a 1 ,a 1 ,   a 2 ,  
Revised:  

Original:  since we supposed the N 1 × N 2 grid was such that N 1 > N 2 , by theorem C.1, we have that φ 1 is depending only in the ﬁrst variable x 1 and is monotone in x 1 .
Revised:  since we supposed the N 1 × N 2 grid was such that N 1 > N 2 , by theorem C.1, we have that φ 1 is depending only in the ﬁrst variable x 1 and is monotone in x 1 .

Original: Recall from C.1 that
Revised: Recall from C.1 that

Original: [Equation]
Revised: [Equation]

Original: The vector N 1 π ∇ arccos( φ 1 ) will be denoted by F 1 in the rest.
Revised: The vector N 1 π ∇ arccos( φ 1 ) will be denoted by F 1 in the rest.

Original: Notice all entries of F 1 are 0 or ± 1 .
Revised: Notice all entries of F 1 are 0 or ± 1 .

Original: Denote by F 2 the gradient vector N 2 π ∇ arccos( φ k ) where φ k is the eigenvector given by theorem
Revised: Denote by F 2 the gradient vector N 2 π ∇ arccos( φ k ) where φ k is the eigenvector given by theorem

Original: C.1 that is depending only in the second variable x 2 and is monotone in x 1 and recall
Revised: C.1 that is depending only in the second variable x 2 and is monotone in x 1 and recall

Original: [Equation]
Revised: [Equation]

Original: For a matrix B , let B ± the positive/negative parts of B , ie matrices with positive entries such that
Revised: For a matrix B , let B ± the positive/negative parts of B , ie matrices with positive entries such that

Original: B = B + − B − .
Revised: B = B + − B − .

Original: Let B r 1 be a matrix representing the radius 1 kernel with weights
Revised: Let B r 1 be a matrix representing the radius 1 kernel with weights

Original: [Equation]
Revised: [Equation]

Original: The matrix B r 1 can be obtained by theorem C.3.
Revised: The matrix B r 1 can be obtained by theorem C.3.

Original: Then the radius 2 kernel k is deﬁned by all the possible combinations of 2 positive/negative steps, plus the initial radius-1 kernel.
Revised: Then the radius 2 kernel k is deﬁned by all the possible combinations of 2 positive/negative steps, plus the initial radius-1 kernel.

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: with sgn the sign function sgn ( i ) =
Revised: with sgn the sign function sgn ( i ) =

Original: + if i ≥ 0 and − if i < 0 .
Revised: + if i ≥ 0 and − if i < 0 .

Original: The matrix B r 2 then realises the kernel a 5 × 5 .
Revised: The matrix B r 2 then realises the kernel a 5 × 5 .

Original: We can further extend the above construction to N dimension grids and radius R kernels k
Revised: We can further extend the above construction to N dimension grids and radius R kernels k

Original: [Equation]
Revised: [Equation]

Revised: in

Original: [Table]
Revised: [Equation]

Original: with F j = N j π ∇ arccos φ j , φ j the eigenvector with lowest eigenvalue only dependent on the j -th variable and given in theorem C.1 and (cid:81) is the matrix multiplication.
Revised: with F j = N j π ∇ arccos φ j , φ j the eigenvector with lowest eigenvalue only dependent on the j -th variable and given in theorem C.1 and (cid:81) is the matrix multiplication.

Original: V represents all the choices of walk { v 1 , v 2 , ..., v n } in the direction of the ﬁelds { F 1 , F 2 , ..., F n } .
Revised: V represents all the choices of walk { v 1 , v 2 , ..., v n } in the direction of the ﬁelds { F 1 , F 2 , ..., F n } .

Original: For example, V = { 3 , 1 , 0 , − 2 } has a radius R = 6 , with 3 steps forward of F 1 , 1 step forward of F 2 , and 2 steps backward of F 4 . 
Revised: For example, V = { 3 , 1 , 0 , − 2 } has a radius R = 6 , with 3 steps forward of F 1 , 1 step forward of F 2 , and 2 steps backward of F 4 . C. P ROOF FOR T HEOREM 2.8 (C OMPARISON WITH 1-WL TEST )

Revised: DGNs using the mean aggregator, any directional aggregator of the ﬁrst eigenvector and injective degree-scalers are strictly more powerful than the 1-WL test.

Revised: Proof.

Revised: We will show that (1) DGNs are at least as powerful as the 1-WL test and (2) there is a pair of graphs which are not distinguishable by the 1-WL test which DGNs can discriminate.

Revised: Since the DGNs include the mean aggregator combined with at least an injective degree-scaler,

Revised: Corso et al.

Revised: (2020) show that the resulting architecture is at least as powerful as the 1-WL test.

Revised: [Figure]

Revised: [Equation]

Revised: [Figure]

Revised: Then, to show that the DGNs are strictly more powerful than the 1-WL test it sufﬁces to provide an example of a pair of graphs that DGNs can differentiate and 1-WL cannot.

Revised: Such a pair of graphs is illustrated in ﬁgure 8.

Revised: The 1-WL test (as any MPNN with, for example, sum aggregator) will always have the same features for all the nodes labelled with a and for all the nodes labelled with b and, therefore, will classify the graphs as isomorphic.

Revised: DGNs, via the directional smoothing or directional derivative aggregators based on the ﬁrst eigenvector of the Laplacian matrix, will update the features of the a nodes differently in the two graphs (ﬁgure 8 presents also the aggregation functions) and will, therefore, be capable of distinguishing them.